Push kkkmrtvotrsu #77
No files matched your search
+1
-1
@@ -1119,7 +1119,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1370,7 +1370,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1128,7 +1128,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1130,7 +1130,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1140,7 +1140,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+179
-1234
File diff suppressed because it is too large.
Load diff
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -1485,8 +1485,9 @@ this pipeline.</p>
|
||||
by <code>verify_presence</code>/<code>verify_merge_presence</code>, which only check index
|
||||
<em>content</em> against the <code>.npz</code> truth, never the query API.</li>
|
||||
<li><code>pack --sparse</code> produces a presence-matrix format documented (see
|
||||
<a href="../../architecture/siblings/">siblings.md</a>) as faster for single-row
|
||||
access (query) and slower for column-oriented access (phylo <code>--metric</code>).
|
||||
<a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats") as faster
|
||||
for single-row access (query) and slower for column-oriented access
|
||||
(phylo <code>--metric</code>).
|
||||
<code>global_index_presence/</code> built by <code>merge_presence.sh</code> is always packed
|
||||
dense (packing is a stage inside <code>merge</code>, not a separate <code>pack</code>
|
||||
invocation) — there was no dense/sparse regression check.</li>
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -579,7 +579,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1388,7 +1388,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1504,7 +1504,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -2208,7 +2208,7 @@
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">])</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">>></span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">fill_sub_matrix</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">slots</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="p">[</span><span class="kt">usize</span><span class="p">],</span><span class="w"> </span><span class="n">out</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="p">[</span><span class="nb">Vec</span><span class="o"><</span><span class="kt">bool</span><span class="o">></span><span class="p">])</span>
|
||||
</code></pre></div>
|
||||
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this degrades to a row-by-row decode; see <a href="../../architecture/siblings/">siblings.md</a>.</p>
|
||||
<p>Column-first to match the on-disk column-major layout. <code>fill_sub_matrix</code> sorts <code>slots</code> once, then calls each column's <code>fill_batch</code> in turn — no redundant per-column sort. On <code>PersistentSparseBitMatrix</code> (k-mer-major, no column method) this goes through <code>nonzero_iter</code> instead, decoding one row at a time; see <a href="../obicompactvec/">obicompactvec.md</a>, "Sparse matrix formats".</p>
|
||||
<hr />
|
||||
<h2 id="layeredstores-and-aggregation-traits">LayeredStore\<S> and aggregation traits</h2>
|
||||
<p><code>LayeredStore<S></code> is a generic aggregation wrapper over <code>Vec<S></code>. It propagates three traits from <code>obicompactvec::traits</code> up the hierarchy via blanket impls:</p>
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1348,7 +1348,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1332,7 +1332,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1343,7 +1343,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -494,7 +494,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -632,6 +632,46 @@
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#sparse-matrix-formats">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sparse matrix formats
|
||||
|
||||
</span>
|
||||
</a>
|
||||
<nav aria-label="Sparse matrix formats" class="md-nav">
|
||||
<ul class="md-nav__list">
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#persistentsparsebitmatrix">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentSparseBitMatrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#nonzero_iter-the-one-traversal-per-format">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
nonzero_iter — the one traversal per format
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#persistentsparsecompactintmatrix">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentSparseCompactIntMatrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
</nav>
|
||||
</li>
|
||||
<li class="md-nav__item">
|
||||
<a class="md-nav__link" href="#aggregation-traits-matrix-level">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
@@ -1191,7 +1231,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
<tr>
|
||||
<td><code>Sparse</code></td>
|
||||
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
||||
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
|
||||
<td><code>pack --sparse</code>; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Implicit</code> (bit only)</td>
|
||||
@@ -1201,7 +1241,6 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tbody>
|
||||
</table>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
|
||||
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||
@@ -1212,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
<p>No wrapper enums (<code>BitColView</code>, <code>IntColView</code>): the caller receives a <code>Copy</code> view struct immediately usable with any view method or bulk builder method.</p>
|
||||
<p><code>pack_compact_int_matrix</code> and <code>pack_bit_matrix</code> convert columnar → packed format.</p>
|
||||
<hr/>
|
||||
<h2 id="sparse-matrix-formats">Sparse matrix formats</h2>
|
||||
<p>Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||
k-mers — present in only a handful of genome columns out of dozens to
|
||||
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||
formats above, built once per layer like the rest of the pipeline (never the
|
||||
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||
Structures for Collections of Sets" (2025,
|
||||
<code>biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf</code>) — a plain dedup dictionary, not their full subset-containment
|
||||
hierarchy.</p>
|
||||
<h3 id="persistentsparsebitmatrix"><code>PersistentSparseBitMatrix</code></h3>
|
||||
<p>Four mmap-backed on-disk components: an <code>is_multi</code> rank-capable flag per row
|
||||
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||
(genome index directly, <code>ceil(log2(n_cols))</code> bits), a separate fixed-width
|
||||
array for multi-genome rows (<code>dict_id</code>, <code>ceil(log2(n_distinct_multi_sets))</code>
|
||||
bits — kept apart from the singleton array because <code>n_distinct_multi_sets</code>
|
||||
can be large in absolute terms even when multi-genome rows are a small
|
||||
<em>fraction</em> of all rows, and a shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||
blob). Supporting primitives: <code>PersistentFixedIntVec</code> (arbitrary,
|
||||
runtime-parameterised bit width, width <code>0</code> included — needed once an
|
||||
<code>EliasFano</code> reopen bug surfaced, see below), <code>PersistentRankSelectBitVec</code>
|
||||
(rank1/rank0/select1 on top of the crate's existing <code>count_ones</code>, via
|
||||
<code>common_traits::SelectInWord</code>), <code>EliasFano</code> (composes the two). <code>BinaryMatrix</code>
|
||||
(<code>n</code>, <code>n_cols</code>, <code>row</code>/<code>fill_row</code>, <code>fill_sub_matrix</code>, <code>count_ones</code>) is the
|
||||
trait unifying dense and sparse at the call sites that need both
|
||||
interchangeably; column-oriented methods (<code>col</code>, <code>col_view</code>, the
|
||||
<code>partial_*_dist_matrix</code> family) stay dense-only — <code>Sparse</code>/<code>Implicit</code> panic
|
||||
there (see the <code>PersistentBitMatrix::open</code> note above).</p>
|
||||
<p>Measured on real data (one layer, 30.2M rows, 91 genomes):</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th></th>
|
||||
<th>dense</th>
|
||||
<th>sparse</th>
|
||||
<th>ratio</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>on-disk size</td>
|
||||
<td>328.1MB</td>
|
||||
<td>43.7MB</td>
|
||||
<td><strong>7.5x</strong> smaller</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>build time / peak RSS</td>
|
||||
<td>—</td>
|
||||
<td>4.26s / 628MB</td>
|
||||
<td>per-layer, in-memory construction</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>row access, sequential (2M reads)</td>
|
||||
<td>43ns/row</td>
|
||||
<td>32ns/row</td>
|
||||
<td>sparse faster (smaller structure, better cache fit)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>row access, random (2M reads)</td>
|
||||
<td>409ns/row</td>
|
||||
<td>85ns/row</td>
|
||||
<td>sparse <strong>~4.8x faster</strong> — the <code>--shannon</code>/family-lookup access shape</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>column access, one full column (30.2M rows)</td>
|
||||
<td>11.5ms</td>
|
||||
<td>993ms</td>
|
||||
<td>sparse <strong>86x slower</strong> — no native column method, every read decodes a full row to keep one bit</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>The row-access wins (compactness was the stated design goal, not speed) come
|
||||
from dense's genome-major layout scattering a single row read across a much
|
||||
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||
is this much smaller. Column access is the flip side of the same layout
|
||||
choice; <code>nonzero_iter</code> (below) is what removes it from the hot path that
|
||||
actually matters (<code>query</code>).</p>
|
||||
<p>Two reopen-correctness bugs, now fixed at the source: (1) <code>EliasFano::open</code>
|
||||
re-derived its low-bits width from the persisted low-vector's own width
|
||||
byte; the width-<code>0</code> case was built with a dummy 1-bit placeholder (the
|
||||
builder used to reject true width <code>0</code>), silently doubling every reopened
|
||||
value. Fixed by making <code>PersistentFixedIntVec</code> genuinely support width <code>0</code>
|
||||
(no storage, <code>get</code> always <code>0</code>). (2) An empty row (cardinality 0) used to be
|
||||
recorded as a singleton at genome <code>0</code>, indistinguishable on read-back from a
|
||||
<em>real</em> singleton at genome <code>0</code>. Fixed by routing cardinality-0 rows through
|
||||
the dictionary path (a genuine empty entry).</p>
|
||||
<p>Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||
exact-duplicate case is built), a sparse <code>PersistentCompactIntMatrix</code>
|
||||
variant beyond what's described below, BRWT-style column-correlation
|
||||
exploitation, and a row-major rewrite of
|
||||
<code>partial_jaccard_dist_matrix</code>/<code>partial_hamming_dist_matrix</code>/etc. as a
|
||||
per-row co-occurrence accumulation (<code>O(Σ_rows k̄²)</code>) so <code>obikindex</code>'s
|
||||
<code>--metric</code> distance-matrix path stops paying the column-access penalty on
|
||||
<code>Sparse</code> — needs its own design pass (how it plugs into
|
||||
<code>BitPartials</code>/<code>ColumnWeights</code> so both matrix types keep serving <code>--metric</code>),
|
||||
not just "port the loop".</p>
|
||||
<h3 id="nonzero_iter-the-one-traversal-per-format"><code>nonzero_iter</code> — the one traversal per format</h3>
|
||||
<p><code>fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a></code>
|
||||
(item: index into <code>slots</code>, column, value) is the single foundational
|
||||
per-format traversal primitive every consumer builds on — <code>fill_sub_matrix</code>
|
||||
and a <code>for_each</code>-style callback are both trivial, free consumers of it
|
||||
(<code>.for_each(f)</code> is <code>Iterator::for_each</code>; <code>fill_sub_matrix</code> drains the
|
||||
iterator and scatters into <code>out[][]</code>), instead of two independently
|
||||
hand-maintained traversals that can silently diverge. <code>Box<dyn Iterator></code>,
|
||||
not a bare <code>impl Iterator</code>: <code>Columnar</code>/<code>Packed</code>/<code>Sparse</code>/<code>Implicit</code> are
|
||||
genuinely different concrete types and the method isn't on a
|
||||
dyn-compatible trait (<code>BinaryMatrix</code> is used as <code>dyn BinaryMatrix</code>
|
||||
elsewhere, and RPITIT methods aren't dyn-compatible) — one <code>Box</code> per
|
||||
<code>nonzero_iter</code> call, not per cell.</p>
|
||||
<p>Per-format implementation: <code>PersistentSparseBitMatrix</code> decodes natively,
|
||||
buffered one row at a time (<code>std::iter::from_fn</code> over the existing
|
||||
<code>for_each_genome_in_row</code>, no <code>n_cols</code>-wide allocation ever). <code>Columnar</code>/
|
||||
<code>Packed</code> loop columns, collecting each column's hits via
|
||||
<code>BitSliceView::nonzero_among_sorted</code>/<code>IntSliceView::nonzero_among_sorted</code>
|
||||
(a <code>filter</code>/<code>filter_map</code> over the vector-level <code>get()</code>, no new state
|
||||
machine — <code>std</code>'s own <code>Filter</code> iterator already is one) into one <code>Vec</code> via
|
||||
<code>.extend()</code> (not <code>flat_map</code>: a <code>flat_map</code> closure can't lazily return
|
||||
something borrowing its own captured sort permutation across separate calls
|
||||
without either boxing per-column or fighting the borrow checker — eager
|
||||
collection sidesteps it at zero extra cost, since <code>fill_sub_matrix</code> already
|
||||
fully materialises anyway). <code>Implicit</code> is trivial (<code>slots.iter().map(|&i| (i, 0, 1))</code>).</p>
|
||||
<p>This closed a real dispatch bug by construction, not by patching it:
|
||||
<code>PersistentBitMatrix::fill_sub_matrix</code>/<code>sub_matrix</code> used to reimplement a
|
||||
naive per-<code>(column, slot)</code> loop instead of delegating to
|
||||
<code>PersistentSparseBitMatrix</code>'s own efficient native method for <code>Self::Sparse</code>
|
||||
— two hand-written traversals for the same format, free to drift apart, and
|
||||
they had (<code>obikphylo</code>'s own cache layer had independently built a parallel
|
||||
enum wrapper that <em>did</em> dispatch correctly, evidence the bug was worked
|
||||
around rather than fixed at the source). With <code>fill_sub_matrix</code>/<code>sub_matrix</code>
|
||||
rewritten to drain <code>nonzero_iter</code>, there is only one traversal per format
|
||||
left to get right. <code>KmerPartition::query_partition_with</code>
|
||||
(<code>obikindex/src/partition/query_layer.rs</code>) — previously column-major
|
||||
(<code>for g { for slot { col_value } }</code>, which for <code>Sparse</code> rebuilt a full row
|
||||
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||
motivation for this primitive) — now issues one <code>layer.nonzero_iter(&slot_list)</code>
|
||||
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||
reproducibly; now within ~1-3% either way — noise level).</p>
|
||||
<h3 id="persistentsparsecompactintmatrix"><code>PersistentSparseCompactIntMatrix</code></h3>
|
||||
<p>The count-matrix analogue, built on <code>PersistentSparseBitMatrix</code> as its
|
||||
"which columns are non-zero" support, values <strong>not</strong> deduplicated (unlike
|
||||
the bit side, two rows can share the same non-zero column set — same
|
||||
<code>dict_id</code> — while carrying different counts, so the dict-multiplicity
|
||||
shortcut <code>BitPartials for PersistentSparseBitMatrix</code> uses doesn't carry
|
||||
over). <code>CountPartials</code> here: a single row-major pass (<code>row_major_pairwise</code>
|
||||
in <code>sparse_intmatrix.rs</code>, decodes each row once via
|
||||
<code>for_each_cell_in_row</code>, nests over that row's own co-present columns) —
|
||||
<code>O(Σ k̄²)</code> over populated rows, avoiding the naive <code>O(n_cols² × n)</code>
|
||||
column-pair rescan. <code>min(a,b)</code>-based kernels (<code>bray</code>, <code>relfreq_bray</code>) need
|
||||
no correction — they vanish when either side is absent. <code>a·b</code> and <code>√(a·b)</code>
|
||||
kernels (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) do need one, since
|
||||
<code>(a-0)² = a² ≠ 0</code> unlike the <code>min</code>-based formulas — reconstructed from
|
||||
per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>.
|
||||
<code>threshold_jaccard(1)</code> shortcuts to <code>BitPartials::partial_jaccard</code>
|
||||
(threshold 1 is exactly presence); <code>threshold_jaccard(0)</code> is closed-form
|
||||
(every <code>u32</code> is <code>≥ 0</code>).</p>
|
||||
<p>Wiring the <code>threshold==1</code> shortcut surfaced two pre-existing formula bugs in
|
||||
<code>BitPartials for PersistentSparseBitMatrix</code> (present since that type's own
|
||||
introduction, never caught because no test had compared <code>Sparse</code>'s raw
|
||||
<code>partial_*</code> output against dense on real data — only the diagonal-blind
|
||||
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested),
|
||||
fixed at the source:</p>
|
||||
<ol>
|
||||
<li><code>partial_jaccard</code>'s diagonal used to be <code>(0, 2×col_weights[i])</code> instead
|
||||
of a genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
|
||||
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with itself
|
||||
by construction, so the diagonal needs its own case.</li>
|
||||
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: it computed
|
||||
<code>total - union</code> (rows where <em>neither</em> column is present) instead of the
|
||||
actual Hamming distance <code>col_weights[i] + col_weights[j] - 2×inter[i,j]</code>
|
||||
(symmetric-difference size) — the two only coincide when
|
||||
<code>col_weights[i] + col_weights[j] == total</code>, which small/synthetic test
|
||||
data can easily satisfy by accident.</li>
|
||||
</ol>
|
||||
<h2 id="aggregation-traits-matrix-level">Aggregation traits (matrix level)</h2>
|
||||
<h3 id="columnweights">ColumnWeights</h3>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">trait</span><span class="w"> </span><span class="n">ColumnWeights</span><span class="p">:</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="w"> </span><span class="p">{</span>
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1293,7 +1293,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1305,7 +1305,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1563,7 +1563,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1405,7 +1405,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1388,7 +1388,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1254,7 +1254,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1382,7 +1382,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1326,7 +1326,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1232,7 +1232,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1354,7 +1354,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -1228,7 +1228,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1314,7 +1314,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1227,7 +1227,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1199,7 +1199,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1265,7 +1265,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -932,6 +932,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Automatic α estimation (--gamma-shape auto)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -943,6 +954,45 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Explicit --session (implemented 2026-09-12)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1691,7 +1741,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
@@ -2268,6 +2318,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#automatic-estimation-gamma-shape-auto" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Automatic α estimation (--gamma-shape auto)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -2279,6 +2340,45 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-estimation-literature-survey-and-design-discussion-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
α-estimation literature survey and design discussion (2026-09-11)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#explicit-session-implemented-2026-09-12" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Explicit --session (implemented 2026-09-12)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -4476,9 +4576,75 @@ F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
||||
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
||||
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
||||
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
|
||||
by the user (<code>--gamma-shape <alpha></code>), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.</p>
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> either
|
||||
supplied by the user (<code>--gamma-shape <alpha></code>) or estimated from the data
|
||||
(<code>--gamma-shape auto</code>/<code>estimate</code>, method-of-moments — not ML; see
|
||||
"Automatic α estimation" below). A modifier on existing values, not a
|
||||
separate enum arm per distance.</p>
|
||||
<h3 id="automatic-estimation-gamma-shape-auto">Automatic α estimation (<code>--gamma-shape auto</code>)</h3>
|
||||
<p><strong>Correction (verified against the primary source, 2026-09-11):</strong> Jin &
|
||||
Nei (1990) itself (<em>"Limitations of the Evolutionary Parsimony Method of
|
||||
Phylogenetic Analysis"</em>, Mol. Biol. Evol. 7(2):82–102 — the paper this
|
||||
whole <code>+Γ</code> correction is cited from, confirmed algebraically to match this
|
||||
codebase's <code>corrected_log</code>/<code>k2p</code> exactly against their eq. A4, general, and
|
||||
A8, the <code>a = 1</code> case) contains <strong>no</strong> data-driven α-estimation procedure.
|
||||
Their own recommendation (p. 98) is a fixed default: <em>"we suggest that the
|
||||
gamma distance with a = 1 [eq. A8] be used. However, one may choose a
|
||||
different gamma distance, estimating a from data. Wilson et al. (1989)
|
||||
recently used a distance with a = 1/2 for restriction-site data of
|
||||
mitochondrial DNA in hominoids."</em> — i.e. Jin & Nei explicitly punt
|
||||
data-driven estimation to a <em>different</em> paper (Wilson et al. 1989), not
|
||||
read/verified here. The estimator below is therefore <strong>not</strong> "Jin & Nei's
|
||||
method" under any framing — that attribution (present in an earlier
|
||||
revision of this section) was wrong, not just under-cited.</p>
|
||||
<p><strong>Implemented</strong> (<code>PartitionDispersion</code>, <code>obikphylo/src/siblings/algorithms/pairwise.rs</code>)
|
||||
as an independent method-of-moments estimator, unrelated to any specific
|
||||
published procedure: pools substitution counts by <strong>partition</strong> rather
|
||||
than by genome pair, during the same <code>reduce_pairwise</code> pass that builds
|
||||
<code>PairwiseTally</code> (no second scan).</p>
|
||||
<p>For partition <code>i</code>: <code>n_i</code> = substitutions pooled over every genome pair,
|
||||
<code>L_i</code> = eligible loci pooled over every genome pair, <code>R_i = n_i / L_i</code>.
|
||||
Modeling among-site rate heterogeneity the same way as the <code>+Γ</code> correction
|
||||
itself (a <code>Gamma(α, α)</code>-distributed, mean-1, multiplicative rate shared by
|
||||
every locus in a partition — the classical Poisson-Gamma/negative-binomial
|
||||
mixture, the general identity behind gamma-rate-heterogeneity corrections,
|
||||
also behind Uzzell & Corbin's (1971) original observation that substitution
|
||||
counts across sites/regions are over-dispersed relative to Poisson):</p>
|
||||
<div class="arithmatex">\[
|
||||
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
|
||||
\]</div>
|
||||
<p>Weighting each partition's squared deviation by its own <code>L_i</code> cancels the
|
||||
Poisson term before attributing what's left to <code>α</code>:</p>
|
||||
<div class="arithmatex">\[
|
||||
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
|
||||
V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
|
||||
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
|
||||
\]</div>
|
||||
<p>where <code>\bar L</code> is the mean partition size. Returns "no estimate" (falls
|
||||
back to the uncorrected formula, warns) when fewer than 2 partitions have
|
||||
data, <code>\hat\mu \le 0</code>, or <code>V</code> doesn't exceed the Poisson floor
|
||||
<code>\hat\mu/\bar L</code> — no detectable over-dispersion, <code>α</code> would be unbounded.</p>
|
||||
<p><strong>Caveat, stated explicitly rather than left implicit</strong>: unlike every
|
||||
closed-form correction in <code>snp_distance.rs</code> (each verified line-by-line
|
||||
against <code>ape</code>'s <code>dist_dna.c</code>, and now also against Jin & Nei 1990 directly
|
||||
for the base <code>+Γ</code> formula), this estimator is derived from first
|
||||
principles (the general Poisson-Gamma/negative-binomial identity) with no
|
||||
primary-source procedure behind it at all — not Jin & Nei's (confirmed
|
||||
above), and Wilson et al. (1989), the paper they point to instead, hasn't
|
||||
been read/verified either. Mathematically self-consistent (re-derived and
|
||||
checked, not guessed), but a from-scratch method, not a literature
|
||||
implementation. If <code>--gamma-shape</code> needs a value with a literature
|
||||
pedigree rather than an estimated one, Jin & Nei's own stated default,
|
||||
<code>α = 1</code> (<code>--gamma-shape 1</code>), is the better-supported choice today.</p>
|
||||
<p>Deliberately <strong>not</strong> gated by <code>--sankoff-ratio-ceiling</code> the way
|
||||
<code>base_pair_tally</code> is (same precedent as <code>cardinality_tally</code> — see its own
|
||||
doc comment): that filter excludes individual saturated <em>pairs</em> from a
|
||||
composition estimate computed once at the very end, from the complete
|
||||
<code>PairwiseTally</code>; the partition axis needed here only exists transiently,
|
||||
one partition at a time, while <code>PairwiseTally</code> is still being built — long
|
||||
before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
|
||||
is known. <code>--exclude-genome</code> isn't applied either, matching
|
||||
<code>reduce_pairwise</code>'s own raw per-pair fold.</p>
|
||||
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
||||
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
||||
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
||||
@@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.</p>
|
||||
<h3 id="-estimation-literature-survey-and-design-discussion-2026-09-11">α-estimation literature survey and design discussion (2026-09-11)</h3>
|
||||
<p>Follow-up discussion after implementing <code>PartitionDispersion::estimate_alpha</code>
|
||||
(previous section), triggered by the user pointing out that the literature
|
||||
conflates two distinct problems: <em>estimating α from an alignment</em> versus
|
||||
<em>using a given α in a Jin-Nei-style corrected distance</em>. Jin & Nei (1990)
|
||||
itself is squarely in the second camp — α is a user-supplied parameter
|
||||
there, not something their paper estimates (confirmed by reading the paper
|
||||
directly, see previous section). This section catalogs the actual
|
||||
α-estimation literature and records which approaches were considered and
|
||||
why each was or wasn't adopted.</p>
|
||||
<p><strong>Bibliography</strong> (methodological papers, not exhaustive users of <code>+Γ</code>):</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Method</th>
|
||||
<th>Reference</th>
|
||||
<th>Needs a tree?</th>
|
||||
<th>Estimates "true" α or a task-specific one?</th>
|
||||
<th>Cost</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>ML, continuous Γ (foundational)</td>
|
||||
<td><strong>Gu, Fu & Li (1995)</strong>, MBE 12:546–557 — read in full</td>
|
||||
<td>yes, topology fixed a priori</td>
|
||||
<td>true α (+ θ jointly, invariant+Γ)</td>
|
||||
<td>very high, ≤5–6 taxa in practice</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>ML, discrete Γ</td>
|
||||
<td>Yang (1993/94), MBE 10:1396; JME 39:306</td>
|
||||
<td>yes</td>
|
||||
<td>true α</td>
|
||||
<td>high</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Parsimony-based ML on inferred changes</td>
|
||||
<td>Yang & Kumar (1996), MBE 13:650</td>
|
||||
<td>yes, equal branch lengths assumed</td>
|
||||
<td>true α, biased when branches unequal</td>
|
||||
<td>low</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Corrected substitutions/site → NB fit</td>
|
||||
<td><strong>Gu & Zhang (1997)</strong>, MBE 14:1106 — read in full</td>
|
||||
<td>yes, + ancestral reconstruction</td>
|
||||
<td>true α, close to ML accuracy</td>
|
||||
<td>low (given a tree)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Topology-optimizing grid search</td>
|
||||
<td><strong>Guindon & Gascuel (2002)</strong>, MBE 19:534 — read in full</td>
|
||||
<td>yes, but built <em>from</em> the candidate distances themselves (BIONJ)</td>
|
||||
<td><strong>not</strong> true α — deliberately biased <em>upward</em> for topological accuracy</td>
|
||||
<td>low–moderate (grid × tree build)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Free (non-parametric) rate distribution</td>
|
||||
<td>Susko et al. (2003), Syst. Biol. 52:594</td>
|
||||
<td>yes</td>
|
||||
<td>tests the Γ assumption itself</td>
|
||||
<td>high</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Parsimony counts/site → NB fit</td>
|
||||
<td><strong>Wakeley (1993)</strong>, JME 37:613 — read in full</td>
|
||||
<td>yes, NJ tree + Fitch parsimony ancestral reconstruction</td>
|
||||
<td>true α, biased <em>upward</em> by parsimony (conservative)</td>
|
||||
<td>low (given a tree)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Poisson-Gamma/negative-binomial, foundational</td>
|
||||
<td>Uzzell & Corbin (1971), Science 172:1089 — <strong>not read</strong>, only seen via citations</td>
|
||||
<td>unknown — likely also count-per-site/tree-based, unverified</td>
|
||||
<td>true α (indirect, via NB overdispersion)</td>
|
||||
<td>unknown</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Bayesian posterior over α</td>
|
||||
<td>BPP/MrBayes/BEAST2/RevBayes</td>
|
||||
<td>yes</td>
|
||||
<td>posterior, not point estimate</td>
|
||||
<td>very high</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong>Correction (2026-09-11, after reading Wakeley in full):</strong> an earlier
|
||||
revision of this table filed Wakeley (1993) under "no tree needed" —
|
||||
wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
|
||||
1987, on the real mtDNA data; coalescent-simulated for the simulation
|
||||
study), reconstruct ancestral states at internal nodes by Fitch (1971)
|
||||
parsimony, count the <em>minimum</em> number of changes per site this implies,
|
||||
then fit either a two-rate Poisson mixture or a gamma-distributed-rates
|
||||
model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
|
||||
Johnson & Kotz 1969) to the resulting per-site count distribution — the
|
||||
same general "tree + ancestral reconstruction + per-site counts + NB fit"
|
||||
shape as Gu & Zhang (1997), just using raw parsimony counts directly
|
||||
rather than Gu & Zhang's multiple-hit-corrected <code>k̂</code>. Confirmed by
|
||||
simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
|
||||
mean <em>and</em> (more severely) the variance of per-site change counts, which
|
||||
biases <code>α</code> <strong>upward</strong> (toward apparent uniformity) — consistent with Gu &
|
||||
Zhang's own citation of Wakeley for exactly this bias, and with the
|
||||
general "parsimony overestimates α" pattern noted throughout this
|
||||
literature. Real hypervariable-region-1 mtDNA data gave <code>α̂ ≈ 0.44–0.60</code>
|
||||
per data set, <code>0.47</code> combined — despite the conservative bias, still
|
||||
solidly in the "strong heterogeneity" range, which is the paper's own
|
||||
point: the method is biased but usefully conservative, not useless.</p>
|
||||
<p><strong>Where <code>PartitionDispersion::estimate_alpha</code> actually sits</strong>: nowhere in
|
||||
this table's tree-dependent rows. It shares only the <em>abstract
|
||||
mathematical identity</em> (Poisson-Gamma mixture ⟹ negative binomial
|
||||
over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
|
||||
genuinely different, tree-free and ancestral-reconstruction-free route:
|
||||
pairwise genome-to-genome substitution counts pooled by <em>partition</em>
|
||||
instead of per-site counts pooled across an alignment's columns after
|
||||
ancestral state reconstruction. As far as this survey has established
|
||||
(Uzzell & Corbin not read, so not fully ruled out), <strong>no tree and no
|
||||
ancestral reconstruction of any kind</strong> appears to be a genuinely
|
||||
distinguishing property of <code>PartitionDispersion</code>, not something it
|
||||
inherits from prior art — consistent with it being an independent
|
||||
derivation rather than a literature implementation (see previous
|
||||
section's caveat, now on firmer footing).</p>
|
||||
<p><strong>Why a tree-free estimate is even valid — the identifiability question.</strong>
|
||||
Raised directly by the user after reading Gu, Fu & Li (1995), which states
|
||||
explicitly: <em>"one cannot estimate the rate heterogeneity when the number
|
||||
of sequences is <3."</em> This is a real non-identifiability result, not a
|
||||
practical inconvenience: with only 2 sequences, a site's history is
|
||||
summarized by one binary observation (differ / don't differ), collapsed
|
||||
across all sites into two aggregate numbers (transition and transversion
|
||||
proportions) — divergence time and among-site rate variance are
|
||||
confounded in that pair of numbers, with no way to separate them. Every
|
||||
method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
|
||||
sequences <em>sharing one evolutionary history</em> (a tree) specifically to get
|
||||
multiple independent looks at the <em>same</em> site's rate across different
|
||||
lineages, which is what identifies its variance.</p>
|
||||
<p><code>PartitionDispersion</code> doesn't violate this, because it isn't estimating
|
||||
the same quantity. The literature above estimates rate variation <strong>across
|
||||
sites, within one shared tree</strong>. <code>PartitionDispersion</code> estimates rate
|
||||
variation <strong>across partitions, pooled over every genome pair in the
|
||||
index</strong> — each genome pair stands in for one independent draw of a
|
||||
partition's relative rate, the same identifying role multiple lineages
|
||||
play in the tree-based methods, just substituting "genome pair" for
|
||||
"lineage" and "partition" for "site." With <code>P</code> genome pairs (potentially
|
||||
in the hundreds here) all contributing to every partition's pooled count,
|
||||
the ≥3-samples identifiability requirement is met by the pair count, not
|
||||
by tree depth — closer in spirit to Jin & Nei's own original motivating
|
||||
scenario (comparing several genomic <em>regions</em> across one fixed panel of
|
||||
taxa) than to single-tree site-rate estimation.</p>
|
||||
<p>This substitution carries its own assumption, which must be named rather
|
||||
than left implicit: that a partition's <em>relative</em> rate (fast/slow) is
|
||||
reasonably stable <strong>across genome pairs</strong>, not just across sites within
|
||||
one pair — the direct analogue of the "no lineage-specific rate variation,
|
||||
only site-specific" assumption every tree-based <code>+Γ</code> method already makes,
|
||||
just moved from lineage→pair and site→partition. If this breaks — e.g. an
|
||||
index mixing very closely related and very divergent genome pairs, where a
|
||||
partition's saturation behavior differs qualitatively between the two —
|
||||
the estimate could be misled in a way a tree-based method would at least
|
||||
have the topology to detect and a tree-free, pool-everything method
|
||||
cannot. Not yet tested against real data with strongly heterogeneous
|
||||
pairwise divergence; worth keeping in mind as the main open validity
|
||||
question for <code>--gamma-shape auto</code>, not the citation question (now settled)
|
||||
this whole discussion started from.</p>
|
||||
<p><strong>Gu, Fu & Li (1995), read in full — the foundational ML method, not a
|
||||
candidate to implement.</strong> Models rate variation as invariant+Γ (<code>θ</code> =
|
||||
proportion of invariant sites, <code>α</code> = gamma shape among the variable sites),
|
||||
and derives the exact joint likelihood over nucleotide <em>configurations</em>
|
||||
across all <code>n</code> sequences on a <strong>fixed, a-priori-known tree topology</strong> —
|
||||
for 3 sequences this is closed-form (Jukes-Cantor), for <code>n</code> sequences it's
|
||||
a sum over <code>4^n</code> configurations weighted by coefficients tied to the
|
||||
specific topology's branching structure (their eq. 32–34), with branch
|
||||
lengths, <code>α</code>, and <code>θ</code> jointly optimized by Newton-Raphson (Hessian over all
|
||||
free parameters). The site-rate integral has the same closed form
|
||||
<code>E = θ + (1-θ)(1+D/α)^{-α}</code> that underlies <code>+Γ</code> distance corrections
|
||||
generally (the same mechanism as <code>corrected_log</code>, not a coincidence — Jin &
|
||||
Nei's own formula is the <code>θ=0</code> special case applied to a pairwise <code>D</code>
|
||||
rather than a whole-tree one). Explicitly stated by the authors as
|
||||
infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
|
||||
candidate for obikmer's typical genome counts. Notable contribution worth
|
||||
keeping in mind regardless: <code>ρ = (1+θα)/(1+α)</code> is shown to be a more robust
|
||||
summary of rate heterogeneity than <code>α</code> alone whenever <code>θ</code> isn't negligible
|
||||
— <code>α</code> and <code>θ</code> are confounded (nearly-invariant sites can come from either a
|
||||
small <code>α</code>'s left tail or from a nonzero <code>θ</code>), so <code>α</code> alone can be very
|
||||
unstable while <code>ρ</code> stays well-behaved. <code>PartitionDispersion</code> has no <code>θ</code>
|
||||
term at all (no invariant-site component), so this confound doesn't arise
|
||||
for it the same way, but it's worth remembering if an invariant+Γ variant
|
||||
is ever considered.</p>
|
||||
<p><strong>Gu & Zhang (1997), read in full — rejected for now.</strong> Their procedure:
|
||||
(1) a tree with least-squares branch lengths must already exist; (2)
|
||||
ancestral states are reconstructed at every internal node (they use Zhang &
|
||||
Nei 1997's likelihood method, but note plain parsimony would work, just
|
||||
less accurately); (3) per site, branches are split into "changed"/"unchanged"
|
||||
given the ancestral reconstruction, and the <em>expected</em> substitution count
|
||||
<code>k̂</code> (corrected for multiple hits) is obtained by solving their eq. (7)
|
||||
(or eq. 12 for the generalized model) numerically — a root-find per site,
|
||||
not a closed form; (4) the site-level <code>k̂</code> values (real-valued, not
|
||||
integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
|
||||
get α — again no closed form, numerical optimization. This is a real
|
||||
per-site pipeline requiring a tree <em>before</em> distances/α can be computed —
|
||||
the reverse of obikmer's current dependency order (<code>snp_distance</code> computes
|
||||
distances, which <code>--nj</code>/<code>--upgma</code> only turn into a tree <em>afterward</em>). Their
|
||||
own answer to this chicken-and-egg problem is an iterative
|
||||
distance→tree→α→distance loop, which they themselves flag as expensive.
|
||||
Adopting this would mean a new ancestral-reconstruction subsystem (even a
|
||||
parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
|
||||
ML fit — a much bigger addition than <code>PartitionDispersion</code>, not something
|
||||
to build without a concrete need beyond "closer to Gu&Zhang's own
|
||||
simulated accuracy than our moment estimator."</p>
|
||||
<p><strong>Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.</strong>
|
||||
Their <strong>Q criterion</strong>: for a candidate α, build a tree (they use BIONJ) from
|
||||
the α-corrected distance matrix; for every internal branch, group taxa into
|
||||
the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
|
||||
distances <span class="arithmatex">\(\bar\delta_{AB}\)</span>, <span class="arithmatex">\(\bar\delta_{CD}\)</span>, etc., and let <code>S ≤ M ≤ L</code>
|
||||
be the three pairings' sums (<code>{AB,CD}</code>, <code>{AC,BD}</code>, <code>{AD,BC}</code>); the branch's
|
||||
reliability score is <code>Q_branch = L − M</code> (zero when the four points are
|
||||
perfectly tree-additive). The whole-tree <code>Q</code> is the mean over internal
|
||||
branches (negative/zero branches excluded). <code>α* = argmin_α Q(α)</code> over a
|
||||
grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
|
||||
sensitivity is highest). Complexity: <code>O(n²l)</code> once for the base
|
||||
frequencies, then <code>O(n²r)</code> for the <code>r</code> candidate distance matrices, then
|
||||
<code>O(n³r)</code> worst case for building <code>r</code> trees and evaluating <code>Q</code> on each
|
||||
(same order as the tree-building step itself, so "free" relative to it).</p>
|
||||
<p>Central finding, <strong>the opposite direction from an initial misreading in
|
||||
this discussion</strong>: <code>α_opt</code> (Guindon-Gascuel's topology-optimal value) is
|
||||
<strong>always ≥ the true α</strong>, not the reverse — underestimating rate
|
||||
heterogeneity (i.e., picking a larger α than reality) reduces the variance
|
||||
of distance estimates and thereby improves NJ/BIONJ topological accuracy,
|
||||
especially when the molecular clock roughly holds (where <code>α_opt → ∞</code>, i.e.
|
||||
no correction at all is topologically best). This is an <em>empirical
|
||||
regularity</em> observed across their tested conditions (η ∈ {0.5, 2.0}, true
|
||||
α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
|
||||
guarantee would be another unverified-claim mistake of the kind this whole
|
||||
discussion has been correcting.</p>
|
||||
<p><strong>Proposed integration (not yet implemented)</strong>: since <code>α_true ≤ α_opt</code>
|
||||
empirically, <code>PartitionDispersion::estimate_alpha()</code>'s output could seed
|
||||
the <em>lower bound</em> of Guindon-Gascuel's grid (search <code>[α̂, ∞)</code> instead of
|
||||
<code>[0.1, 5000]</code>), narrowing the grid substantially and reusing the existing
|
||||
estimator rather than needing an independent "true α" source — this also
|
||||
sidesteps Gu & Zhang's tree-first dependency problem entirely, since
|
||||
Guindon-Gascuel's own grid already builds a tree per candidate. Still
|
||||
requires: (a) a <strong>BIONJ</strong> implementation (see below), (b) the Q-criterion
|
||||
subtree-grouping logic, (c) a grid-search driver. Not started.</p>
|
||||
<p><strong>BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.</strong>
|
||||
Searched crates.io directly (API query for "bionj": 0 results) and read the
|
||||
docs.rs pages of the most plausible candidates: <code>phylotree</code> (builds/reads/
|
||||
manipulates trees but has no distance-matrix reconstruction — no NJ, no
|
||||
BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
|
||||
after checking the actual docs), <code>speedytree</code> (NJ only — Canonical/RapidNJ/
|
||||
Hybrid variants, explicitly no BIONJ), <code>nj</code> (plain NJ only), <code>phylo</code> (no
|
||||
distance-based reconstruction at all). <code>obikphylo::neighbor_joining</code>
|
||||
(<code>tree.rs</code>) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
|
||||
how branch lengths are weighted during agglomeration (accounts for
|
||||
distance-estimate variance, not just the sum), a modest delta over an
|
||||
existing NJ implementation rather than a rewrite, but would need to be
|
||||
written in-house if the Guindon-Gascuel integration above is ever pursued
|
||||
(their own paper notes other tree-building methods gave similar results in
|
||||
their simulations, so plain NJ may be an acceptable substitute if BIONJ
|
||||
proper is judged not worth the effort).</p>
|
||||
<h3 id="sharing-one-sample-across-algorithms-sankoff-tnt-phyg-iqtree-snp-distance-implemented-2026-09-11">Sharing one sample across algorithms (<code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/<code>--iqtree</code> + <code>snp-*</code> <code>--distance</code>) — implemented 2026-09-11</h3>
|
||||
<p><strong>Correctness bug, not just a performance one.</strong> <code>--sankoff</code>/<code>--tnt</code>/<code>--phyg</code>/
|
||||
<code>--iqtree</code> (via <code>SiblingExt::sankoff_bundle</code>) and a <code>snp-*</code> <code>--distance</code>
|
||||
(via <code>SiblingExt::snp_distance</code>) always consume the <em>exact same</em> selection
|
||||
parameters when requested together in one <code>obikmer phylo</code> invocation
|
||||
(<code>n</code>/<code>free_loss</code>/<code>no_ambiguity</code>/excluded-set/<code>entropy_bias</code> — the CLI has
|
||||
no way to give them different values in one run, see <code>cmd/phylo/args.rs</code>).
|
||||
Before this fix, each independently called <code>sample_index</code>, and
|
||||
<code>sample_layer</code>'s <code>rand::rng()</code> (<code>subsample.rs:235</code>) is a <strong>thread-local
|
||||
generator that advances across calls, not reseeded each time</strong> — so the
|
||||
second call silently drew a <em>different</em> random sample of sites than the
|
||||
first, even with identical parameters. Verified empirically (throwaway
|
||||
8-genome index): running <code>snp-jc</code> alone vs. combined with <code>--sankoff</code>
|
||||
produced visibly different distance matrices from the same index/params.
|
||||
This defeated the actual point of combining these flags — running several
|
||||
algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a <code>snp-*</code> distance) on
|
||||
one <em>identical</em> site selection for direct comparison.</p>
|
||||
<p><strong>Fix</strong>: <code>SankoffBundle</code> (<code>sankoff.rs</code>) now retains its <code>PairwiseTally</code>/
|
||||
<code>PartitionDispersion</code> internally (previously consumed into <code>raw</code>/
|
||||
<code>base_pair_tally</code>/<code>cardinality_tally</code> and dropped) and exposes
|
||||
<code>SankoffBundle::snp_distance(kind, gamma_shape)</code>, which computes the
|
||||
matrix from that <em>same</em> already-sampled tally — no second <code>sample_index</code>
|
||||
call. The shared post-sampling logic (the <code>--gamma-shape</code> support check,
|
||||
<code>alpha</code> resolution, final matrix build — previously all inline in the
|
||||
<code>snp_distance()</code> free function) was factored into
|
||||
<code>pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape)</code> in
|
||||
<code>snp_distance.rs</code>, called by both the standalone <code>snp_distance()</code> (after
|
||||
its own fresh sampling) and <code>SankoffBundle::snp_distance</code> (reusing the
|
||||
bundle's). <code>cmd/phylo/mod.rs</code> keeps the <code>Option<SankoffBundle></code> alive past
|
||||
the Sankoff-family <code>if</code> block and, when a <code>snp-*</code> <code>--distance</code> is also
|
||||
requested, calls <code>bundle.snp_distance(...)</code> instead of
|
||||
<code>cache.snp_distance(...)</code> whenever a bundle was built — since both branches
|
||||
are driven by the same <code>args.*</code> fields, the parameters trivially always
|
||||
match when both fire; no runtime "do the params match" check needed.
|
||||
Verified end-to-end: the reused path logs <code>(reusing the Sankoff bundle's
|
||||
sample)</code> and the <code>snp_distance</code> stage timer reads <code>0ms</code> (<code>formula(kind)</code>
|
||||
is <code>O(n²)</code> post-processing, no I/O), vs. ~150-230ms for a fresh sample on
|
||||
the same tiny test index.</p>
|
||||
<h4 id="explicit-session-implemented-2026-09-12">Explicit <code>--session</code> (implemented 2026-09-12)</h4>
|
||||
<p>The in-process fix above only covered reuse <em>within one command</em>.
|
||||
<code>--session DIR</code> (<code>cmd/phylo/args.rs</code>) names a directory (outside the
|
||||
index) that persists the <code>snp-*</code> <code>--distance</code> sample — the
|
||||
<code>PairwiseTally</code>/<code>PartitionDispersion</code> pair <code>sample_index</code> would otherwise
|
||||
rebuild from scratch — across <em>separate</em> <code>obikmer phylo</code> invocations.</p>
|
||||
<p><strong>Crate placement, as planned</strong>: a new <strong><code>obiksession</code></strong> crate
|
||||
(<code>src/obiksession/</code>), domain-agnostic — no <code>PairwiseTally</code>/"site" concept
|
||||
anywhere in it. <code>Session::open(dir, params: &[u8], force: bool)</code> handles
|
||||
directory lifecycle and the params-conflict check against an opaque byte
|
||||
blob; <code>Session::store(name, bytes)</code>/<code>Session::restore(name) -> Option<Mmap></code>
|
||||
handle a checksummed artifact cache. <code>obikphylo</code>
|
||||
(<code>siblings/algorithms/session_cache.rs</code>, shared by <code>snp_distance.rs</code> and
|
||||
<code>sankoff.rs</code> — see below) is the consumer: it serializes its own
|
||||
<code>PairwiseTally</code>/<code>PartitionDispersion</code>/<code>SnpAlignment</code> to bytes (via <code>rkyv</code>)
|
||||
and hands them to <code>obiksession</code>, which never sees their type.</p>
|
||||
<p><strong>Serialization</strong>: <code>rkyv</code> 0.8.18, added as planned. One deviation from the
|
||||
original zero-copy pitch, scoped down deliberately: artifacts are restored
|
||||
via <code>rkyv::from_bytes</code> (full owned deserialize) rather than <code>rkyv::access</code>
|
||||
(zero-copy over the <code>mmap</code>). True zero-copy would require every
|
||||
<code>PairwiseTally</code> query method (<code>categories</code>, <code>base_freq</code>, <code>pair</code>, ...) to
|
||||
work generically over <code>Archived<PairwiseTally></code> as well as the owned type
|
||||
— a separate, larger change not needed to get the actual win (skipping
|
||||
<code>sample_index</code>'s expensive re-scan of the sibling annex; deserializing an
|
||||
already-in-memory-sized count array is comparatively cheap CPU, not I/O).
|
||||
<code>#[derive(Archive, Serialize, Deserialize)]</code> was added to <code>PairStats</code>,
|
||||
<code>PairwiseTally</code>, and <code>PartitionDispersion</code> (<code>pairwise.rs</code>) for this.</p>
|
||||
<p><strong>Startup behavior, as designed</strong>: <code>Session::open</code> creates <code>DIR</code> if
|
||||
missing (fresh, params recorded); if <code>DIR</code> already holds different saved
|
||||
params, <code>cmd/phylo</code> reports a hard error and exits rather than silently
|
||||
preferring one side, unless <code>--session-force</code> is given (which discards the
|
||||
directory's cached artifacts and starts over under the new params) —
|
||||
implements exactly the "explicit escape hatch, not silent override"
|
||||
decision from the design discussion. <code>obiksession</code> itself doesn't decode
|
||||
either side's bytes to produce a field-by-field diff (true to "opaque
|
||||
blob," left to whoever needs it); <code>cmd/phylo</code>'s conflict message is
|
||||
currently a single generic sentence listing which flags could be the
|
||||
cause, not a computed diff — an honest v1 simplification, not a limitation
|
||||
of <code>obiksession</code> itself.</p>
|
||||
<p><strong>Per-artifact caching, as designed</strong>: only <code>PairwiseTally</code>/
|
||||
<code>PartitionDispersion</code> are cached (<code>snp_distance.rs</code>'s <code>restore_tally</code>/
|
||||
<code>store_tally</code>) — <code>ratio_ceiling</code>/<code>gamma_shape</code> remain pure post-processing
|
||||
over whatever tally is in hand, cached or fresh, exactly as planned.</p>
|
||||
<p><strong>Locking, reconsidered and kept as the whole-session <code>DirLock</code>.</strong> First
|
||||
written up as a corner cut ("per-entry locking was sketched, not
|
||||
implemented"). On actually working through the concrete race, whole-session
|
||||
locking turned out to be the <em>correct</em> choice, not a simplification:
|
||||
<code>--session-force</code> resets a session by deleting its whole <code>artifacts/</code>
|
||||
directory. If the directory lock were only held briefly (during
|
||||
<code>Session::open</code>, then released so a long-running <code>Session</code> doesn't block
|
||||
other processes from touching <em>different</em> artifacts), a second process
|
||||
calling <code>open(..., force: true)</code> could acquire the lock, see mismatched
|
||||
params, and wipe <code>artifacts/</code> <strong>while a first process's already-open
|
||||
<code>Session</code> is mid-checkpoint</strong> — genuine corruption risk, not just wasted
|
||||
work. Holding the lock for the <code>Session</code>'s entire lifetime makes that
|
||||
sequence impossible: a <code>force</code> reset must wait for every currently-open
|
||||
<code>Session</code> on that directory to be dropped first. The concurrency this
|
||||
gives up (two <em>different</em> processes reading/writing <em>different</em> artifacts
|
||||
of the same session at once) isn't exercised by anything today — only one
|
||||
code path (<code>snp_distance</code>/<code>sankoff_bundle</code>, sequentially per invocation)
|
||||
uses sessions at all.</p>
|
||||
<p><strong>Per-layer chunked/resumable dumps — implemented.</strong> <code>subsample::sample_index</code>
|
||||
gained a <code>skip_layers: usize</code> parameter (skip the first N (partition,
|
||||
layer) pairs, in its own deterministic iteration order, entirely — no
|
||||
I/O, no <code>on_layer</code> call) and its <code>on_layer</code> callback gained a <code>raw_index</code>
|
||||
argument (that pair's 0-based position in the same order), so a caller can
|
||||
tell exactly how far a run has gotten. <code>snp_distance</code>/<code>sankoff_bundle</code>
|
||||
checkpoint their tally/dispersion (and, for <code>sankoff_bundle</code>, the
|
||||
in-progress <code>SnpAlignment</code>) to the session every
|
||||
<code>session_cache::CHECKPOINT_INTERVAL_LAYERS</code> (8) layers or
|
||||
<code>CHECKPOINT_INTERVAL</code> (30s) of wall time, whichever comes first — bounding
|
||||
how much completed scanning a crash can lose, without paying a full
|
||||
tally-rewrite on every one of what can be thousands of layers. A <code>progress</code>
|
||||
artifact records how many layers are reflected in the checkpoint, with a
|
||||
<code>PROGRESS_COMPLETE</code> (<code>u64::MAX</code>) sentinel once every layer has been
|
||||
processed (distinct from any real layer count, so "finished" is never
|
||||
confused with "stopped after N layers where N happens to equal the total").</p>
|
||||
<p>Verified with a real kill: built a 10-genome/64-partition index, ran
|
||||
<code>--distance snp-jc --session DIR</code> under <code>timeout 0.4</code>, confirmed a
|
||||
checkpoint at <code>progress=8</code> (exactly <code>CHECKPOINT_INTERVAL_LAYERS</code>) was on
|
||||
disk, then re-ran the same command and confirmed it resumed from layer 8
|
||||
and completed successfully — 0-effort verification that the mechanism
|
||||
itself is real, not just plausible-sounding.</p>
|
||||
<p><strong>Important correction from that same test — resume is <em>not</em> bit-reproducible,
|
||||
and shouldn't be.</strong> The output matrix from the resumed run differed from
|
||||
an uninterrupted control run against the same index/params. Root cause:
|
||||
<code>sample_layer</code>'s random draws come from <code>rand::rng()</code>, a per-process,
|
||||
OS-seeded generator with no continuity across a process boundary — the
|
||||
layers processed before a kill and the layers processed after a resume
|
||||
come from two independent random streams. An earlier revision of this doc
|
||||
(and of the code comments) claimed resume "reproduces the exact same
|
||||
sample an uninterrupted run would have" — that's wrong, and was corrected
|
||||
after the user pointed out the actual reasoning error: true randomness has
|
||||
no notion of continuity in the first place, so there was never a
|
||||
correctness requirement for a resumed run to match an uninterrupted one
|
||||
bit-for-bit. The sample produced after a resume is exactly as legitimate a
|
||||
random draw as one from an uninterrupted run — just a <em>different</em> one,
|
||||
same as running the command twice without <code>--session</code> already gives two
|
||||
different samples today. That variance is explicitly wanted (see below),
|
||||
not a defect to fix. Comments in <code>subsample.rs</code>/<code>session_cache.rs</code> were
|
||||
corrected to state this plainly instead of the false reproducibility claim.</p>
|
||||
<p><strong>Explicitly declined: deterministic/seeded sampling.</strong> Raised as a
|
||||
possible fix for the above (seed <code>sample_layer</code>'s RNG from a per-session
|
||||
value plus <code>(partition, layer)</code>, making a resume reproduce bit-identical
|
||||
results, and, if applied everywhere, making <em>any</em> two runs with identical
|
||||
parameters produce identical samples). The user explicitly declined this
|
||||
for now: repeated runs are relied on to measure sampling variance, and a
|
||||
<code>--seed</code> flag would need its own explicit design if wanted later — this is
|
||||
not an accidental gap, it's a stated preference to revisit only if asked.</p>
|
||||
<p><strong><code>SankoffBundle</code> integration — implemented.</strong> <code>sankoff_bundle</code> accepts
|
||||
the same <code>session: Option<&Session></code> and uses the <em>same</em> artifact names
|
||||
(<code>session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}</code>)
|
||||
as <code>snp_distance</code>, plus its own <code>ALIGNMENT_ARTIFACT</code> (<code>sankoff.rs</code>) folded
|
||||
in lockstep with the tally at every checkpoint (same <code>on_layer</code> call
|
||||
produces both, so they can never drift out of sync). Consequence verified
|
||||
directly: a tally checkpointed by <code>--sankoff --session DIR</code> is restored by
|
||||
a <em>later, separate</em> <code>--distance snp-k2p --session DIR</code> invocation with no
|
||||
<code>--sankoff</code> at all, and vice versa — cross-consumer reuse, not just
|
||||
same-command reuse, exactly the synergy this design aimed for.</p>
|
||||
<p><strong>Per-artifact caching, as designed</strong>: only the tally/dispersion (plus,
|
||||
for <code>sankoff_bundle</code>, the alignment) are cached — <code>ratio_ceiling</code>/
|
||||
<code>gamma_shape</code> remain pure post-processing over whatever tally is in hand,
|
||||
cached or fresh, exactly as planned.</p>
|
||||
<p><strong>Still not done — genuine follow-up, not a justified tradeoff:</strong>
|
||||
<strong>exhaustive-sample caching for <code>--subsample</code>-varying exploration.</strong> The
|
||||
design discussion's suggestion — cache the exhaustive (<code>n = None</code>) sample
|
||||
once and subsample <em>from</em> it on each run instead of one session per exact
|
||||
<code>n</code> — wasn't implemented; a session today is still scoped to one exact
|
||||
parameter tuple, <code>--subsample</code> included.</p>
|
||||
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
||||
<p><strong>Implemented.</strong> The primary distance-matrix output
|
||||
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1199,7 +1199,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1124,7 +1124,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1243,7 +1243,7 @@
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sibling annex (discussion)
|
||||
Sibling annex
|
||||
|
||||
|
||||
|
||||
|
||||
+160
-1024
File diff suppressed because it is too large.
Load diff
@@ -12,8 +12,9 @@ this pipeline.
|
||||
by `verify_presence`/`verify_merge_presence`, which only check index
|
||||
*content* against the `.npz` truth, never the query API.
|
||||
- `pack --sparse` produces a presence-matrix format documented (see
|
||||
[siblings.md](../architecture/siblings.md)) as faster for single-row
|
||||
access (query) and slower for column-oriented access (phylo `--metric`).
|
||||
[obicompactvec.md](obicompactvec.md), "Sparse matrix formats") as faster
|
||||
for single-row access (query) and slower for column-oriented access
|
||||
(phylo `--metric`).
|
||||
`global_index_presence/` built by `merge_presence.sh` is always packed
|
||||
dense (packing is a stage inside `merge`, not a separate `pack`
|
||||
invocation) — there was no dense/sparse regression check.
|
||||
|
||||
@@ -299,7 +299,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
|
||||
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
|
||||
```
|
||||
|
||||
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
|
||||
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this goes through `nonzero_iter` instead, decoding one row at a time; see [obicompactvec.md](obicompactvec.md), "Sparse matrix formats".
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -298,13 +298,11 @@ Both matrix types are enums behind a transparent API — the caller never matche
|
||||
|---|---|---|
|
||||
| `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) |
|
||||
| `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` |
|
||||
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off |
|
||||
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off |
|
||||
| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` |
|
||||
|
||||
`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`.
|
||||
|
||||
Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation.
|
||||
|
||||
`col_view(c)` returns the appropriate view directly:
|
||||
|
||||
```rust
|
||||
@@ -321,6 +319,163 @@ No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view
|
||||
|
||||
---
|
||||
|
||||
## Sparse matrix formats
|
||||
|
||||
Motivated by real k-mer-family data being extremely row-sparse (most rows —
|
||||
k-mers — present in only a handful of genome columns out of dozens to
|
||||
thousands): a row-major, deduplicated alternative to the dense column-major
|
||||
formats above, built once per layer like the rest of the pipeline (never the
|
||||
whole multi-billion-row index at once). Implements the exact-duplicate
|
||||
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
|
||||
Structures for Collections of Sets" (2025,
|
||||
`biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf`) — a plain dedup dictionary, not their full subset-containment
|
||||
hierarchy.
|
||||
|
||||
### `PersistentSparseBitMatrix`
|
||||
|
||||
Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row
|
||||
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
|
||||
(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width
|
||||
array for multi-genome rows (`dict_id`, `ceil(log2(n_distinct_multi_sets))`
|
||||
bits — kept apart from the singleton array because `n_distinct_multi_sets`
|
||||
can be large in absolute terms even when multi-genome rows are a small
|
||||
*fraction* of all rows, and a shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of distinct
|
||||
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
|
||||
blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary,
|
||||
runtime-parameterised bit width, width `0` included — needed once an
|
||||
`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec`
|
||||
(rank1/rank0/select1 on top of the crate's existing `count_ones`, via
|
||||
`common_traits::SelectInWord`), `EliasFano` (composes the two). `BinaryMatrix`
|
||||
(`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, `count_ones`) is the
|
||||
trait unifying dense and sparse at the call sites that need both
|
||||
interchangeably; column-oriented methods (`col`, `col_view`, the
|
||||
`partial_*_dist_matrix` family) stay dense-only — `Sparse`/`Implicit` panic
|
||||
there (see the `PersistentBitMatrix::open` note above).
|
||||
|
||||
Measured on real data (one layer, 30.2M rows, 91 genomes):
|
||||
|
||||
| | dense | sparse | ratio |
|
||||
|---|---|---|---|
|
||||
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
|
||||
| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction |
|
||||
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) |
|
||||
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** — the `--shannon`/family-lookup access shape |
|
||||
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit |
|
||||
|
||||
The row-access wins (compactness was the stated design goal, not speed) come
|
||||
from dense's genome-major layout scattering a single row read across a much
|
||||
bigger file — costlier than sparse's rank/select/varint decode once the file
|
||||
is this much smaller. Column access is the flip side of the same layout
|
||||
choice; `nonzero_iter` (below) is what removes it from the hot path that
|
||||
actually matters (`query`).
|
||||
|
||||
Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open`
|
||||
re-derived its low-bits width from the persisted low-vector's own width
|
||||
byte; the width-`0` case was built with a dummy 1-bit placeholder (the
|
||||
builder used to reject true width `0`), silently doubling every reopened
|
||||
value. Fixed by making `PersistentFixedIntVec` genuinely support width `0`
|
||||
(no storage, `get` always `0`). (2) An empty row (cardinality 0) used to be
|
||||
recorded as a singleton at genome `0`, indistinguishable on read-back from a
|
||||
*real* singleton at genome `0`. Fixed by routing cardinality-0 rows through
|
||||
the dictionary path (a genuine empty entry).
|
||||
|
||||
Still deferred: full Alanko et al. subset-hierarchy compression (only the
|
||||
exact-duplicate case is built), a sparse `PersistentCompactIntMatrix`
|
||||
variant beyond what's described below, BRWT-style column-correlation
|
||||
exploitation, and a row-major rewrite of
|
||||
`partial_jaccard_dist_matrix`/`partial_hamming_dist_matrix`/etc. as a
|
||||
per-row co-occurrence accumulation (`O(Σ_rows k̄²)`) so `obikindex`'s
|
||||
`--metric` distance-matrix path stops paying the column-access penalty on
|
||||
`Sparse` — needs its own design pass (how it plugs into
|
||||
`BitPartials`/`ColumnWeights` so both matrix types keep serving `--metric`),
|
||||
not just "port the loop".
|
||||
|
||||
### `nonzero_iter` — the one traversal per format
|
||||
|
||||
`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a>`
|
||||
(item: index into `slots`, column, value) is the single foundational
|
||||
per-format traversal primitive every consumer builds on — `fill_sub_matrix`
|
||||
and a `for_each`-style callback are both trivial, free consumers of it
|
||||
(`.for_each(f)` is `Iterator::for_each`; `fill_sub_matrix` drains the
|
||||
iterator and scatters into `out[][]`), instead of two independently
|
||||
hand-maintained traversals that can silently diverge. `Box<dyn Iterator>`,
|
||||
not a bare `impl Iterator`: `Columnar`/`Packed`/`Sparse`/`Implicit` are
|
||||
genuinely different concrete types and the method isn't on a
|
||||
dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix`
|
||||
elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per
|
||||
`nonzero_iter` call, not per cell.
|
||||
|
||||
Per-format implementation: `PersistentSparseBitMatrix` decodes natively,
|
||||
buffered one row at a time (`std::iter::from_fn` over the existing
|
||||
`for_each_genome_in_row`, no `n_cols`-wide allocation ever). `Columnar`/
|
||||
`Packed` loop columns, collecting each column's hits via
|
||||
`BitSliceView::nonzero_among_sorted`/`IntSliceView::nonzero_among_sorted`
|
||||
(a `filter`/`filter_map` over the vector-level `get()`, no new state
|
||||
machine — `std`'s own `Filter` iterator already is one) into one `Vec` via
|
||||
`.extend()` (not `flat_map`: a `flat_map` closure can't lazily return
|
||||
something borrowing its own captured sort permutation across separate calls
|
||||
without either boxing per-column or fighting the borrow checker — eager
|
||||
collection sidesteps it at zero extra cost, since `fill_sub_matrix` already
|
||||
fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))`).
|
||||
|
||||
This closed a real dispatch bug by construction, not by patching it:
|
||||
`PersistentBitMatrix::fill_sub_matrix`/`sub_matrix` used to reimplement a
|
||||
naive per-`(column, slot)` loop instead of delegating to
|
||||
`PersistentSparseBitMatrix`'s own efficient native method for `Self::Sparse`
|
||||
— two hand-written traversals for the same format, free to drift apart, and
|
||||
they had (`obikphylo`'s own cache layer had independently built a parallel
|
||||
enum wrapper that *did* dispatch correctly, evidence the bug was worked
|
||||
around rather than fixed at the source). With `fill_sub_matrix`/`sub_matrix`
|
||||
rewritten to drain `nonzero_iter`, there is only one traversal per format
|
||||
left to get right. `KmerPartition::query_partition_with`
|
||||
(`obikindex/src/partition/query_layer.rs`) — previously column-major
|
||||
(`for g { for slot { col_value } }`, which for `Sparse` rebuilt a full row
|
||||
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
|
||||
motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)`
|
||||
call per layer, format-agnostic. Dense/sparse query performance parity
|
||||
confirmed on real data (previously sparse 30-50% slower than dense,
|
||||
reproducibly; now within ~1-3% either way — noise level).
|
||||
|
||||
### `PersistentSparseCompactIntMatrix`
|
||||
|
||||
The count-matrix analogue, built on `PersistentSparseBitMatrix` as its
|
||||
"which columns are non-zero" support, values **not** deduplicated (unlike
|
||||
the bit side, two rows can share the same non-zero column set — same
|
||||
`dict_id` — while carrying different counts, so the dict-multiplicity
|
||||
shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry
|
||||
over). `CountPartials` here: a single row-major pass (`row_major_pairwise`
|
||||
in `sparse_intmatrix.rs`, decodes each row once via
|
||||
`for_each_cell_in_row`, nests over that row's own co-present columns) —
|
||||
`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)`
|
||||
column-pair rescan. `min(a,b)`-based kernels (`bray`, `relfreq_bray`) need
|
||||
no correction — they vanish when either side is absent. `a·b` and `√(a·b)`
|
||||
kernels (`euclidean`/`relfreq_euclidean`/`hellinger`) do need one, since
|
||||
`(a-0)² = a² ≠ 0` unlike the `min`-based formulas — reconstructed from
|
||||
per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`.
|
||||
`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard`
|
||||
(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form
|
||||
(every `u32` is `≥ 0`).
|
||||
|
||||
Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in
|
||||
`BitPartials for PersistentSparseBitMatrix` (present since that type's own
|
||||
introduction, never caught because no test had compared `Sparse`'s raw
|
||||
`partial_*` output against dense on real data — only the diagonal-blind
|
||||
`jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested),
|
||||
fixed at the source:
|
||||
|
||||
1. `partial_jaccard`'s diagonal used to be `(0, 2×col_weights[i])` instead
|
||||
of a genuine self-comparison `(col_weights[i], col_weights[i])` —
|
||||
`col_weights_and_pair_counts`'s `inter` never pairs a column with itself
|
||||
by construction, so the diagonal needs its own case.
|
||||
2. `partial_hamming`'s off-diagonal formula itself was wrong: it computed
|
||||
`total - union` (rows where *neither* column is present) instead of the
|
||||
actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]`
|
||||
(symmetric-difference size) — the two only coincide when
|
||||
`col_weights[i] + col_weights[j] == total`, which small/synthetic test
|
||||
data can easily satisfy by accident.
|
||||
|
||||
## Aggregation traits (matrix level)
|
||||
|
||||
### ColumnWeights
|
||||
|
||||
+1
-1
@@ -60,7 +60,7 @@ nav:
|
||||
- Architecture:
|
||||
- Sequences: architecture/sequences/invariant.md
|
||||
- Kmer index: architecture/index_architecture.md
|
||||
- Sibling annex (discussion): architecture/siblings.md
|
||||
- Sibling annex: architecture/siblings.md
|
||||
- NUMA-aware worker pools: architecture/numa_worker_pools.md
|
||||
- NUMA-aware partition runner: architecture/numa_partition_runner.md
|
||||
|
||||
|
||||
Reference in new issue
Block a user