Push zunrplorkwkt #70
@@ -1367,6 +1367,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1652,6 +1663,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -2582,6 +2604,98 @@ mismatches. The dense/sparse performance gap is gone — previously sparse
|
||||
systematic gap. <code>pack --sparse</code>'s claimed query win isn't confirmed
|
||||
outright by this (sparse should arguably now <em>beat</em> dense on truly sparse
|
||||
real data, not just tie), but the pathological regression is fixed.</p>
|
||||
<h2 id="persistentcompactintmatrixsparse-implemented-2026-08-26"><code>PersistentCompactIntMatrix::Sparse</code> — implemented (2026-08-26)</h2>
|
||||
<p>Closes the gap flagged throughout this document ("no sparse count format
|
||||
exists yet", <code>traits.rs:9-12</code>'s "Explicitly deferred"): <code>obicompactvec</code>
|
||||
already had <code>PersistentSparseCompactIntMatrix</code> (row-major, built on top of
|
||||
<code>PersistentSparseBitMatrix</code> as its "which columns are non-zero" support,
|
||||
values <em>not</em> deduplicated — see that struct's own doc comment), but it was
|
||||
never wired into <code>PersistentCompactIntMatrix</code>, the dense-dispatching enum
|
||||
every real consumer (<code>TypedLayer<PersistentCompactIntMatrix></code>,
|
||||
<code>KmerLayer::Count</code>) actually holds. Concretely: <code>kmer_index.rs::
|
||||
pack_matrices(sparse=true)</code> already called <code>pack_sparse_compact_int_matrix</code>
|
||||
on every layer's <code>counts/</code> — but <code>PersistentCompactIntMatrix::open</code> had no
|
||||
code path back to what that just wrote, so a <code>Count</code> layer became
|
||||
unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
|
||||
anyone ran <code>pack --sparse</code> on an index with count layers. Root cause, not a
|
||||
workaround: add the missing <code>Sparse</code> variant.</p>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Enum + dispatch</strong> (<code>intmatrix.rs</code>): <code>PersistentCompactIntMatrix::Sparse
|
||||
(PersistentSparseCompactIntMatrix)</code>, detected in <code>open</code>/<code>detect_storage</code>
|
||||
via a <code>singleton_values.pciv</code> marker (mirrors <code>PersistentBitMatrix</code>'s own
|
||||
<code>sparse_meta.json</code> check), reported via <code>storage_kind()</code>. <code>col</code>/
|
||||
<code>col_view</code>/<code>col_persist</code> panic/<code>Unsupported</code> on <code>Sparse</code>, same convention
|
||||
as the bit side. <code>sub_matrix</code>/<code>fill_sub_matrix</code> and <code>nonzero_iter</code>
|
||||
unified the same way <code>PersistentBitMatrix</code>'s already are (drain
|
||||
<code>nonzero_iter</code>, one traversal per format — see "Implemented
|
||||
(2026-08-20)" above); <code>nonzero_iter</code> had to become <code>Box<dyn Iterator<...>></code>
|
||||
for the same reason (<code>Columnar</code>/<code>Packed</code>/<code>Sparse</code> are different concrete
|
||||
types). No change needed in <code>obikindex</code> at all — <code>KmerLayer::Count</code>
|
||||
already only ever holds <code>TypedLayer<PersistentCompactIntMatrix></code>, so the
|
||||
enum absorbing <code>Sparse</code> fixes the unreadable-layer bug for free, same as
|
||||
<code>PersistentBitMatrix::Sparse</code> already did on the presence side.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong><code>CountPartials</code>, non-naive</strong> (<code>sparse_intmatrix.rs</code>): unlike
|
||||
<code>PersistentSparseBitMatrix</code>'s dict-driven <code>col_weights_and_pair_counts</code>,
|
||||
values here aren't deduplicated (two rows can share the same non-zero
|
||||
column set via the same <code>dict_id</code> while carrying different counts), so
|
||||
the "weight by how many rows share a dict entry" shortcut doesn't carry
|
||||
over. What does: a single row-major pass (<code>row_major_pairwise</code>, decodes
|
||||
each row once via <code>for_each_cell_in_row</code>, nests over that row's own
|
||||
co-present columns) — <code>O(Σ k̄²)</code> over populated rows instead of the naive
|
||||
<code>O(n_cols² × n)</code> column-pair rescan, same complexity class as the bit
|
||||
side minus the dict multiplicity discount. Kernels used: <code>min(a,b)</code>
|
||||
(bray, relfreq-bray — both vanish when either side is absent, so no
|
||||
correction needed), <code>a·b</code> and <code>√(a·b)</code> (euclidean/relfreq-euclidean and
|
||||
hellinger — these <em>do</em> need a correction, reconstructed from per-column
|
||||
marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>, since <code>(a-0)² = a² ≠ 0</code> unlike
|
||||
the <code>min</code>-based formulas). <code>threshold_jaccard(1)</code> shortcuts straight to
|
||||
<code>support</code>'s own <code>BitPartials::partial_jaccard</code> (threshold 1 is exactly
|
||||
presence); <code>threshold_jaccard(0)</code> is closed-form (every <code>u32</code> is <code>≥ 0</code>).</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Two pre-existing bugs found and fixed while wiring the <code>threshold==1</code>
|
||||
shortcut</strong> (<code>bitmatrix/sparse.rs</code>, <code>BitPartials for
|
||||
PersistentSparseBitMatrix</code>, present since the 2026-08-15 implementation
|
||||
above, never caught because no test compared <code>Sparse</code>'s raw <code>partial_*</code>
|
||||
output against dense on real data — only the diagonal-blind
|
||||
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested):</p>
|
||||
</li>
|
||||
<li><code>partial_jaccard</code>'s diagonal was <code>(0, 2×col_weights[i])</code> instead of a
|
||||
genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
|
||||
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with
|
||||
itself by construction.</li>
|
||||
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: <code>total -
|
||||
union</code> (count of rows where <em>neither</em> column is present) instead of
|
||||
the actual Hamming distance <code>col_weights[i] + col_weights[j] -
|
||||
2×inter[i,j]</code> (symmetric-difference size). Only coincides with the
|
||||
correct value when <code>col_weights[i] + col_weights[j] == total</code>, so
|
||||
small/synthetic test data could easily have hidden it.</li>
|
||||
</ul>
|
||||
<p>Neither surfaced through <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code>
|
||||
(both explicitly zero their own diagonal at finalisation, and the
|
||||
off-diagonal <code>partial_hamming</code> bug had gone untested against dense
|
||||
entirely) — only visible to a caller of the raw <code>partial_*</code> methods
|
||||
directly, which is exactly what <code>partial_threshold_jaccard(1)</code>'s new
|
||||
shortcut became. Fixed at the source, not patched around at the call
|
||||
site; regression test added:
|
||||
<code>tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal</code>.</p>
|
||||
<ul>
|
||||
<li><strong>Tests</strong>: <code>tests::intmatrix::sparse_roundtrip_matches_columnar</code>/
|
||||
<code>sparse_roundtrip_from_packed</code> (the <code>open</code>-dispatch fix, both build
|
||||
paths); <code>tests::intmatrix::sparse_count_partials_match_dense</code> (all six
|
||||
<code>CountPartials</code> formulas, thresholds 0/1/2/3, against <code>Columnar</code> on
|
||||
asymmetric-presence data — this is what caught the diagonal gap in the
|
||||
int side's own new code before it shipped, the same way it exposed the
|
||||
two pre-existing bit-side bugs above); <code>obikindex</code>'s
|
||||
<code>count_layer_transparently_reads_sparse_after_pack</code> — the actual
|
||||
end-to-end regression test for the original "layer unreadable after
|
||||
<code>pack --sparse</code>" bug, built → packed sparse → reopened, compared against
|
||||
the pre-pack dense read. <code>cargo test -p obicompactvec -p obikindex</code>:
|
||||
green, no regressions (180 + 12 tests).</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1168,7 +1168,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</code></pre></div>
|
||||
<hr/>
|
||||
<h2 id="matrix-types">Matrix types</h2>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has two variants (<code>Columnar</code>, <code>Packed</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -1185,12 +1185,12 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Packed</code></td>
|
||||
<td>single <code>matrix.pbmx</code> mmap file</td>
|
||||
<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
|
||||
<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Sparse</code> (bit only)</td>
|
||||
<td><code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major</td>
|
||||
<td><code>Sparse</code></td>
|
||||
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
||||
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -1200,7 +1200,8 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
|
||||
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||
|
||||
@@ -1213,6 +1213,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1719,6 +1835,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1853,6 +2085,21 @@ The underlying module boundary and its rationale (Layer tier / Partition
|
||||
tier / Index tier, each depending only downward) are unchanged; only the
|
||||
crate-vs-module packaging changed. See <a href="../layer_tier/">obikindex::layer</a>
|
||||
for the current module doc.</p>
|
||||
<p><strong>Superseded, second event, same day (2026-08-21):</strong> <code>obikpartitionner</code>
|
||||
and <code>obikderep</code>, the two algorithm crates, are also gone — but unlike
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> above, they were <strong>not</strong> folded into
|
||||
<code>obikindex</code>. They were first (mistakenly) merged into <code>obikindex</code> as an
|
||||
<code>algorithms</code> submodule, then corrected into a new sibling crate,
|
||||
<strong><code>obikindexer</code></strong>, holding <code>obikindexer::algorithms::{partitionner,
|
||||
dereplicator}</code> and depending on <code>obikindex</code> — never the reverse, same
|
||||
dependency direction <code>obikpartitionner</code>/<code>obikderep</code> already had. Read
|
||||
<code>obikpartitionner::X</code> as <code>obikindexer::algorithms::partitionner::X</code> and
|
||||
<code>obikderep::X</code> as <code>obikindexer::algorithms::dereplicator::X</code> throughout
|
||||
what follows. The distinction the mistake surfaced, worth keeping: data
|
||||
crates (<code>obikindex</code>, holding the <code>index</code>/<code>partition</code>/<code>layer</code> model) merge
|
||||
naturally into one crate as submodules; algorithm crates that operate on
|
||||
that model from outside stay separate, so the dependency only ever runs
|
||||
one way.</p>
|
||||
<p>Status (2026-08-20, latest pass): (1) done — <code>obilayeredmap::Layer</code>
|
||||
exists, <code>Mat</code> is gone. (1b) done — <code>Layer::Empty</code>, the first non-ready
|
||||
state, added (panics on every read method). (2a) done — the
|
||||
@@ -1872,7 +2119,24 @@ turned out to be bigger than <code>KmerPartition</code> alone: <code>Layer</code
|
||||
constructors don't self-name either. Full redesign of both, agreed in
|
||||
detail, session ended (budget) before implementation — see "(5) design
|
||||
agreed" below; <strong>read it before touching <code>KmerPartition</code>/<code>Layer</code>
|
||||
signatures</strong>, the shape is fully specified. Earlier mix-up, for
|
||||
signatures</strong>, the shape is fully specified. (6) done — <code>Counter</code>, a third
|
||||
algorithm, extracted from <code>PartitionRouter</code> the same way <code>Dereplicator</code>
|
||||
was in (4). (7) done — <code>LayerBuilder</code>, the fourth and last pipeline
|
||||
algorithm; the indexing pipeline is now fully decomposed into
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator, counter,
|
||||
layer_builder}</code>. (8) design agreed, item 1 done in (9) —
|
||||
<code>obikindexer::extensions::PrivateBuilder</code>, private, six construction-only
|
||||
<code>KmerIndex</code> methods moved out. (10) done, same session — <code>obikindex::
|
||||
IndexBuilder</code>, public, the four maintenance methods
|
||||
(<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>)
|
||||
shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>. Item 2 from (8)
|
||||
(<code>obikalgorithm::Algorithm</code>) done in (12) — new crate, <code>type Output</code> +
|
||||
<code>fn run(&mut self) -> SKResult<Self::Output></code>, <code>on_progress</code> moved off
|
||||
<code>run()</code>'s signature entirely into a per-algorithm <code>.on_progress(...)</code>
|
||||
setter. Note: <code>Layer</code> renamed
|
||||
<code>KmerLayer</code> (2026-08-21, outside this conversation). (5) itself still not
|
||||
implemented, still first on the "order of remaining work" list. Earlier
|
||||
mix-up, for
|
||||
context: an earlier
|
||||
version of this doc used the name <code>KmerPartition</code> (singular) for what was
|
||||
actually the <em>collection</em> type (later renamed <code>KmerPartitions</code>, later
|
||||
@@ -2515,6 +2779,622 @@ itself or a new crate.</p>
|
||||
examples were judged not enough to be sure of the shape (<code>Fn+Sync</code> vs
|
||||
<code>FnMut</code> callback bound already diverged between the two that exist).</li>
|
||||
</ol>
|
||||
<h2 id="6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator">(6) done (2026-08-21): <code>Counter</code> — third algorithm, extracted the same way as <code>Dereplicator</code></h2>
|
||||
<p>Between (5) and this, the user did a session of their own crate
|
||||
restructuring (see the two "Superseded" notes at the top of this file):
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> folded into <code>obikindex</code> as submodules
|
||||
(<code>obikindex::partition</code>, <code>obikindex::layer</code>), and <code>obikpartitionner</code>/
|
||||
<code>obikderep</code> merged into one sibling crate, <code>obikindexer</code>, holding
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator}</code>. (5)'s design
|
||||
(<code>Layer</code>/<code>KmerPartition</code> self-naming by number, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code> → <code>&</code> fix) was <strong>not</strong> part of that — pure crate/module packaging,
|
||||
confirmed by reading the actual code (<code>Layer::open</code>/<code>create</code> still take an
|
||||
external <code>dir: &Path</code>, <code>KmerPartition</code> still eagerly opens all layers,
|
||||
<code>PartitionRouter</code> still holds <code>&mut KmerIndex</code>). (5) remains exactly as
|
||||
specified, not yet implemented.</p>
|
||||
<p>This step: <code>count_kmer</code> (still living on <code>PartitionRouter</code>, per (4)'s own
|
||||
"still not done" note) extracted into <code>obikindexer::algorithms::counter::
|
||||
Counter</code>, mirroring <code>Dereplicator</code> exactly — third data point for the
|
||||
eventual <code>obikalgorithm</code> trait, still not extracted (still only 3 examples
|
||||
with 2 different callback bounds; holding off per (5)'s "order of
|
||||
remaining work").</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_partial</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_partial</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span><span class="w"> </span><span class="c1">// setter, mirrors PartitionRouter's style; defaults to false</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="n">KmerSpectrum</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Same shape as <code>Dereplicator</code> throughout: <code>Fn(Progress) + Sync</code> (not
|
||||
<code>FnMut</code>) since counting is also a parallel <code>par_iter</code> over partitions, an
|
||||
<code>AtomicU64</code> position counter incremented from inside the parallel closure
|
||||
so progress reports arrive in real time rather than bursting at the end
|
||||
once <code>.collect()</code> finishes, <code>total: Some(n_partitions)</code> (known up front).
|
||||
<code>KmerSpectrum</code> (the <code>{f0, f1, counts}</code> aggregate) moved from
|
||||
<code>partitionner::router</code> to <code>counter</code>, since it's <code>Counter::run</code>'s return
|
||||
value now, not <code>PartitionRouter</code>'s. <code>count.rs</code>/<code>kmer_sort.rs</code> moved
|
||||
verbatim from <code>partitionner/</code> to <code>counter/</code> (unchanged bodies — only
|
||||
<code>count_kmer</code> itself, <code>KmerSpectrum</code>, and the imports they pulled in were
|
||||
removed from <code>router.rs</code>).</p>
|
||||
<p>One divergence from <code>Dereplicator</code>: a <code>keep_partial</code> setter exists (no
|
||||
equivalent on <code>Dereplicator</code>, which has no setters at all) — a real,
|
||||
already-present parameter (<code>keep_intermediate</code> at the CLI), not a
|
||||
speculative addition.</p>
|
||||
<p><code>count_kmer</code>'s three former callers (<code>obikmer::cmd::index</code>, <code>obikphylo</code>'s
|
||||
test harness, <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test helper) all updated to <code>Counter::new(&idx).
|
||||
run(...)</code> — the last one simplified further: it used to read back
|
||||
<code>kmer_spectrum_raw.json</code> from disk after calling <code>count_partition</code>
|
||||
directly (white-box), now it just uses the <code>KmerSpectrum</code> <code>Counter::run</code>
|
||||
already returns.</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus an end-to-end CLI smoke test
|
||||
against real FASTA data (scatter → dereplicate → count → index-build →
|
||||
query) — required every time per (3)'s lesson, and it earned its keep
|
||||
again: the very first smoke-test query returned zero matches, which
|
||||
looked like a regression until traced to the query sequence itself being
|
||||
low-complexity ("GGCCCCCCACG", six same-base runs) and rejected by
|
||||
<em>query's own</em> default entropy threshold — nothing to do with this change.
|
||||
Re-tested with a different substring, confirmed working (kmer found,
|
||||
count matched the index).</p>
|
||||
<p>Still not done: (5) (<code>Layer</code>/<code>KmerPartition</code> redesign, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code>→<code>&</code>), the future cache crate, <code>build_layers</code> (still a <code>KmerIndex</code>
|
||||
inherent method, not an algorithm), and <code>obikalgorithm</code> itself.</p>
|
||||
<h2 id="7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm">(7) done (2026-08-21): <code>LayerBuilder</code> — fourth and last pipeline algorithm</h2>
|
||||
<p>Closes out the indexing pipeline: <code>build_layers</code>/<code>build_index_layer</code>
|
||||
(the last stage still living as <code>KmerIndex</code> inherent methods, flagged as
|
||||
inconsistent since (6)) extracted into <code>obikindexer::algorithms::
|
||||
layer_builder::LayerBuilder</code>, same two-phase shape as the other three.</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">min_abundance</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">max_abundance</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_intermediate</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">min_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">max_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_intermediate</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">FnMut</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="p">;</span><span class="w"> </span><span class="c1">// returns total kmers built</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p><strong>Different from all three prior extractions in one respect, deliberately
|
||||
not "fixed" to match them</strong>: the actual per-partition construction logic
|
||||
(De Bruijn graph from dereplicated superkmers + provisional counts →
|
||||
unitigs → MPHF → matrix) stayed put as <code>KmerIndex::build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> (both already <code>pub</code>) — not moved into
|
||||
<code>obikindexer</code>. Checked first: unlike <code>dereplicate_partition</code>/
|
||||
<code>count_partition</code> (which only ever had one caller), <code>build_index_layer</code>
|
||||
depends on several <code>obikindex</code>-internal helpers (<code>graph_pipeline::
|
||||
{write_graph_as_unitigs, materialize_layer}</code>, <code>common::olm_to_sk</code>) that
|
||||
are <code>pub(crate)</code> and shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>'s own
|
||||
layer-construction paths — moving <code>build_index_layer</code> out would have
|
||||
meant either exporting that internal surface just for this one algorithm
|
||||
or duplicating it. Neither was needed: <code>build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> were <em>already</em> public <code>KmerIndex</code> methods, so
|
||||
<code>LayerBuilder</code>'s job is purely the orchestration around them (scheduling,
|
||||
config, progress) — the exact same "algorithm calls already-public
|
||||
<code>KmerIndex</code> primitives" shape <code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>
|
||||
already have, just at a coarser grain for this one stage. This is the
|
||||
"is the producer's API actually deficient?" check from
|
||||
[[feedback_no_spaghetti_petits_pois]] applied and answered "no" — not
|
||||
skipped.</p>
|
||||
<p><strong>Two more real divergences, both forced by <code>PartitionRunner</code>, not
|
||||
arbitrary:</strong>
|
||||
- Uses <code>obikindex::PartitionRunner</code> (NUMA-aware scheduler, already
|
||||
<code>pub use</code>d from <code>obikindex</code>) instead of plain <code>rayon::into_par_iter</code>
|
||||
like <code>Dereplicator</code>/<code>Counter</code> — matches what <code>build_layers</code> already used
|
||||
before extraction; this stage is more CPU/memory-intensive per partition
|
||||
(graph construction) than scatter/dereplicate/count.
|
||||
- Callback bound is <code>FnMut(Progress) + Send</code> — a third variant, not
|
||||
matching either prior shape. <code>PartitionRunner::run</code>'s <code>on_done</code> is
|
||||
invoked from its own single controller thread (never concurrently, so
|
||||
no <code>Sync</code> needed, unlike <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn + Sync</code>), but
|
||||
that controller thread is itself <code>std::thread::scope</code>-spawned, so the
|
||||
closure still has to be <code>Send</code> to cross into it — caught immediately by
|
||||
the compiler (<code>cannot be sent between threads safely</code>) when <code>Send</code> was
|
||||
first omitted, not a design guess. <code>obikalgorithm</code>'s eventual shared
|
||||
trait now has three real callback-bound data points to reconcile
|
||||
(<code>FnMut</code> alone for <code>PartitionRouter::run</code>'s sequential loop, <code>FnMut +
|
||||
Send</code> here, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code>'s <code>rayon</code>
|
||||
<code>par_iter</code>), not two.</p>
|
||||
<p><code>KmerIndex::build_layers</code> deleted outright (<code>KmerIndex</code> stays a pure data
|
||||
structure — no compute orchestration methods, consistent with <code>dereplicate</code>/
|
||||
<code>count_kmer</code>'s removal in (4)/(6)). New <code>KmerIndex::mark_indexed()</code> added,
|
||||
symmetric to <code>mark_scattered</code>/<code>mark_counted</code>, replacing the inline
|
||||
<code>touch(SENTINEL_INDEXED)</code> that used to live inside <code>build_layers</code>.
|
||||
<code>Stage::start("index")</code>/<code>rep.push(...)</code> and the <code>progress_bar</code>/
|
||||
<code>"{n} total kmers indexed"</code> log line both moved to <code>cmd/index/mod.rs</code>,
|
||||
same pattern as (3)/(4)/(6) — <code>LayerBuilder</code> renders nothing itself, just
|
||||
reports <code>Progress</code>.</p>
|
||||
<p>All callers updated: <code>cmd/index/mod.rs</code> (Stage 3), <code>obikphylo</code>'s test
|
||||
harness (also gained a <code>mark_indexed()</code> call it was missing — harmless
|
||||
before since nothing checked <code>IndexState::Indexed</code> in that test, but now
|
||||
correct).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the end-to-end CLI smoke test
|
||||
(<code>scripts/smoke_test_index.sh</code>, built earlier specifically so this
|
||||
verification step is a one-liner from now on) — 870 kmers indexed, query
|
||||
round-trip confirmed, same numbers as (6).</p>
|
||||
<p><strong>The indexing pipeline is now fully decomposed</strong>: <code>obikindexer::
|
||||
algorithms::{partitionner, dereplicator, counter, layer_builder}</code>, each a
|
||||
<code>new</code>/(setters)/<code>run</code> algorithm operating on a <code>&KmerIndex</code> (or <code>&mut</code> for
|
||||
<code>PartitionRouter</code>, not yet fixed — see (5)), <code>KmerIndex</code> itself holding no
|
||||
pipeline-orchestration logic anymore. Still not done: (5), the future
|
||||
cache crate, <code>obikalgorithm</code> (now unblocked — three real callback-bound
|
||||
variants observed, worth revisiting whether a single trait can express
|
||||
all three or whether that's itself the answer: it can't, and the trait
|
||||
should not force it).</p>
|
||||
<h2 id="8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename">(8) design agreed, not yet implemented (2026-08-21): <code>obikalgorithm::Algorithm</code> trait + <code>obikindexer::extensions</code> — private/public extension-trait split, <code>KmerLayer</code> rename</h2>
|
||||
<p>Session note: <code>Layer</code> was renamed <code>KmerLayer</code> (user, outside this
|
||||
conversation, alongside other naming homogenisation with <code>KmerIndex</code>/
|
||||
<code>KmerPartition</code>) — every reference to <code>Layer</code> in this doc from before
|
||||
2026-08-21 means today's <code>obikindex::layer::KmerLayer</code>.</p>
|
||||
<h3 id="why-this-came-up">Why this came up</h3>
|
||||
<p>Verifying "does <code>cmd/index</code> now rest entirely on the algorithm structs"
|
||||
(it doesn't quite — see below) led to sorting <code>KmerIndex</code>'s own methods by
|
||||
a criterion the user was explicit is <strong>semantic, not mechanical</strong>: "les
|
||||
méthodes qui, sémantiquement, n'ont pas d'intérêt hors de la construction
|
||||
de l'index" (methods that have no semantic interest outside index
|
||||
construction) — not "methods only called from <code>cmd/index</code> today," which
|
||||
a grep could answer but would miss methods construction-adjacent code
|
||||
elsewhere (<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>) also depends on for the
|
||||
same reason.</p>
|
||||
<p><strong>Checked, not assumed</strong> (grepped every call site before classifying):</p>
|
||||
<ul>
|
||||
<li><strong>Construction-only, real candidates for a private extension trait</strong>:
|
||||
<code>KmerIndex::{mark_scattered, mark_counted, mark_indexed, write_spectrum,
|
||||
build_index_layer, remove_build_artifacts, clear_output_for_create,
|
||||
create_skeleton, finalize_indexed, state}</code>. The last four are called
|
||||
from <code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> too (as
|
||||
precondition checks — "is my source <code>Indexed</code>?" — or shared
|
||||
skeleton/finalize machinery), not just from the 4-stage pipeline — so
|
||||
this extension trait's scope is "construction of any kind," not
|
||||
narrowly "the initial build pipeline."</li>
|
||||
<li><strong>Looked construction-only by name, checked, and kept on <code>KmerIndex</code></strong>:
|
||||
<code>layer_unitigs_path</code> (unitigs are the only way to recover a built
|
||||
index's kmer sequences — read by <code>rebuild_layer.rs</code> and others, well
|
||||
beyond construction — see [[project_unitigs_always_kept]]),
|
||||
<code>pack_matrices</code> (re-runnable maintenance on an already-finished index
|
||||
via <code>obikmer pack</code>, not just a pipeline step), <code>upgrade_layer_meta</code>
|
||||
(migration, runnable on any existing index at any time).</li>
|
||||
</ul>
|
||||
<h3 id="the-general-pattern-not-obikindexer-specific">The general pattern (not obikindexer-specific)</h3>
|
||||
<p><code>KmerIndex</code>/<code>KmerPartition</code>/<code>KmerLayer</code> stay generic, in <code>obikindex</code> —
|
||||
every domain-specific consumer crate gets to attach its own extension
|
||||
trait(s), of two kinds:</p>
|
||||
<ul>
|
||||
<li><strong>Private</strong> (<code>pub(crate)</code>, invisible outside the defining crate) — for
|
||||
plumbing only that crate's own algorithms need. <code>obikindexer</code> gets
|
||||
exactly one of these (see below); no public counterpart makes sense for
|
||||
it — "l'index est tellement central que le second trait n'a pas
|
||||
vraiment d'intérêt" for construction specifically: nothing external
|
||||
should ever want to call <code>mark_scattered</code> or <code>build_index_layer</code>.</li>
|
||||
<li><strong>Public</strong> — for a genuinely reusable domain extension. The user's own
|
||||
example, found while discussing this, not hypothetical: <code>obikindex/src/
|
||||
index/distance.rs</code> (phylogenetic distance metrics) is currently an
|
||||
<code>impl KmerIndex</code> block <strong>inside <code>obikindex</code> itself</strong> — under this
|
||||
principle it should be a public extension trait owned by <code>obikphylo</code>
|
||||
instead (distance metrics are a phylo concept, <code>obikindex</code> has no more
|
||||
business defining them than <code>obikindex::layer</code> has defining
|
||||
"family"/"minorant", the reasoning <code>SiblingLayerExt</code> already followed
|
||||
for <code>KmerLayer</code> — see <code>obikphylo/src/siblings/iter.rs</code>). <strong>Explicitly
|
||||
deferred</strong> — noted here so it isn't lost, not part of this round.</li>
|
||||
<li>The future cache-manager crate (still blocked on (5), see above) will
|
||||
add its own <strong>public</strong> extension trait mirroring part of <code>KmerIndex</code>'s/
|
||||
<code>KmerPartition</code>'s own read API in cached form (e.g. a cached
|
||||
<code>.partition(i)</code> that doesn't re-touch disk) — same pattern, third data
|
||||
point once built.</li>
|
||||
</ul>
|
||||
<h3 id="concretely-next-to-implement-two-items-in-order">Concretely, next to implement (two items, in order)</h3>
|
||||
<ol>
|
||||
<li><strong><code>obikindexer::extensions</code></strong> — a private (<code>pub(crate)</code>) extension
|
||||
trait, most likely named something like <code>IndexBuildExt</code> (final name
|
||||
not yet chosen), implemented for <code>KmerIndex</code>, carrying the ten methods
|
||||
listed above, moved out of <code>obikindex::index::{kmer_index,
|
||||
index_layer}</code>. Every algorithm in <code>obikindexer::algorithms::*</code> that
|
||||
currently calls <code>idx.mark_scattered()</code>/etc. keeps the same call syntax
|
||||
(extension trait methods are called the same way as inherent ones,
|
||||
just need the trait in scope) — <code>cmd/index/mod.rs</code> itself would need
|
||||
<code>use obikindexer::extensions::IndexBuildExt;</code> (or the module re-exports
|
||||
it) to keep compiling, since it's the one place outside <code>obikindexer</code>'s
|
||||
own algorithms that currently calls <code>mark_scattered</code>/<code>write_spectrum</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code> directly. <strong>Not yet decided</strong>: exact
|
||||
trait name, whether it's one trait or split further (e.g. sentinel
|
||||
marking vs. skeleton/finalize machinery), and whether <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> (not yet extracted into algorithms themselves) move
|
||||
onto it now too or keep calling the soon-to-be-inherent-no-longer
|
||||
methods some other way in the meantime — <strong>ask before implementing</strong>,
|
||||
this changes the blast radius significantly (4 more <code>obikindex</code>
|
||||
internal files depend on <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code>).</li>
|
||||
<li><strong><code>obikalgorithm::Algorithm</code> trait</strong> — new crate, the shared trait
|
||||
<code>obikpartitionner</code>→<code>obikindexer</code> merge (session start of 2026-08-21)
|
||||
and (6)/(7) were deliberately building toward, now with four real
|
||||
<code>new</code>/(setters)/<code>run</code> examples and three distinct callback-bound
|
||||
shapes to reconcile (plain <code>FnMut</code> for <code>PartitionRouter</code>, <code>FnMut +
|
||||
Send</code> for <code>LayerBuilder</code>, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code> —
|
||||
see (7)). Exact shape not yet drafted in this doc — do that as its own
|
||||
design pass before coding, same discipline as everything above.</li>
|
||||
</ol>
|
||||
<p>Both items: <strong>design only, nothing implemented yet</strong> — this section is
|
||||
the record to resume from, not a plan already executed.</p>
|
||||
<h2 id="9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented">(9) done (2026-08-21): <code>obikindexer::extensions::PrivateBuilder</code> — item 1 above, implemented</h2>
|
||||
<p>Renamed from <code>IndexBuilder</code> to <code>PrivateBuilder</code> immediately after (same
|
||||
session), freeing the name <code>IndexBuilder</code> for (10)'s public trait — read
|
||||
<code>IndexBuilder</code> below as <code>PrivateBuilder</code> throughout this section.</p>
|
||||
<p>Scoped down from (8)'s six-method list to the concrete set that's
|
||||
genuinely movable without further ripple — checked, not assumed, before
|
||||
writing anything:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_scattered</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_counted</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_indexed</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">write_spectrum</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">f0</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">f1</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">counts</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">BTreeMap</span><span class="o"><</span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="kt">u64</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">build_index_layer</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">min_ab</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="n">max_ab</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">with_counts</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="n">mode</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMode</span><span class="p">,</span><span class="w"> </span><span class="n">block_bits</span><span class="p">:</span><span class="w"> </span><span class="kt">u8</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">SKError</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">remove_build_artifacts</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">);</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>All six moved bodily out of <code>obikindex::index::{kmer_index, index_layer}</code>
|
||||
into <code>obikindexer::extensions</code> (new module, <code>pub(crate)</code>) — <code>index_layer.rs</code>
|
||||
is now empty and deleted outright.
|
||||
<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>
|
||||
stayed inherent on <code>KmerIndex</code>, per (8)'s reasoning: <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> — living <em>inside</em> <code>obikindex</code> itself — call them too,
|
||||
and <code>obikindex</code> can never depend on <code>obikindexer</code> to reach a trait defined
|
||||
there. Moving those four is real future work (extract
|
||||
merge/select/rebuild/reindex into algorithms first), not part of this
|
||||
step.</p>
|
||||
<p><strong>One new, small, deliberate API widening in <code>obikindex</code></strong>: <code>build_index_layer</code>
|
||||
depends on three helpers that were <code>pub(crate)</code> to <code>obikindex</code>
|
||||
(<code>graph_pipeline::{write_graph_as_unitigs, materialize_layer}</code>,
|
||||
<code>common::olm_to_sk</code>) — widened to <code>pub</code> (re-exported from <code>obikindex</code>'s
|
||||
crate root) so <code>obikindexer</code> could reach them. This is exactly the
|
||||
"enrich shared/lower-level APIs instead of ad hoc local code" call the
|
||||
project's own rules ask for, made explicitly rather than routed around:
|
||||
three functions, already generically written (no rewrite needed), now
|
||||
serve a second caller instead of being duplicated.</p>
|
||||
<p><strong>Why the trait had to be defined in <code>obikindexer</code>, not <code>obikindex</code></strong>:
|
||||
Rust's orphan rule — implementing a trait for a foreign type requires
|
||||
either the trait or the type to be local to the current crate. <code>KmerIndex</code>
|
||||
is foreign to <code>obikindexer</code>, so the trait must be the local half; if it
|
||||
were defined in <code>obikindex</code> instead, <code>pub(crate)</code> there would make it
|
||||
invisible to <code>obikindexer</code> too (crate-private means private to <em>that</em>
|
||||
crate, not "private except to one named dependent") — the opposite of
|
||||
what was wanted.</p>
|
||||
<p><strong>A real design decision made while wiring callers up, not a mechanical
|
||||
rename</strong>: <code>PrivateBuilder</code> being genuinely <code>pub(crate)</code> to <code>obikindexer</code>
|
||||
means <code>obikmer::cmd::index</code> (a different crate) can no longer call
|
||||
<code>mark_scattered</code>/<code>mark_counted</code>/<code>mark_indexed</code>/<code>write_spectrum</code> directly —
|
||||
it never could have, once privacy was real rather than aspirational. Each
|
||||
algorithm now marks its own completion as part of <code>run()</code>/<code>close()</code>
|
||||
instead of leaving it to the caller:
|
||||
- <code>PartitionRouter::close()</code> (not <code>run()</code>) calls <code>mark_scattered()</code> —
|
||||
<code>close()</code>, not <code>run()</code>, is the actual shared completion point between
|
||||
the file-driven <code>run()</code> path and the manual <code>write</code>/<code>write_batch</code>+
|
||||
<code>close()</code> path low-level callers (tests) use; putting it in <code>run()</code>
|
||||
alone would have silently skipped marking for every caller that never
|
||||
calls <code>run()</code>. <code>run()</code> already calls <code>self.close()</code> at its own end, so
|
||||
this covers both paths through one line, not two.
|
||||
- <code>Counter::run</code> calls <code>write_spectrum</code> then <code>mark_counted</code> before
|
||||
returning.
|
||||
- <code>LayerBuilder::run</code> calls <code>mark_indexed</code> before returning.</p>
|
||||
<p><code>cmd/index/mod.rs</code> lost all four direct calls (<code>mark_scattered</code>/
|
||||
<code>write_spectrum</code>/<code>mark_counted</code>/<code>mark_indexed</code>) — each stage's <code>if
|
||||
idx.state() < IndexState::X { ... }</code> block is now purely "run the
|
||||
algorithm," no separate bookkeeping call after it. Confirms, precisely
|
||||
this time (checked by re-reading the whole file, not assumed): <code>cmd/index</code>
|
||||
now rests on the four algorithms for every read/write of pipeline state
|
||||
except <code>KmerIndex::{exists, create, state, n_partitions}</code>, which are
|
||||
genuinely index-identity concerns, not construction bookkeeping — the
|
||||
original question this whole design pass started from.</p>
|
||||
<p>Same fix applied to <code>obikphylo</code>'s test harness (its four explicit
|
||||
<code>mark_*</code>/<code>write_spectrum</code> calls removed, relying on the algorithms now
|
||||
doing it themselves) — <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test needed no change (never called <code>mark_*</code> directly).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the CLI smoke test — 870
|
||||
kmers, same as (6)/(7).</p>
|
||||
<p>Still not done at the time of writing: item 2 from (8) (<code>obikalgorithm::
|
||||
Algorithm</code>), (5), the future cache crate, the <code>distance.rs</code> →
|
||||
<code>obikphylo</code> relocation (noted in (8), explicitly deferred), and
|
||||
extracting <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code> into algorithms.</p>
|
||||
<h2 id="10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session">(10) done (2026-08-21): <code>obikindex::IndexBuilder</code> — the public counterpart, same session</h2>
|
||||
<p>Immediate correction to (9): the private trait built there was renamed
|
||||
<code>PrivateBuilder</code> (freeing the name), and the four methods (9) had left
|
||||
inherent on <code>KmerIndex</code> — <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code> — got their own trait after all: <strong><code>IndexBuilder</code></strong>,
|
||||
public, defined in <code>obikindex</code> itself (not <code>obikindexer</code>):</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="p">:</span><span class="w"> </span><span class="nb">Sized</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">clear_output_for_create</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">force</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">create_skeleton</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">meta</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMeta</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">finalize_indexed</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">rep</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">Reporter</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">state</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">IndexState</span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>User's framing: these four are "maintenance", not "scientific computation
|
||||
on an index" — a different kind of non-generic-ness than (9)'s six
|
||||
(<code>mark_*</code>/<code>write_spectrum</code>/<code>build_index_layer</code>/<code>remove_build_artifacts</code>,
|
||||
exclusive to the 4-stage pipeline). Maintenance is used more broadly
|
||||
(<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>), so it gets a real, public trait —
|
||||
not folded back into <code>KmerIndex</code>'s inherent surface, and not private
|
||||
either.</p>
|
||||
<p><strong>Where it lives, and why that's not arbitrary</strong>: (9) needed the orphan
|
||||
rule to force its trait into <code>obikindexer</code>, to achieve genuine
|
||||
crate-private visibility. Here the requirement is the opposite:
|
||||
<code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> — the trait's own
|
||||
heaviest users — live <em>inside</em> <code>obikindex</code>. A trait they need to reach
|
||||
must be local to <code>obikindex</code> (or a crate <code>obikindex</code> itself depends on,
|
||||
which doesn't exist for this). So <code>IndexBuilder</code> lives in a new
|
||||
<code>obikindex/src/index/builder.rs</code>, <code>pub trait</code> (no orphan-rule tension at
|
||||
all here — both trait and type are local to the same crate), re-exported
|
||||
from <code>obikindex</code>'s crate root alongside <code>PrivateBuilder</code>'s sibling
|
||||
<code>obikindexer::extensions::PrivateBuilder</code> staying where it is. Two
|
||||
traits, two crates, two different reasons, not a contradiction.</p>
|
||||
<p><strong>Blast radius, all inside <code>obikindex</code> plus one external crate</strong>: every
|
||||
internal caller of these four methods needs the trait imported now that
|
||||
they're no longer inherent — <code>merge.rs</code>, <code>select.rs</code>, <code>rebuild.rs</code>,
|
||||
<code>reindex.rs</code> (<code>use crate::index::builder::IndexBuilder;</code>) and, externally,
|
||||
<code>obikmer::cmd::index::mod</code> (<code>use obikindex::IndexBuilder;</code>, for the three
|
||||
<code>idx.state() < IndexState::X</code> resumability checks). Call syntax at every
|
||||
site is unchanged (<code>KmerIndex::create_skeleton(...)</code>,
|
||||
<code>self.state()</code>) — only trait-in-scope requirements are new, which is
|
||||
exactly the point: same ergonomics, less surface baked into <code>KmerIndex</code>
|
||||
itself.</p>
|
||||
<p>Verification went one step further than (9): beyond
|
||||
<code>cargo check --workspace --all-targets</code> + <code>cargo test --workspace</code> +
|
||||
<code>scripts/smoke_test_index.sh</code> (all green, 870 kmers again), ran
|
||||
<code>obikmer merge</code> end to end on two freshly built indexes (exercises
|
||||
<code>clear_output_for_create</code>/<code>finalize_indexed</code> directly, the two methods
|
||||
<code>scripts/smoke_test_index.sh</code> itself never touches) — exit 0, <code>pack</code>
|
||||
stage completed. Test suite alone would not have caught a regression
|
||||
here: no existing test builds two real indexes and merges them through
|
||||
the CLI.</p>
|
||||
<p><code>KmerIndex</code> itself now carries only: identity/config accessors
|
||||
(<code>root_path</code>/<code>meta</code>/<code>kmer_size</code>/<code>n_bits</code>/<code>evidence_mode</code>/<code>genomes</code>/...),
|
||||
path resolution (<code>partition_dir</code>/<code>index_dir</code>/<code>layer_dir</code>/
|
||||
<code>partition_meta</code>/<code>n_layers</code>), and a few index-maintenance operations not
|
||||
yet sorted into either trait (<code>layer_unitigs_path</code>, <code>pack_matrices</code>,
|
||||
<code>upgrade_layer_meta</code> — see (8)'s "tested and discarded" list; still
|
||||
correctly inherent, not construction-only by the semantic criterion) —
|
||||
<code>create</code>/<code>open</code>/<code>exists</code> (identity, can't be trait methods needing <code>Self</code>
|
||||
before one exists) round that out.</p>
|
||||
<h2 id="11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files">(11) done (2026-08-21): <code>KmerIndex</code>/<code>IndexMeta</code> made fully stateless, <code>IndexState</code> moved off sentinel files</h2>
|
||||
<p>Triggered mid-discussion of <code>obikalgorithm::Algorithm</code> (still not started —
|
||||
see "Still not done" below): user asked why <code>PartitionRouter::new</code> still
|
||||
took <code>&mut KmerIndex</code> at all, and questioned whether <code>mark_scattered</code>
|
||||
belonged in the algorithm or in <code>cmd/index</code>. Investigation found <code>&mut</code>
|
||||
had become <em>newly</em> necessary since (9) — <code>mark_scattered</code> was mutating
|
||||
<code>self.meta.genomes</code> in memory so <code>Counter</code>'s later <code>write_spectrum</code> call
|
||||
(same <code>idx</code> instance) would see the derived label. User's resolution: the
|
||||
"disk is truth, stateless" principle already agreed for <code>KmerPartition</code>/
|
||||
<code>Layer</code> in (5) (still unimplemented for those two) should extend to
|
||||
<code>KmerIndex</code> itself — move <code>IndexState</code> (<code>Empty</code>/<code>Scattered</code>/<code>Counted</code>/
|
||||
<code>Indexed</code>) off the three sentinel files (<code>scatter.done</code>/<code>count.done</code>/
|
||||
<code>index.done</code>, detected by existence) and into a field of <code>index.meta</code>'s
|
||||
own JSON, so the <code>mark_*</code> calls become plain disk writes an algorithm can
|
||||
legitimately make on <code>&self</code> — no in-memory mutation left to protect.</p>
|
||||
<p><strong>Shape of <code>IndexMeta</code>, per the user's explicit spec</strong>: one JSON file per
|
||||
index (<code>index.meta</code>), one <code>IndexMeta</code> instance per index, held and
|
||||
returned as <code>Arc<IndexMeta></code> (not <code>&IndexMeta</code>) by <code>KmerIndex::meta()</code>.
|
||||
<code>config</code> (<code>kmer_size</code>/<code>minimizer_size</code>/<code>n_bits</code>/<code>with_counts</code>/<code>evidence</code>/
|
||||
<code>block_bits</code>) is fixed at construction, cached as a <code>pub</code> field (getter
|
||||
kept alongside, for symmetry) — "les champs constants restent des champs
|
||||
de la structure", read once, never re-read from disk. <code>genomes</code> and
|
||||
<code>state</code> are the opposite: no in-memory cache at all, every accessor
|
||||
(<code>genomes()</code>, <code>state()</code>) re-reads <code>index.meta</code> from disk, every mutator
|
||||
(<code>push_genome</code>/<code>rename_genome</code>/<code>set_genomes</code>/<code>set_state</code>/<code>mark_scattered</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code>) does a full read-modify-write of the same
|
||||
file. An internal <code>std::sync::RwLock<()></code> is held across each
|
||||
read-modify-write sequence (not just the write) so two callers sharing
|
||||
the same <code>Arc<IndexMeta></code> can't lose an update to each other — this is
|
||||
<em>not</em> a cross-process lock (that's <code>obisys::DirLock</code>, already held by
|
||||
<code>cmd/index</code> for the whole build); it only serialises access through one
|
||||
shared in-process instance.</p>
|
||||
<p><strong>Construction, chain-of-responsibility style, matching (5)'s pattern</strong>:
|
||||
<code>IndexMeta::create(&KmerIndex, config, genomes)</code> / <code>IndexMeta::open(&KmerIndex)</code>
|
||||
ask the index for its own root path rather than taking one directly. Since
|
||||
<code>KmerIndex::create</code> doesn't have a complete <code>KmerIndex</code> yet to hand in
|
||||
(it's what's being built), added lower-level <code>pub(crate)</code> path-based
|
||||
primitives <code>create_at(&Path, ...)</code> / <code>open_at(&Path)</code> that <code>KmerIndex::create</code>/
|
||||
<code>open</code> and <code>builder.rs</code>'s <code>create_skeleton</code> call directly, bypassing the
|
||||
convenience wrappers for that one bootstrap case.</p>
|
||||
<p><strong>The one deliberate exception</strong>: <code>select_in_place</code> and <code>reindex</code>
|
||||
genuinely rewrite <code>config</code> after an index already exists (output
|
||||
type/evidence mode changes in place) — contradicting "config never
|
||||
changes" for the general case. Resolved with a separate, explicitly
|
||||
rare-labelled <code>IndexMeta::rewrite_config(config, genomes)</code> (preserves
|
||||
<code>state</code>, overwrites everything else); callers refresh their own cached
|
||||
<code>Arc<IndexMeta></code> afterward (<code>self.meta = Arc::new(IndexMeta::open(self)?)</code>)
|
||||
since <code>IndexMeta</code> has no way to reach back into whichever <code>KmerIndex</code>
|
||||
holds it.</p>
|
||||
<p><strong>Consequence confirmed, not just hoped for</strong>: with <code>mark_scattered</code> no
|
||||
longer touching anything in memory, <code>PartitionRouter</code> genuinely never
|
||||
needs <code>&mut KmerIndex</code> — <code>PartitionRouter<'a> { index: &'a KmerIndex }</code>,
|
||||
<code>new(&'a KmerIndex)</code>. This is effectively the <code>PartitionRouter</code> half of
|
||||
(5)'s "order of remaining work" item done as a side effect; <code>KmerPartition</code>/
|
||||
<code>Layer</code> themselves are still unimplemented for (5).</p>
|
||||
<p><strong>Blast radius — much larger than (9)/(10), touched nearly every crate</strong>:
|
||||
every <code>.meta().genomes</code>/<code>.meta.genomes</code> field access became a fallible
|
||||
<code>.genomes()?</code> method call (<code>genomes</code> reads <code>io::Result<Vec<GenomeInfo>></code>
|
||||
now, not a field), and <code>.meta_mut()</code> was removed outright (no more direct
|
||||
field mutation from outside <code>IndexMeta</code>). Fixed across:
|
||||
- <code>obikindex</code> internals: <code>meta.rs</code>/<code>state.rs</code>/<code>kmer_index.rs</code>/<code>builder.rs</code>
|
||||
(full rewrites), <code>reindex.rs</code>/<code>select.rs</code> (switched to <code>rewrite_config</code>),
|
||||
<code>merge.rs</code> (heaviest single file — genome counts precomputed once per
|
||||
source into a <code>Vec<Vec<GenomeInfo>></code> up front rather than re-reading
|
||||
<code>index.meta</code> from disk repeatedly through the function, sentinel write
|
||||
replaced with <code>dst2.meta.mark_indexed()</code>), <code>stats.rs</code>, <code>distance.rs</code>,
|
||||
<code>dump.rs</code>, <code>predicate.rs</code> (its <code>IndexMeta</code>-inherent <code>matching_genome_indices</code>/
|
||||
<code>build_group_filter</code> now read genomes fresh internally), <code>mod.rs</code>/<code>lib.rs</code>
|
||||
(sentinel constant re-exports removed — <code>IndexState</code> no longer has
|
||||
<code>SENTINEL_*</code>/<code>detect()</code> at all).
|
||||
- <code>obikindexer::extensions::PrivateBuilder</code>: <code>mark_scattered</code> signature
|
||||
dropped <code>&mut self</code> → <code>&self</code>; the four <code>mark_*</code>/<code>write_spectrum</code> bodies
|
||||
became one-line delegations to <code>self.meta().mark_*()</code>.
|
||||
- <code>obikphylo::siblings</code>: <code>alignment.rs</code>/<code>cardinality.rs</code>/<code>distance.rs</code>/
|
||||
<code>entropy.rs</code>/<code>sankoff_bundle.rs</code>/<code>stats.rs</code>/<code>tests.rs</code> — all had
|
||||
<code>self.meta().genomes.len()</code>-shaped reads, mechanically fixed to
|
||||
<code>.genomes().map_err(OKIError::Io)?.len()</code> (tests: <code>.unwrap()</code>).
|
||||
- <code>obikmer::cmd::*</code>: <code>annotate</code> (rewrote its rename path to load genomes
|
||||
once, mutate the in-memory <code>Vec</code>, then <code>idx.meta().set_genomes(...)</code>
|
||||
instead of <code>meta_mut()</code>), <code>filter</code>/<code>pack</code>/<code>dump</code>/<code>unitig</code>/<code>merge</code>/<code>select</code>/
|
||||
<code>phylo</code> (fetch-once-then-use pattern for genome counts/labels),
|
||||
<code>utils/maintenance.rs</code> (<code>run_rename</code> now calls the pre-existing
|
||||
<code>IndexMeta::rename_genome</code>, dropping its own hand-rolled field mutation
|
||||
entirely), <code>index/mod.rs</code> (three <code>idx.state() < IndexState::X</code>
|
||||
resumability checks needed a fallible read — factored into a small
|
||||
<code>current_state(&KmerIndex) -> IndexState</code> helper rather than repeating
|
||||
the same <code>unwrap_or_else</code> three times), <code>query/*</code> (<code>emit_batch</code>'s
|
||||
signature changed from <code>&IndexMeta</code> to <code>&[GenomeInfo]</code>, and <code>genomes</code> is
|
||||
now fetched once in <code>run()</code> and threaded down through <code>process_chunk</code>
|
||||
as <code>Arc<Vec<GenomeInfo>></code> rather than re-reading <code>index.meta</code> from disk
|
||||
on every chunk — a deliberate deviation from the "always re-read"
|
||||
default, justified because this is a genuine per-chunk hot path, unlike
|
||||
every other call site touched in this pass).
|
||||
- One <code>&IndexMeta</code>-vs-<code>Arc<IndexMeta></code> argument-type mismatch pattern
|
||||
recurred at several CLI call sites (<code>build_filters</code>/<code>build_specs</code>/
|
||||
<code>emit_batch</code>'s original signature) — resolved via <code>Arc</code>'s deref
|
||||
coercion (<code>&idx.meta()</code> coerces to <code>&IndexMeta</code>) rather than changing
|
||||
every downstream signature to accept <code>Arc<IndexMeta></code>.</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and
|
||||
<code>cargo test --workspace</code> both green (0 failures) after the full
|
||||
propagation, <code>scripts/smoke_test_index.sh</code> green (870 kmers, same as every
|
||||
prior round), plus a manual CLI run of <code>index</code> (×2) → <code>merge</code> → <code>select</code>
|
||||
→ <code>reindex</code> → <code>utils --new-label</code> (rename) → <code>utils --stats</code>, all exit 0,
|
||||
confirming the four most-affected commands (the ones (10)'s verification
|
||||
already flagged as under-covered by the automated test suite) still work
|
||||
end to end against the new <code>Arc<IndexMeta></code>/on-disk-<code>IndexState</code> shape.</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign itself
|
||||
(only the <code>PartitionRouter</code>-<code>&mut</code>-removal piece landed, as a side
|
||||
effect); the <code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly
|
||||
deferred); the future cache-manager crate. (8)'s <code>obikalgorithm::
|
||||
Algorithm</code> trait, resumed and closed in (12) below.</p>
|
||||
<h2 id="12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session">(12) done (2026-08-21): <code>obikalgorithm::Algorithm</code> — the shared trait, resumed and closed in one session</h2>
|
||||
<p>Resumed (8)'s point 2 through a point-by-point discussion of what's
|
||||
actually common across the four pipeline algorithms, now that (11) made
|
||||
<code>KmerIndex</code> itself immutable everywhere. Four sub-points, each closed
|
||||
before moving to the next:</p>
|
||||
<p><strong>1. Receiver (<code>&self</code> vs <code>&mut self</code>)</strong> — investigated whether (11)'s
|
||||
removal of <code>&mut KmerIndex</code> also removed the need for <code>PartitionRouter::
|
||||
run</code> to take <code>&mut self</code>. It didn't: <code>PartitionRouter</code> holds real
|
||||
per-run state of its own (<code>writers: Vec<Option<SKFileWriter>></code>, open file
|
||||
handles, purely in-process RAM — confirmed by checking where <code>writers</code> is
|
||||
stored, nothing to do with <code>KmerIndex</code>/disk truth), unrelated to the
|
||||
index. First proposal (wrap <code>writers</code> in <code>RefCell</code> so all four could
|
||||
share a uniform <code>&self</code>) was retracted on pushback: manufacturing
|
||||
interior mutability with runtime borrow checks to satisfy a cosmetic
|
||||
uniformity that Rust doesn't even require is over-engineering — a trait
|
||||
method's receiver must match exactly across implementors, but nothing
|
||||
stops that shared receiver from being <code>&mut self</code> with three of the four
|
||||
implementations simply not using the mutability. Settled: trait declares
|
||||
<code>&mut self</code>; <code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> (previously <code>&self</code>)
|
||||
now also take <code>&mut self</code>, unused.</p>
|
||||
<p><strong>2. <code>path_source</code> as a <code>PartitionRouter</code> setter, not a <code>run()</code> param</strong> —
|
||||
added a <code>files: Option<Box<dyn Iterator<Item = PathBuf> + Send>></code> field +
|
||||
<code>.files(impl Iterator<Item = PathBuf> + Send + 'static) -> Self</code> setter
|
||||
(boxed rather than a generic type parameter on <code>PartitionRouter<'a></code>:
|
||||
negligible cost — one <code>PathBuf</code> per input <em>file</em>, not per k-mer — for a
|
||||
much more usable type when passing the builder around). <code>run</code> now does
|
||||
<code>self.files.take().ok_or_else(...)</code>, erroring if <code>.files(...)</code> was never
|
||||
called, instead of taking <code>path_source</code> as a parameter.</p>
|
||||
<p><strong>3. Unifying the three progress-callback bound shapes</strong> — reopened, then
|
||||
resolved differently than (8) originally framed it. First proposal
|
||||
(force everything to <code>FnMut(Progress) + Send</code>) was rejected on the same
|
||||
principle as point 1: <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn(Progress) + Sync</code>
|
||||
isn't arbitrary — their callback is invoked concurrently from multiple
|
||||
rayon worker threads, and <code>FnMut</code> requires exclusive access, so forcing
|
||||
it would mean wrapping the callback in a <code>Mutex</code> for zero benefit at the
|
||||
one real call site (<code>pb.inc(1)</code>, already thread-safe). The actual
|
||||
resolution: move <code>on_progress</code> off <code>run()</code>'s signature entirely, onto a
|
||||
per-algorithm <code>.on_progress(...)</code> setter — same treatment as point 2's
|
||||
<code>path_source</code> — so each algorithm keeps its own bound (<code>PartitionRouter</code>:
|
||||
<code>FnMut(Progress) + 'a</code>, sequential, no <code>Send</code> needed; <code>LayerBuilder</code>:
|
||||
<code>FnMut(Progress) + Send + 'a</code>, crosses into <code>PartitionRunner</code>'s
|
||||
<code>thread::scope</code>-spawned controller thread once; <code>Dereplicator</code>/<code>Counter</code>:
|
||||
<code>Fn(Progress) + Sync + 'a</code>, invoked concurrently from rayon workers).
|
||||
This <em>also</em> dissolves the original problem <code>run()</code> had: once the
|
||||
callback isn't part of <code>run</code>'s signature at all, there's nothing left to
|
||||
unify there, and point 4 (below) becomes trivial.</p>
|
||||
<p><strong>4. <code>Output</code> as an associated type, <code>Error</code> fixed</strong> — trivial once (3)
|
||||
moved the callback out: <code>Error</code> was already uniform (all four return
|
||||
<code>obiskio::SKResult<T></code> = <code>Result<T, SKError></code>), only <code>Output</code> varied
|
||||
(<code>()</code>/<code>()</code>/<code>KmerSpectrum</code>/<code>usize</code>). First cut reused <code>obiskio::SKResult</code>
|
||||
directly as the trait's return type — <strong>caught and corrected the same
|
||||
session</strong>: <code>SKError</code> enumerates I/O-specific cases (<code>BadMagic</code>/
|
||||
<code>Truncated</code>/<code>Compression</code>/...), meaningless at the level of a generic
|
||||
"algorithm" abstraction, and borrowing it made <code>obikalgorithm</code> — meant to
|
||||
be minimal and neutral — depend on a low-level I/O crate purely to reuse
|
||||
its error type. Textbook instance of the "petits pois" failure mode
|
||||
(patch around a convenient existing type instead of asking what this
|
||||
crate should actually own). Fixed to a genuinely generic, boxed error
|
||||
type owned by <code>obikalgorithm</code> itself:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// obikalgorithm — no dependency on obiskio or any other crate</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">error</span><span class="p">::</span><span class="n">Error</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">result</span><span class="p">::</span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">Error</span><span class="o">></span><span class="p">;</span>
|
||||
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Algorithm</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Output</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="bp">Self</span><span class="p">::</span><span class="n">Output</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Any concrete error (<code>SKError</code>, <code>std::io::Error</code>, ...) converts
|
||||
automatically via <code>?</code>, through <code>std</code>'s own blanket <code>impl<E: Error + Send
|
||||
+ Sync> From<E> for Box<dyn Error + Send + Sync></code> — no custom <code>From</code> impl
|
||||
needed, no dependency on the crate that defines the concrete error type.
|
||||
The four algorithms' <code>run</code> bodies needed no change beyond the signature's
|
||||
return type (every existing <code>?</code> on an <code>SKError</code>-returning subcall keeps
|
||||
compiling, converting through the same blanket impl at the boundary).</p>
|
||||
<p><code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> each <code>impl
|
||||
Algorithm for X<'_> { type Output = ...; fn run(&mut self) -> obikalgorithm::Result<...> { ... } }</code>
|
||||
— the old inherent <code>run</code> methods were removed outright (not kept as
|
||||
duplicates), so callers now <code>use obikalgorithm::Algorithm;</code> to call
|
||||
<code>.run()</code>. Every field-lifetime-bound boxed callback (<code>Box<dyn
|
||||
FnMut(Progress) + 'a></code> etc.) is tied to the algorithm's own <code>'a</code> (the
|
||||
<code>&'a KmerIndex</code> lifetime already on the struct), not <code>'static</code> — avoids
|
||||
forcing callers' progress closures to <code>move</code>-capture (and therefore clone
|
||||
or <code>Arc</code>-wrap) local state like <code>TracedBar</code>/EMA-rate accumulators that
|
||||
they'd otherwise want to keep using by reference after <code>run()</code> returns.</p>
|
||||
<p><strong>Why a new crate, not a submodule of <code>obikindexer</code></strong>: <code>obikmer::cmd::
|
||||
index::mod</code> and <code>obikphylo</code>'s own test helpers both need to call <code>.run()</code>
|
||||
on these algorithms, so the trait has to be reachable from outside
|
||||
<code>obikindexer</code> — putting it in <code>obikindexer</code> itself would work file-wise
|
||||
but conflates "the trait every algorithm implements" with "one crate's
|
||||
particular four implementations of it", the same reasoning that already
|
||||
separated <code>obikindex</code> (data model) from <code>obikindexer</code> (algorithms
|
||||
operating on it). <code>obikalgorithm</code> has <strong>no dependencies at all</strong> (see
|
||||
above); <code>obikindexer</code>, <code>obikmer</code>, and <code>obikphylo</code> (dev-dependency, for
|
||||
its test helper) all depend on it.</p>
|
||||
<p><strong>Blast radius</strong>: <code>obikindexer</code>'s four algorithm modules (struct field +
|
||||
setter + trait impl each); <code>obikmer::cmd::index::mod</code> (three call sites:
|
||||
<code>.on_progress(cb)</code> before <code>.run()</code>, unqualified now that the trait is in
|
||||
scope); <code>obikindexer::algorithms::partitionner::tests</code> and <code>obikphylo::
|
||||
siblings::tests</code> (both had direct <code>.run(None::<fn(Progress)>)</code>-shaped
|
||||
calls needing the same treatment). New <code>obikalgorithm</code> crate registered
|
||||
in the workspace <code>Cargo.toml</code>, depended on by <code>obikindexer</code>/<code>obikmer</code>
|
||||
(regular) and <code>obikphylo</code> (dev).</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and <code>cargo test
|
||||
--workspace</code> both green (0 failures), <code>scripts/smoke_test_index.sh</code> green
|
||||
(870 kmers, same as every prior round) — this round didn't repeat the
|
||||
manual <code>merge</code>/<code>select</code>/<code>reindex</code> CLI exercise from (11), since nothing
|
||||
in this pass touched those commands' code paths (only the four pipeline
|
||||
algorithms and <code>cmd::index</code>, already covered by the smoke test). Reverified
|
||||
after the <code>obiskio</code>-dependency fix above (same three checks, still green,
|
||||
<code>obikalgorithm/Cargo.toml</code> now has zero <code>[dependencies]</code>).</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign; the
|
||||
<code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly deferred); the
|
||||
future cache-manager crate (mentioned in (8) as a later, mirrored
|
||||
extension-trait exercise, not started).</p>
|
||||
<h2 id="the-problem">The problem</h2>
|
||||
<p>Reading a layer's data (MPHF + matrix) is not free: <code>MphfLayer::open</code> mmaps
|
||||
<code>mphf.bin</code> plus (<code>evidence.bin</code>/<code>fingerprint.bin</code> + <code>unitigs.bin</code>), and the
|
||||
|
||||
Binary file not shown.
@@ -909,6 +909,56 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -2195,6 +2245,56 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -4261,6 +4361,249 @@ among the survivors) — a single extra pass is sufficient.</p>
|
||||
<code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
|
||||
from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
|
||||
workspace <code>cargo test</code> green.</p>
|
||||
<h2 id="-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28"><code>--distance</code> unification: SNP corrections as first-class metrics (implemented, 2026-08-28)</h2>
|
||||
<p><strong>Implemented.</strong> <code>--metric</code> (renamed <code>--distance</code> — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
<code>--metric</code> was a misnomer) gains a family of <code>snp-*</code> values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
machinery (<code>PairwiseTally</code>, <code>obikphylo::siblings::algorithms::pairwise</code>)
|
||||
instead of <code>cache.distance(...)</code>'s existing per-layer traversal — a different
|
||||
code path behind the same CLI surface, not just another branch of one
|
||||
formula function.</p>
|
||||
<p><strong>Why unify at the CLI level despite the implementation split</strong>: phylogenetically
|
||||
a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
|
||||
to how the matrix was produced, so exposing it as a special-cased subcommand
|
||||
instead of a <code>--distance</code> value would misrepresent its role. The
|
||||
implementation divergence (sibling-annex-based vs. plain index scan) is real
|
||||
but belongs at the routing layer, invisible to the CLI's own vocabulary.</p>
|
||||
<p><strong><code>--subsample</code> becomes optional for <code>snp-*</code> distances</strong> (it stays mandatory
|
||||
for <code>--sankoff</code>/<code>--pseudo-alignment</code>, unrelated commands): absent means
|
||||
exhaustive, achieved for free by reusing <code>sample_index</code>'s existing
|
||||
proportional-per-layer-quota mechanism with <code>n</code> set to the index-wide total
|
||||
non-monomorphic-minorant count (already available from the sibling-annex
|
||||
stats) — every layer's quota then equals its own full count, giving Bernoulli
|
||||
<code>p = 1</code> everywhere, i.e. every eligible family is drawn. No second,
|
||||
exhaustive-only driver needed. Present means sampled, exactly as <code>--sankoff</code>
|
||||
already behaves.</p>
|
||||
<p><strong>One shared tally, many derived formulas.</strong> <code>PairwiseTally</code>'s <code>subst[4][4]</code>
|
||||
per-pair substitution counts (plus marginal base frequencies derived from it)
|
||||
are the sufficient statistic for every closed-form correction below — each
|
||||
is a small pure function <code>PairwiseTally -> Array2<f64></code>, at the same level as
|
||||
the already-implemented <code>raw_snp_distance</code>/<code>base_pair_tally</code>/
|
||||
<code>cardinality_tally</code>. No new full scan per formula, whether the tally itself
|
||||
was built exhaustively or from a subsample.</p>
|
||||
<p><strong><code>--raw-snp-counts</code> stays a separate, unrelated flag</strong> — same underlying
|
||||
tally, but a diagnostic (<code>n_snp</code>/<code>n_shared</code>/<code>n_eligible</code> per genome pair, one
|
||||
row per pair) rather than a distance value, and its long-table shape doesn't
|
||||
fold into a single N×N matrix the way a distance does. No change to its
|
||||
existing CSV format.</p>
|
||||
<h3 id="snp-distance-catalog"><code>snp-*</code> distance catalog</h3>
|
||||
<p>All closed-form (method-of-moments / direct formula), none requiring
|
||||
per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
|
||||
<em>tree</em>-ML usage but not its <em>pairwise</em> estimator, which is closed-form like
|
||||
F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>value</th>
|
||||
<th>corrects for</th>
|
||||
<th>inputs beyond raw counts</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>snp-raw</code></td>
|
||||
<td>nothing (uncorrected p-distance)</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-jc</code> (Jukes-Cantor, JC69)</td>
|
||||
<td>multiple substitutions per site</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k2p</code> (Kimura 2-parameter, K80)</td>
|
||||
<td>+ transition/transversion rate bias</td>
|
||||
<td>ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k81</code> (Kimura 3-parameter, K3ST)</td>
|
||||
<td>+ splits transversions into 2 categories</td>
|
||||
<td>ts/tv split, by category</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f81</code> (Felsenstein 81)</td>
|
||||
<td>+ unequal base frequencies (no ts/tv split)</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tajima-nei</code> (Tajima-Nei 1984)</td>
|
||||
<td>same goal as F81 (equal-input model), different formula, better small-sample behavior</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-t92</code> (Tamura 3-parameter)</td>
|
||||
<td>K2P + GC-content bias</td>
|
||||
<td>ts/tv split, GC content</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f84</code> (Felsenstein 84)</td>
|
||||
<td>full empirical base freqs + single ts/tv rate</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-hky85</code> (Hasegawa-Kishino-Yano, pairwise estimator)</td>
|
||||
<td>same inputs as F84, different formula</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tn93</code> (Tamura-Nei)</td>
|
||||
<td>full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate</td>
|
||||
<td>empirical base freqs, purine-ts/pyrimidine-ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-logdet</code> (LogDet / paralinear)</td>
|
||||
<td>no shared-model or stationarity assumption at all — general divergence-matrix determinant</td>
|
||||
<td>full empirical 4×4 divergence matrix (already <code>subst[4][4]</code>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tv</code> (transversions-only p-distance)</td>
|
||||
<td>diagnostic/deep-divergence variant — drops transitions entirely (they saturate first)</td>
|
||||
<td>tv-only counts</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
||||
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
||||
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
|
||||
by the user (<code>--gamma-shape <alpha></code>), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.</p>
|
||||
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
||||
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
||||
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
||||
<code>snp-logdet</code> are catalogued above but <strong>not implemented</strong>: <code>snp-logdet</code>
|
||||
needs the true <em>directional</em> per-pair base co-occurrence matrix
|
||||
(<code>PairwiseTally</code> only keeps the symmetrised substitution counts
|
||||
<code>BasePairTally</code> itself wants — see <code>snp_distance.rs</code>'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), <code>snp-tajima-nei</code> needs each genome's <em>own</em> base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
<code>snp-f84</code>/<code>snp-hky85</code> had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against <a href="https://github.com/emmanuelparadis/ape">ape</a>'s own
|
||||
<code>src/dist_dna.c</code>, not re-derived from memory). Adding any of these later is
|
||||
a new function in <code>obikphylo::siblings::algorithms::snp_distance</code>, plus for
|
||||
<code>snp-logdet</code>/<code>snp-tajima-nei</code> a new field on <code>PairStats</code>/a per-genome
|
||||
accumulator — not an architecture change.</p>
|
||||
<h3 id="exact-formulas-implemented-2026-08-28">Exact formulas (implemented, 2026-08-28)</h3>
|
||||
<p>Sufficient statistic, per genome pair <code>(i, j)</code>, from
|
||||
<code>PairwiseTally::categories</code>/<code>PairwiseTally::base_freq</code> (base order always
|
||||
<code>0=A, 1=C, 2=G, 3=T</code>, matching <code>FamilyMask</code>/<code>STATE_SYMBOL</code>):</p>
|
||||
<ul>
|
||||
<li><span class="arithmatex">\(n_{ts1}\)</span>: A↔G substitutions (purine transitions), <span class="arithmatex">\(n_{ts2}\)</span>: C↔T
|
||||
(pyrimidine transitions)</li>
|
||||
<li><span class="arithmatex">\(n_{tv1}\)</span>: A↔C and G↔T substitutions, <span class="arithmatex">\(n_{tv2}\)</span>: A↔T and C↔G
|
||||
(Kimura's two transversion categories)</li>
|
||||
<li><span class="arithmatex">\(n_{shared}\)</span>: loci where both genomes agree</li>
|
||||
<li><span class="arithmatex">\(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\)</span> (total eligible
|
||||
loci for the pair)</li>
|
||||
<li><span class="arithmatex">\(\pi_A, \pi_C, \pi_G, \pi_T\)</span>: pair-pooled base frequencies,
|
||||
<span class="arithmatex">\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)</span>
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)</li>
|
||||
</ul>
|
||||
<p>Derived proportions used below:</p>
|
||||
<div class="arithmatex">\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]</div>
|
||||
<p>Every formula below was checked term-by-term against <code>ape</code>'s own
|
||||
<code>src/dist_dna.c</code> (not re-derived from memory) before being ported to
|
||||
<code>obikphylo::siblings::algorithms::snp_distance</code>.</p>
|
||||
<p><strong><code>snp-raw</code></strong> — uncorrected p-distance:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{raw} = p
|
||||
\]</div>
|
||||
<p><strong><code>snp-tv</code></strong> — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, <em>is</em> the correction):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{tv} = Q
|
||||
\]</div>
|
||||
<p><strong><code>snp-jc</code></strong> (Jukes-Cantor, JC69):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-k2p</code></strong> (Kimura 2-parameter, K80), with <span class="arithmatex">\(a_1 = 1-2P-Q\)</span>, <span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-k81</code></strong> (Kimura 3-parameter, K3ST), with <span class="arithmatex">\(a_1 = 1-2P-2Q_1\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2P-2Q_2\)</span>, <span class="arithmatex">\(a_3 = 1-2Q_1-2Q_2\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-f81</code></strong> (Felsenstein 81), with <span class="arithmatex">\(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-t92</code></strong> (Tamura 3-parameter), with GC content
|
||||
<span class="arithmatex">\(g = \pi_C+\pi_G\)</span>, <span class="arithmatex">\(w = 2g(1-g)\)</span>, <span class="arithmatex">\(a_1 = 1 - \dfrac{P}{w} - Q\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-tn93</code></strong> (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
<span class="arithmatex">\(g_R = \pi_A+\pi_G\)</span>, <span class="arithmatex">\(g_Y = \pi_C+\pi_T\)</span>, and</p>
|
||||
<div class="arithmatex">\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]</div>
|
||||
<p><strong><code>+Γ</code> gamma correction</strong> (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of <span class="arithmatex">\(-\ln(x)\)</span> terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
<span class="arithmatex">\(\alpha\left(x^{-1/\alpha} - 1\right)\)</span> instead — the standard mechanical
|
||||
substitution (as <span class="arithmatex">\(\alpha \to \infty\)</span>, this expression → <span class="arithmatex">\(-\ln(x)\)</span>,
|
||||
recovering the uncorrected formula exactly). E.g. for JC:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]</div>
|
||||
<p>Verified term-by-term against <code>ape</code>'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to <code>snp-k2p</code>'s own <span class="arithmatex">\(a_1\)</span>/<span class="arithmatex">\(a_2\)</span> terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one <code>corrected_log</code> helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.</p>
|
||||
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
||||
<p><strong>Implemented.</strong> The primary distance-matrix output
|
||||
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
||||
default</strong> (widely read by external NJ tools — PHYLIP <code>neighbor</code>, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
label truncation, since genome labels here routinely exceed it), a <code>--csv</code>
|
||||
flag opts back into the current CSV format, PHYLIP-strict is a possible
|
||||
future addition (not now). This changes the <em>default</em> output of every
|
||||
existing <code>--distance</code> value (jaccard, hamming, bray-curtis, ...), not just
|
||||
the new <code>snp-*</code> ones — accepted explicitly (pre-release, single developer
|
||||
user, no external consumers to break). Scoped to the distance matrix only:
|
||||
<code>--shared-kmers</code> and <code>--raw-snp-counts</code> are counts, not distances, and keep
|
||||
their existing CSV-only format.</p>
|
||||
<h2 id="references">References</h2>
|
||||
<p>The Mash mutation-rate model this discussion contrasts with:
|
||||
(Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>
|
||||
|
||||
@@ -2182,9 +2182,9 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
|
||||
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
|
||||
workspace `cargo test` green.
|
||||
|
||||
## `--distance` unification: SNP corrections as first-class metrics (discussion, 2026-08-28)
|
||||
## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
**Decided, not yet implemented.** `--metric` (renamed `--distance` — several of
|
||||
**Implemented.** `--metric` (renamed `--distance` — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
`--metric` was a misnomer) gains a family of `snp-*` values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
@@ -2253,9 +2253,147 @@ formula as the base correction, weighted by a shape parameter `α` supplied
|
||||
by the user (`--gamma-shape <alpha>`), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.
|
||||
|
||||
**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
|
||||
`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
|
||||
"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
|
||||
`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
|
||||
needs the true *directional* per-pair base co-occurrence matrix
|
||||
(`PairwiseTally` only keeps the symmetrised substitution counts
|
||||
`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), `snp-tajima-nei` needs each genome's *own* base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
`snp-f84`/`snp-hky85` had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
|
||||
`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
|
||||
a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
|
||||
`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
|
||||
accumulator — not an architecture change.
|
||||
|
||||
### Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
Sufficient statistic, per genome pair `(i, j)`, from
|
||||
`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
|
||||
`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
|
||||
|
||||
- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
|
||||
(pyrimidine transitions)
|
||||
- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
|
||||
(Kimura's two transversion categories)
|
||||
- \(n_{shared}\): loci where both genomes agree
|
||||
- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
|
||||
loci for the pair)
|
||||
- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
|
||||
\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)
|
||||
|
||||
Derived proportions used below:
|
||||
|
||||
\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]
|
||||
|
||||
Every formula below was checked term-by-term against `ape`'s own
|
||||
`src/dist_dna.c` (not re-derived from memory) before being ported to
|
||||
`obikphylo::siblings::algorithms::snp_distance`.
|
||||
|
||||
**`snp-raw`** — uncorrected p-distance:
|
||||
|
||||
\[
|
||||
d_{raw} = p
|
||||
\]
|
||||
|
||||
**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, *is* the correction):
|
||||
|
||||
\[
|
||||
d_{tv} = Q
|
||||
\]
|
||||
|
||||
**`snp-jc`** (Jukes-Cantor, JC69):
|
||||
|
||||
\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]
|
||||
|
||||
**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
|
||||
\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
|
||||
|
||||
\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]
|
||||
|
||||
**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
|
||||
|
||||
\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]
|
||||
|
||||
**`snp-t92`** (Tamura 3-parameter), with GC content
|
||||
\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
|
||||
\(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
|
||||
|
||||
\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]
|
||||
|
||||
\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]
|
||||
|
||||
\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]
|
||||
|
||||
**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
|
||||
substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
|
||||
recovering the uncorrected formula exactly). E.g. for JC:
|
||||
|
||||
\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]
|
||||
|
||||
Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one `corrected_log` helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
`ape`-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.
|
||||
|
||||
### Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
**Decided, not yet implemented.** The primary distance-matrix output
|
||||
**Implemented.** The primary distance-matrix output
|
||||
(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
|
||||
default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
|
||||
Generated
+1
@@ -1670,6 +1670,7 @@ version = "1.2.2"
|
||||
dependencies = [
|
||||
"clap",
|
||||
"csv",
|
||||
"ndarray",
|
||||
"obifastwrite",
|
||||
"obikalgorithm",
|
||||
"obikdump",
|
||||
|
||||
@@ -29,6 +29,7 @@ obifastwrite = { path = "../obifastwrite" }
|
||||
obiskbuilder = { path = "../obiskbuilder" }
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
csv = "1"
|
||||
ndarray = "0.17"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_yaml = "0.9"
|
||||
tracing = "0.1.44"
|
||||
|
||||
@@ -1,10 +1,16 @@
|
||||
use std::path::PathBuf;
|
||||
|
||||
use clap::Args;
|
||||
use obikphylo::DistanceMetric;
|
||||
use obikphylo::{DistanceMetric, SnpDistanceKind};
|
||||
|
||||
/// `--distance` value — either one of `obikphylo::DistanceMetric`'s
|
||||
/// whole-index metrics (routed to `IndexCache::distance`) or one of
|
||||
/// `obikphylo::SnpDistanceKind`'s `snp-*` corrections (routed to
|
||||
/// `SiblingExt::snp_distance`, the sibling-annex pipeline) — two genuinely
|
||||
/// different code paths behind one CLI vocabulary, see
|
||||
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance` unification".
|
||||
#[derive(clap::ValueEnum, Clone, Copy, Debug)]
|
||||
pub enum MetricArg {
|
||||
pub enum DistanceArg {
|
||||
Jaccard,
|
||||
Mash,
|
||||
Hamming,
|
||||
@@ -17,35 +23,69 @@ pub enum MetricArg {
|
||||
Hellinger,
|
||||
#[value(name = "hellinger-euclidean")]
|
||||
HellingerEuclidean,
|
||||
#[value(name = "snp-raw")]
|
||||
SnpRaw,
|
||||
#[value(name = "snp-jc")]
|
||||
SnpJc,
|
||||
#[value(name = "snp-k2p")]
|
||||
SnpK2p,
|
||||
#[value(name = "snp-k81")]
|
||||
SnpK81,
|
||||
#[value(name = "snp-f81")]
|
||||
SnpF81,
|
||||
#[value(name = "snp-t92")]
|
||||
SnpT92,
|
||||
#[value(name = "snp-tn93")]
|
||||
SnpTn93,
|
||||
#[value(name = "snp-tv")]
|
||||
SnpTv,
|
||||
}
|
||||
|
||||
impl From<MetricArg> for DistanceMetric {
|
||||
fn from(m: MetricArg) -> Self {
|
||||
match m {
|
||||
MetricArg::Jaccard => DistanceMetric::Jaccard,
|
||||
MetricArg::Mash => DistanceMetric::Mash,
|
||||
MetricArg::Hamming => DistanceMetric::Hamming,
|
||||
MetricArg::BrayCurtis => DistanceMetric::BrayCurtis,
|
||||
MetricArg::RelfreqBrayCurtis => DistanceMetric::RelfreqBrayCurtis,
|
||||
MetricArg::Euclidean => DistanceMetric::Euclidean,
|
||||
MetricArg::RelfreqEuclidean => DistanceMetric::RelfreqEuclidean,
|
||||
MetricArg::Hellinger => DistanceMetric::Hellinger,
|
||||
MetricArg::HellingerEuclidean => DistanceMetric::HellingerEuclidean,
|
||||
}
|
||||
impl DistanceArg {
|
||||
/// `Some` for the whole-index metrics, `None` for `snp-*` values.
|
||||
pub fn as_classic(self) -> Option<DistanceMetric> {
|
||||
Some(match self {
|
||||
DistanceArg::Jaccard => DistanceMetric::Jaccard,
|
||||
DistanceArg::Mash => DistanceMetric::Mash,
|
||||
DistanceArg::Hamming => DistanceMetric::Hamming,
|
||||
DistanceArg::BrayCurtis => DistanceMetric::BrayCurtis,
|
||||
DistanceArg::RelfreqBrayCurtis => DistanceMetric::RelfreqBrayCurtis,
|
||||
DistanceArg::Euclidean => DistanceMetric::Euclidean,
|
||||
DistanceArg::RelfreqEuclidean => DistanceMetric::RelfreqEuclidean,
|
||||
DistanceArg::Hellinger => DistanceMetric::Hellinger,
|
||||
DistanceArg::HellingerEuclidean => DistanceMetric::HellingerEuclidean,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
/// `Some` for the `snp-*` values, `None` for the whole-index metrics.
|
||||
pub fn as_snp(self) -> Option<SnpDistanceKind> {
|
||||
Some(match self {
|
||||
DistanceArg::SnpRaw => SnpDistanceKind::Raw,
|
||||
DistanceArg::SnpJc => SnpDistanceKind::Jc,
|
||||
DistanceArg::SnpK2p => SnpDistanceKind::K2p,
|
||||
DistanceArg::SnpK81 => SnpDistanceKind::K81,
|
||||
DistanceArg::SnpF81 => SnpDistanceKind::F81,
|
||||
DistanceArg::SnpT92 => SnpDistanceKind::T92,
|
||||
DistanceArg::SnpTn93 => SnpDistanceKind::Tn93,
|
||||
DistanceArg::SnpTv => SnpDistanceKind::Tv,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Partial transfer of `obikmer`'s `phylo` command: the plain distance-metric
|
||||
/// path (`--metric`/NJ/UPGMA), annex construction (`--sibling-annex`),
|
||||
/// annex diagnostics (`--sibling-stats`, `--sibling-hist`), entropy
|
||||
/// reporting (`--shannon`), SNP pseudo-alignment sampling
|
||||
/// (`--pseudo-alignment`, `--subsample`, `--free-loss`, `--no-ambiguity`,
|
||||
/// `--entropy`/`--entropy-sd`), Sankoff cost-matrix calibration
|
||||
/// (`--sankoff`, `--sankoff-ratio-ceiling`) and its TNT/PhyG/IQ-TREE exports
|
||||
/// (`--tnt`, `--phyg`, `--iqtree`/`--iqtree-min-freq`,
|
||||
/// `--sankoff-cost-scale`) — everything else sibling-annex-based (raw SNP
|
||||
/// distance, family overlap, ...) stays in `obikmer` until the rest of
|
||||
/// `obikphylo::siblings` is reconnected (see the project memory on this).
|
||||
/// Partial transfer of `obikmer`'s `phylo` command: the whole-index
|
||||
/// `--distance` path (classic metrics + `snp-*` corrections/NJ/UPGMA),
|
||||
/// annex construction (`--sibling-annex`), annex diagnostics
|
||||
/// (`--sibling-stats`, `--sibling-hist`), entropy reporting (`--shannon`),
|
||||
/// SNP pseudo-alignment sampling (`--pseudo-alignment`, `--subsample`,
|
||||
/// `--free-loss`, `--no-ambiguity`, `--entropy`/`--entropy-sd`), Sankoff
|
||||
/// cost-matrix calibration (`--sankoff`, `--sankoff-ratio-ceiling`) and its
|
||||
/// TNT/PhyG/IQ-TREE exports (`--tnt`, `--phyg`, `--iqtree`/
|
||||
/// `--iqtree-min-freq`, `--sankoff-cost-scale`) — everything else
|
||||
/// sibling-annex-based (family overlap, ...) stays in `obikmer` until the
|
||||
/// rest of `obikphylo::siblings` is reconnected (see the project memory on
|
||||
/// this).
|
||||
#[derive(Args)]
|
||||
pub struct PhyloArgs {
|
||||
/// Index directory
|
||||
@@ -97,9 +137,13 @@ pub struct PhyloArgs {
|
||||
pub pseudo_alignment: bool,
|
||||
|
||||
/// Target number of variable sites to sample index-wide for
|
||||
/// `--pseudo-alignment` — a target, not a guarantee (proportional
|
||||
/// per-layer sampling; see `obikphylo::siblings::SiblingExt::snp_pseudo_alignment`'s
|
||||
/// own docs).
|
||||
/// `--pseudo-alignment`/`--sankoff` (mandatory for both) and for a
|
||||
/// `snp-*` `--distance` value (optional there: omitted means exhaustive
|
||||
/// — every non-monomorphic minorant of the whole index, not an
|
||||
/// approximation, see `obikphylo::siblings::SiblingExt::snp_distance`'s
|
||||
/// own docs) — a target, not a guarantee when given (proportional
|
||||
/// per-layer sampling; see
|
||||
/// `obikphylo::siblings::SiblingExt::snp_pseudo_alignment`'s own docs).
|
||||
#[arg(long)]
|
||||
pub subsample: Option<usize>,
|
||||
|
||||
@@ -205,14 +249,43 @@ pub struct PhyloArgs {
|
||||
#[arg(long, default_value = "100")]
|
||||
pub sankoff_cost_scale: f64,
|
||||
|
||||
/// Distance metric to compute
|
||||
/// Distance to compute — either a whole-index metric (`jaccard`,
|
||||
/// `mash`, `hamming`, `bray-curtis`, ...) or a `snp-*` correction over
|
||||
/// the central-position SNP substitution spectrum (`snp-raw`, `snp-jc`,
|
||||
/// `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`) —
|
||||
/// the latter route to a different computation entirely
|
||||
/// (`SiblingExt::snp_distance`, requires `--sibling-annex` first; see
|
||||
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance`
|
||||
/// unification" for the full catalog and why LogDet/Tajima-Nei/F84/
|
||||
/// HKY85 aren't offered yet).
|
||||
#[arg(long, value_enum, default_value = "jaccard")]
|
||||
pub metric: MetricArg,
|
||||
pub distance: DistanceArg,
|
||||
|
||||
/// Rate-heterogeneity correction (Jin-Nei gamma shape parameter `α`)
|
||||
/// for `snp-*` `--distance` values that support it
|
||||
/// (`obikphylo::SnpDistanceKind::supports_gamma`: every one except
|
||||
/// `snp-raw`/`snp-tv`, which have nothing to correct/are deliberately
|
||||
/// uncorrected). Has no effect on the whole-index metrics. Rejected at
|
||||
/// runtime if given alongside an unsupported `--distance` value.
|
||||
#[arg(long, value_name = "ALPHA")]
|
||||
pub gamma_shape: Option<f64>,
|
||||
|
||||
/// Minimum count to consider a kmer present when computing Jaccard on count indexes
|
||||
#[arg(long, default_value = "1")]
|
||||
pub presence_threshold: u32,
|
||||
|
||||
/// Write the primary distance matrix as plain CSV instead of the
|
||||
/// default relaxed-PHYLIP format (`n` on the first line, then one
|
||||
/// `label<TAB>value...` row per genome — no 10-character label
|
||||
/// truncation, unlike strict PHYLIP, not yet offered here). PHYLIP is
|
||||
/// the default because it's what external NJ tools (PHYLIP `neighbor`,
|
||||
/// FastME, T-REX, SplitsTree) actually read; CSV stays available for
|
||||
/// scripting/inspection. Only affects the primary distance matrix —
|
||||
/// `--shared-kmers` keeps its own CSV-only format regardless of this
|
||||
/// flag.
|
||||
#[arg(long)]
|
||||
pub csv: bool,
|
||||
|
||||
/// Also output the shared-kmer count matrix (CSV)
|
||||
#[arg(long)]
|
||||
pub shared_kmers: bool,
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
mod args;
|
||||
mod iqtree;
|
||||
mod phyg;
|
||||
mod phylip;
|
||||
mod sankoff;
|
||||
mod tnt;
|
||||
|
||||
@@ -19,6 +20,7 @@ use tracing::info;
|
||||
|
||||
use iqtree::write_iqtree;
|
||||
use phyg::write_sankoff_phyg;
|
||||
use phylip::write_phylip_relaxed;
|
||||
use sankoff::{write_sankoff_alignment_fasta, write_sankoff_matrix_csv, write_sankoff_params};
|
||||
use tnt::write_sankoff_tnt;
|
||||
|
||||
@@ -271,57 +273,100 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
}
|
||||
|
||||
info!("computing {:?} distances for {} genome(s)", args.metric, n);
|
||||
// ── Distance computation: classic whole-index metric vs. `snp-*` ───────────
|
||||
// Two genuinely different code paths behind one `--distance` value — see
|
||||
// `args::DistanceArg`'s own docs.
|
||||
let (matrix, shared_kmers) = match args.distance.as_classic() {
|
||||
Some(metric) => {
|
||||
info!("computing {metric:?} distances for {n} genome(s)");
|
||||
let need_shared = args.shared_kmers || args.nj || args.upgma;
|
||||
let t = Stage::start("distance");
|
||||
let result = cache
|
||||
.distance(metric, need_shared, args.presence_threshold)
|
||||
.unwrap_or_else(|e| {
|
||||
eprintln!("error computing distances: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
rep.push(t.stop());
|
||||
(result.matrix, result.shared_kmers)
|
||||
}
|
||||
None => {
|
||||
if args.shared_kmers {
|
||||
eprintln!("error: --shared-kmers has no meaning for a snp-* --distance value");
|
||||
std::process::exit(1);
|
||||
}
|
||||
let kind = args.distance.as_snp().expect("DistanceArg is always classic or snp");
|
||||
info!(
|
||||
"computing {kind:?} SNP distance for {n} genome(s){}",
|
||||
match args.subsample {
|
||||
Some(n) => format!(" (subsampled, target {n} site(s))"),
|
||||
None => " (exhaustive)".into(),
|
||||
}
|
||||
);
|
||||
let t = Stage::start("snp_distance");
|
||||
let matrix = cache
|
||||
.snp_distance(
|
||||
kind,
|
||||
args.subsample,
|
||||
args.free_loss,
|
||||
args.no_ambiguity,
|
||||
&exclude_mask,
|
||||
entropy_bias,
|
||||
args.gamma_shape,
|
||||
)
|
||||
.unwrap_or_else(|e| {
|
||||
eprintln!("error computing SNP distance: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
rep.push(t.stop());
|
||||
(matrix, None)
|
||||
}
|
||||
};
|
||||
|
||||
let need_shared = args.shared_kmers || args.nj || args.upgma;
|
||||
let t = Stage::start("distance");
|
||||
let result = cache
|
||||
.distance(args.metric.into(), need_shared, args.presence_threshold)
|
||||
.unwrap_or_else(|e| {
|
||||
eprintln!("error computing distances: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
rep.push(t.stop());
|
||||
|
||||
// Rows/columns kept in every matrix CSV below — `cache.distance(...)`
|
||||
// above is computed over every genome regardless; only the writers skip
|
||||
// Rows/columns kept in every matrix output below — the computation
|
||||
// above runs over every genome regardless; only the writers skip
|
||||
// excluded ones.
|
||||
let kept: Vec<usize> = (0..n).filter(|&i| !exclude_mask[i]).collect();
|
||||
|
||||
// ── Distance matrix → CSV ─────────────────────────────────────────────────
|
||||
let write_dist_csv = |w: &mut dyn Write| {
|
||||
write!(w, "genome").unwrap();
|
||||
for &j in &kept { write!(w, ",{}", labels[j]).unwrap(); }
|
||||
writeln!(w).unwrap();
|
||||
for &i in &kept {
|
||||
write!(w, "{}", labels[i]).unwrap();
|
||||
for &j in &kept {
|
||||
write!(w, ",{:.6}", result.matrix[[i, j]]).unwrap();
|
||||
}
|
||||
// ── Distance matrix → relaxed PHYLIP (default) or CSV (`--csv`) ────────────
|
||||
let write_dist = |w: &mut dyn Write| {
|
||||
if args.csv {
|
||||
write!(w, "genome").unwrap();
|
||||
for &j in &kept { write!(w, ",{}", labels[j]).unwrap(); }
|
||||
writeln!(w).unwrap();
|
||||
for &i in &kept {
|
||||
write!(w, "{}", labels[i]).unwrap();
|
||||
for &j in &kept {
|
||||
write!(w, ",{:.6}", matrix[[i, j]]).unwrap();
|
||||
}
|
||||
writeln!(w).unwrap();
|
||||
}
|
||||
} else {
|
||||
write_phylip_relaxed(w, &labels, &kept, &matrix);
|
||||
}
|
||||
};
|
||||
|
||||
match &args.output {
|
||||
Some(prefix) => {
|
||||
let path = format!("{}_dist.csv", prefix.display());
|
||||
let suffix = if args.csv { "_dist.csv" } else { "_dist.phy" };
|
||||
let path = format!("{}{suffix}", prefix.display());
|
||||
let mut f = BufWriter::new(std::fs::File::create(&path).unwrap_or_else(|e| {
|
||||
eprintln!("error creating {path}: {e}");
|
||||
std::process::exit(1);
|
||||
}));
|
||||
write_dist_csv(&mut f);
|
||||
write_dist(&mut f);
|
||||
info!("distance matrix → {path}");
|
||||
}
|
||||
None => {
|
||||
let stdout = io::stdout();
|
||||
let mut out = BufWriter::new(stdout.lock());
|
||||
write_dist_csv(&mut out);
|
||||
write_dist(&mut out);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Shared-kmer matrix → CSV ──────────────────────────────────────────────
|
||||
if args.shared_kmers {
|
||||
if let Some(shared) = &result.shared_kmers {
|
||||
if let Some(shared) = &shared_kmers {
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_shared.csv", p.display()))
|
||||
.unwrap_or_else(|| "shared.csv".into());
|
||||
@@ -343,7 +388,7 @@ pub fn run(args: PhyloArgs) {
|
||||
|
||||
// ── NJ tree ────────────────────────────────────────────────────────────────
|
||||
if args.nj {
|
||||
let tree = neighbor_joining(&result.matrix, &labels).unwrap_or_else(|e| {
|
||||
let tree = neighbor_joining(&matrix, &labels).unwrap_or_else(|e| {
|
||||
eprintln!("error computing NJ tree: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
@@ -360,7 +405,7 @@ pub fn run(args: PhyloArgs) {
|
||||
|
||||
// ── UPGMA tree ───────────────────────────────────────────────────────────────
|
||||
if args.upgma {
|
||||
let newick = upgma(&result.matrix, &labels).to_newick();
|
||||
let newick = upgma(&matrix, &labels).to_newick();
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_upgma.nwk", p.display()))
|
||||
.unwrap_or_else(|| "upgma.nwk".into());
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
//! Relaxed-PHYLIP distance-matrix writer — the default output format for
|
||||
//! `--distance` (see `args::PhyloArgs::csv`'s own docs for why): `n` on the
|
||||
//! first line, then one `label<TAB>value...` row per genome. "Relaxed"
|
||||
//! (unlike strict PHYLIP) means no 10-character label truncation/padding —
|
||||
//! just whitespace-separated fields, which every external NJ tool this
|
||||
//! targets (PHYLIP `neighbor`, FastME, T-REX, SplitsTree) already reads,
|
||||
//! and genome labels here routinely exceed strict PHYLIP's 10 characters.
|
||||
|
||||
use std::io::Write;
|
||||
|
||||
use ndarray::Array2;
|
||||
|
||||
pub(super) fn write_phylip_relaxed(w: &mut dyn Write, labels: &[String], indices: &[usize], matrix: &Array2<f64>) {
|
||||
writeln!(w, "{}", indices.len()).unwrap();
|
||||
for &i in indices {
|
||||
write!(w, "{}", labels[i]).unwrap();
|
||||
for &j in indices {
|
||||
write!(w, "\t{:.6}", matrix[[i, j]]).unwrap();
|
||||
}
|
||||
writeln!(w).unwrap();
|
||||
}
|
||||
}
|
||||
@@ -19,4 +19,5 @@ mod tree;
|
||||
pub mod siblings;
|
||||
|
||||
pub use distance::{DistanceMetric, DistanceOutput, Metrics};
|
||||
pub use siblings::SnpDistanceKind;
|
||||
pub use tree::{Tree, neighbor_joining, upgma};
|
||||
|
||||
@@ -22,6 +22,7 @@ mod masking;
|
||||
mod minorant_selection;
|
||||
mod pairwise;
|
||||
mod sankoff;
|
||||
mod snp_distance;
|
||||
mod stats;
|
||||
mod subsample;
|
||||
|
||||
@@ -33,6 +34,7 @@ pub use cardcomp::{
|
||||
};
|
||||
pub use pairwise::{BasePairTally, CardinalityTally, RawSnpDistanceOutput};
|
||||
pub use sankoff::SankoffBundle;
|
||||
pub use snp_distance::SnpDistanceKind;
|
||||
pub use stats::SiblingAnnexStats;
|
||||
pub use subsample::{EntropyBias, SurvivingFamily};
|
||||
|
||||
@@ -41,6 +43,7 @@ pub(crate) use annex::build_layer_sibling_annex;
|
||||
pub(crate) use entropy::{ensure_layer_entropy_annex, family_entropy, family_entropy_4};
|
||||
pub(crate) use family_scan::{Selection, scan_layer_families};
|
||||
pub(crate) use sankoff::sankoff_bundle;
|
||||
pub(crate) use snp_distance::snp_distance;
|
||||
pub(crate) use stats::{sibling_annex_stats, sibling_family_size_histogram};
|
||||
|
||||
/// Whether every layer number in `cache` fits in a `FamilyMask` field
|
||||
|
||||
@@ -175,6 +175,66 @@ impl PairwiseTally {
|
||||
}
|
||||
CardinalityTally { counts }
|
||||
}
|
||||
|
||||
/// Categorised substitution counts for pair `(i, j)`, plus its eligible
|
||||
/// ("shared") locus count — the sufficient statistic every closed-form
|
||||
/// `algorithms::snp_distance` correction is built from. Base order
|
||||
/// `0=A,1=C,2=G,3=T` (same convention as `FamilyMask`/`STATE_SYMBOL`).
|
||||
/// `ts1`/`ts2` split transitions by purine (A↔G) vs pyrimidine (C↔T)
|
||||
/// pair — required for TN93, collapse to a single `ts1+ts2` for
|
||||
/// K80/K81/T92, which don't distinguish them. `tv1`/`tv2` split
|
||||
/// transversions the way Kimura's 3-parameter model does (A↔C/G↔T vs
|
||||
/// A↔T/C↔G) — collapse to `tv1+tv2` for every model that doesn't need
|
||||
/// the distinction (K80, F81, T92, TN93).
|
||||
pub(crate) fn categories(&self, i: usize, j: usize) -> PairCategories {
|
||||
let stats = self.pair(i, j);
|
||||
PairCategories {
|
||||
ts1: stats.subst[0][2],
|
||||
ts2: stats.subst[1][3],
|
||||
tv1: stats.subst[0][1] + stats.subst[2][3],
|
||||
tv2: stats.subst[0][3] + stats.subst[1][2],
|
||||
shared: stats.same_all.iter().sum(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Pooled base composition for pair `(i, j)`, estimated from both
|
||||
/// genomes' observed calls at their `shared`/differing eligible loci —
|
||||
/// `2 * same_all[a]` (each agreement locus contributes `a` to both
|
||||
/// genomes) plus `Σ_b subst[a][b]` (each differing locus contributes
|
||||
/// `a` to whichever of the two genomes carried it — `subst` is
|
||||
/// symmetric by construction, so summing one row already counts each
|
||||
/// such event exactly once, see `reduce_pairwise`'s own docs), over
|
||||
/// `2 * n_eligible_loci` total base observations. Feeds F81/T92/TN93,
|
||||
/// which all correct for base-composition bias using exactly this kind
|
||||
/// of pooled empirical frequency (Nei & Kumar's standard estimator for
|
||||
/// a pairwise comparison, not a whole-index average).
|
||||
pub(crate) fn base_freq(&self, i: usize, j: usize) -> [f64; 4] {
|
||||
let stats = self.pair(i, j);
|
||||
let mut counts = [0u64; 4];
|
||||
for a in 0..4 {
|
||||
counts[a] = 2 * stats.same_all[a] + (0..4).map(|b| stats.subst[a][b]).sum::<u64>();
|
||||
}
|
||||
let total: u64 = counts.iter().sum();
|
||||
if total == 0 {
|
||||
return [0.25; 4];
|
||||
}
|
||||
counts.map(|c| c as f64 / total as f64)
|
||||
}
|
||||
}
|
||||
|
||||
/// See [`PairwiseTally::categories`]'s own docs.
|
||||
pub(crate) struct PairCategories {
|
||||
pub ts1: u64,
|
||||
pub ts2: u64,
|
||||
pub tv1: u64,
|
||||
pub tv2: u64,
|
||||
pub shared: u64,
|
||||
}
|
||||
|
||||
impl PairCategories {
|
||||
pub(crate) fn n_eligible(&self) -> u64 {
|
||||
self.ts1 + self.ts2 + self.tv1 + self.tv2 + self.shared
|
||||
}
|
||||
}
|
||||
|
||||
/// Raw p-distance restricted to loci that are single-copy in **both**
|
||||
|
||||
@@ -0,0 +1,398 @@
|
||||
//! `snp-*` `--distance` values — closed-form (method-of-moments) pairwise
|
||||
//! corrections over the central-position SNP substitution spectrum, all
|
||||
//! derived from one shared [`PairwiseTally`] (see its own module docs: a
|
||||
//! single `O(n²)` structure, built once, from which every correction below
|
||||
//! is cheap post-processing, never a second index scan). See
|
||||
//! `DevDocMD/theory/evolutionary_distances.md`, "`--distance` unification"
|
||||
//! for the design discussion and the literature this implements
|
||||
//! ([`ape`](https://github.com/emmanuelparadis/ape)'s `dist.dna` C source,
|
||||
//! `src/dist_dna.c`, verified formula-by-formula against the upstream
|
||||
//! implementation rather than re-derived from memory).
|
||||
//!
|
||||
//! Deliberately **not** included here: LogDet/paralinear (needs the true
|
||||
//! *directional* per-pair base co-occurrence matrix — `PairwiseTally`
|
||||
//! only keeps the symmetrised substitution counts `BasePairTally` itself
|
||||
//! wants, which loses exactly the compositional-asymmetry information
|
||||
//! LogDet exists to detect), Tajima-Nei (needs each genome's *own* base
|
||||
//! composition, not the pair-pooled estimate `base_freq` below provides),
|
||||
//! F84 and the ML-fit HKY85 tree usage (no formula independently verified
|
||||
//! against a primary source at implementation time). Adding any of these
|
||||
//! later is a new function in this module plus, for LogDet/Tajima-Nei, a
|
||||
//! new field on `PairStats`/a per-genome accumulator — not an architecture
|
||||
//! change.
|
||||
|
||||
use ndarray::Array2;
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
use obikindex::OKIResult;
|
||||
|
||||
use super::pairwise::PairwiseTally;
|
||||
use super::sibling_family_size_histogram;
|
||||
use super::subsample::{EntropyBias, sample_index};
|
||||
|
||||
/// One `snp-*` `--distance` value. `pub`: part of
|
||||
/// [`crate::siblings::extensions::SiblingExt::snp_distance`]'s public
|
||||
/// signature.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum SnpDistanceKind {
|
||||
/// Uncorrected p-distance (`snp / (snp + shared)`).
|
||||
Raw,
|
||||
/// Jukes-Cantor (JC69): corrects for multiple substitutions per site,
|
||||
/// equal rates, equal base frequencies.
|
||||
Jc,
|
||||
/// Kimura 2-parameter (K80): + transition/transversion rate bias.
|
||||
K2p,
|
||||
/// Kimura 3-parameter (K81/K3ST): splits transversions into the two
|
||||
/// categories A↔C/G↔T and A↔T/C↔G, each its own rate.
|
||||
K81,
|
||||
/// Felsenstein 81 (F81): JC69 + unequal (pair-pooled empirical) base
|
||||
/// frequencies, no ts/tv distinction.
|
||||
F81,
|
||||
/// Tamura 3-parameter (T92): K80 + GC-content bias.
|
||||
T92,
|
||||
/// Tamura-Nei (TN93): unequal base frequencies + separate purine
|
||||
/// (A↔G) / pyrimidine (C↔T) transition rates + a transversion rate —
|
||||
/// the richest closed-form correction implemented here.
|
||||
Tn93,
|
||||
/// Transversions-only p-distance — diagnostic/deep-divergence variant,
|
||||
/// deliberately uncorrected (dropping transitions, which saturate
|
||||
/// first, is itself the correction).
|
||||
Tv,
|
||||
}
|
||||
|
||||
impl SnpDistanceKind {
|
||||
/// Whether `--gamma-shape` applies to this correction — every
|
||||
/// rate-based model here except the two with no "-ln(x)" term to
|
||||
/// reinterpret as a gamma mixture (`Raw`, nothing to correct; `Tv`,
|
||||
/// deliberately uncorrected).
|
||||
pub fn supports_gamma(self) -> bool {
|
||||
!matches!(self, SnpDistanceKind::Raw | SnpDistanceKind::Tv)
|
||||
}
|
||||
}
|
||||
|
||||
/// `-ln(x)`, gamma-mixture corrected when `alpha` is given: replaces the
|
||||
/// single-rate `-ln(x)` term with the standard Jin-Nei (1990) gamma
|
||||
/// substitution `alpha * (x^(-1/alpha) - 1)` — the mechanical term-wise
|
||||
/// rewrite every gamma-corrected distance in the literature (JC+Γ, K80+Γ,
|
||||
/// F81+Γ, ...) applies to each log term of its base formula; verified here
|
||||
/// against `ape`'s own JC69/K80/F81 gamma branches (`dist_dna.c`), then
|
||||
/// generalised the same way to every other log term this module uses
|
||||
/// (K81/T92/TN93 each being multi-term sums of exactly this shape).
|
||||
fn corrected_log(x: f64, alpha: Option<f64>) -> f64 {
|
||||
match alpha {
|
||||
Some(alpha) => alpha * (x.powf(-1.0 / alpha) - 1.0),
|
||||
None => -x.ln(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn raw(tally: &PairwiseTally, i: usize, j: usize) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
(c.ts1 + c.ts2 + c.tv1 + c.tv2) as f64 / l as f64
|
||||
}
|
||||
|
||||
pub(crate) fn tv_only(tally: &PairwiseTally, i: usize, j: usize) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
(c.tv1 + c.tv2) as f64 / l as f64
|
||||
}
|
||||
|
||||
pub(crate) fn jc(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let p = raw(tally, i, j);
|
||||
0.75 * corrected_log(1.0 - 4.0 * p / 3.0, alpha)
|
||||
}
|
||||
|
||||
pub(crate) fn k2p(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let p = (c.ts1 + c.ts2) as f64 / l as f64;
|
||||
let q = (c.tv1 + c.tv2) as f64 / l as f64;
|
||||
let a1 = 1.0 - 2.0 * p - q;
|
||||
let a2 = 1.0 - 2.0 * q;
|
||||
0.5 * corrected_log(a1, alpha) + 0.25 * corrected_log(a2, alpha)
|
||||
}
|
||||
|
||||
pub(crate) fn k81(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let p = (c.ts1 + c.ts2) as f64 / l as f64;
|
||||
let q = c.tv1 as f64 / l as f64;
|
||||
let r = c.tv2 as f64 / l as f64;
|
||||
let a1 = 1.0 - 2.0 * p - 2.0 * q;
|
||||
let a2 = 1.0 - 2.0 * p - 2.0 * r;
|
||||
let a3 = 1.0 - 2.0 * q - 2.0 * r;
|
||||
0.25 * (corrected_log(a1, alpha) + corrected_log(a2, alpha) + corrected_log(a3, alpha))
|
||||
}
|
||||
|
||||
pub(crate) fn f81(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let p = raw(tally, i, j);
|
||||
let freq = tally.base_freq(i, j);
|
||||
let e = 1.0 - freq.iter().map(|f| f * f).sum::<f64>();
|
||||
match alpha {
|
||||
Some(a) => e * a * ((1.0 - p / e).powf(-1.0 / a) - 1.0),
|
||||
None => -e * (1.0 - p / e).ln(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn t92(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let p = (c.ts1 + c.ts2) as f64 / l as f64;
|
||||
let q = (c.tv1 + c.tv2) as f64 / l as f64;
|
||||
let freq = tally.base_freq(i, j);
|
||||
let gc = freq[1] + freq[2]; // C + G
|
||||
let wg = 2.0 * gc * (1.0 - gc);
|
||||
let a1 = 1.0 - p / wg - q;
|
||||
let a2 = 1.0 - 2.0 * q;
|
||||
wg * corrected_log(a1, alpha) + 0.5 * (1.0 - wg) * corrected_log(a2, alpha)
|
||||
}
|
||||
|
||||
pub(crate) fn tn93(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let freq = tally.base_freq(i, j);
|
||||
let g_r = freq[0] + freq[2]; // A + G (purines)
|
||||
let g_y = freq[1] + freq[3]; // C + T (pyrimidines)
|
||||
let k1 = 2.0 * freq[0] * freq[2] / g_r;
|
||||
let k2 = 2.0 * freq[1] * freq[3] / g_y;
|
||||
let k3 = 2.0 * (g_r * g_y - freq[0] * freq[2] * g_y / g_r - freq[1] * freq[3] * g_r / g_y);
|
||||
let p1 = c.ts1 as f64 / l as f64; // A<->G
|
||||
let p2 = c.ts2 as f64 / l as f64; // C<->T
|
||||
let q = (c.tv1 + c.tv2) as f64 / l as f64;
|
||||
let w1 = 1.0 - p1 / k1 - q / (2.0 * g_r);
|
||||
let w2 = 1.0 - p2 / k2 - q / (2.0 * g_y);
|
||||
let w3 = 1.0 - q / (2.0 * g_r * g_y);
|
||||
k1 * corrected_log(w1, alpha) + k2 * corrected_log(w2, alpha) + k3 * corrected_log(w3, alpha)
|
||||
}
|
||||
|
||||
fn formula(kind: SnpDistanceKind) -> fn(&PairwiseTally, usize, usize, Option<f64>) -> f64 {
|
||||
match kind {
|
||||
SnpDistanceKind::Raw => |t, i, j, _| raw(t, i, j),
|
||||
SnpDistanceKind::Tv => |t, i, j, _| tv_only(t, i, j),
|
||||
SnpDistanceKind::Jc => jc,
|
||||
SnpDistanceKind::K2p => k2p,
|
||||
SnpDistanceKind::K81 => k81,
|
||||
SnpDistanceKind::F81 => f81,
|
||||
SnpDistanceKind::T92 => t92,
|
||||
SnpDistanceKind::Tn93 => tn93,
|
||||
}
|
||||
}
|
||||
|
||||
/// Build (or reuse) a [`PairwiseTally`] and reduce it to one `n×n` distance
|
||||
/// matrix under `kind`. `n`: `Some(target)` samples proportionally per
|
||||
/// layer exactly like `--sankoff`/`--pseudo-alignment` (see
|
||||
/// `algorithms::subsample::sample_index`'s own docs); `None` means
|
||||
/// exhaustive — every non-monomorphic minorant of the whole index, achieved
|
||||
/// by handing `sample_index` a quota equal to the index-wide total (from
|
||||
/// [`sibling_family_size_histogram`], annex-bits-only, no cross-partition
|
||||
/// resolution), which makes every per-layer quota equal to that layer's own
|
||||
/// full eligible count, i.e. Bernoulli `p = 1` everywhere — no separate
|
||||
/// exhaustive-only driver needed. Entropy-biased sampling
|
||||
/// (`entropy_bias`) only makes sense for a genuine subsample, so it's
|
||||
/// ignored (forced to `None`) when `n` is `None`, regardless of what the
|
||||
/// caller passes.
|
||||
///
|
||||
/// `gamma_shape`: `--gamma-shape`, `None` disables the correction. Rejected
|
||||
/// with [`obikindex::OKIError::InvalidInput`] if given alongside a `kind`
|
||||
/// that doesn't support it ([`SnpDistanceKind::supports_gamma`]) — checked
|
||||
/// here rather than left to silently no-op.
|
||||
pub(crate) fn snp_distance(
|
||||
cache: &IndexCache,
|
||||
kind: SnpDistanceKind,
|
||||
n: Option<usize>,
|
||||
free_loss: bool,
|
||||
no_ambiguity: bool,
|
||||
excluded: &[bool],
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
gamma_shape: Option<f64>,
|
||||
) -> OKIResult<Array2<f64>> {
|
||||
if gamma_shape.is_some() && !kind.supports_gamma() {
|
||||
return Err(obikindex::OKIError::InvalidInput(
|
||||
"--gamma-shape has no effect on this --distance value".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let n_genomes = cache.meta().genomes().len();
|
||||
let mut tally = PairwiseTally::new(n_genomes);
|
||||
|
||||
let (target, entropy_bias) = match n {
|
||||
Some(target) => (target, entropy_bias),
|
||||
None => {
|
||||
let counts = sibling_family_size_histogram(cache)?;
|
||||
let total_eligible = (counts[1] + counts[2] + counts[3]) as usize;
|
||||
(total_eligible, None)
|
||||
}
|
||||
};
|
||||
|
||||
if target > 0 {
|
||||
sample_index(
|
||||
cache,
|
||||
target,
|
||||
free_loss,
|
||||
no_ambiguity,
|
||||
excluded,
|
||||
entropy_bias,
|
||||
|_partition, _layer, survivors| {
|
||||
super::pairwise::reduce_pairwise(&survivors, &mut tally);
|
||||
},
|
||||
)?;
|
||||
}
|
||||
|
||||
let f = formula(kind);
|
||||
Ok(Array2::from_shape_fn((n_genomes, n_genomes), |(i, j)| {
|
||||
if i == j { 0.0 } else { f(&tally, i, j, gamma_shape) }
|
||||
}))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::siblings::FamilyMask;
|
||||
use crate::siblings::algorithms::SurvivingFamily;
|
||||
use super::super::pairwise::reduce_pairwise;
|
||||
|
||||
// Base bit positions, matching `PairwiseTally::categories`' convention
|
||||
// (0=A, 1=C, 2=G, 3=T).
|
||||
const A: u8 = 1 << 0;
|
||||
const C: u8 = 1 << 1;
|
||||
const G: u8 = 1 << 2;
|
||||
const T: u8 = 1 << 3;
|
||||
|
||||
/// Two-genome synthetic tally: one variable family per `(genome0,
|
||||
/// genome1)` pair in `pairs`, plus `n_shared` fully-agreeing families to
|
||||
/// pad the eligible-locus denominator — cycled over all 4 bases (not a
|
||||
/// single one) so the pooled base composition stays non-degenerate
|
||||
/// (`F81`/`T92`/`TN93`'s denominators — `E`/`wg` — are exactly `0` for a
|
||||
/// single-base composition, an edge case irrelevant to real data, not
|
||||
/// something these tests need to exercise).
|
||||
fn tally_2genomes(pairs: &[(u8, u8)], n_shared: usize) -> PairwiseTally {
|
||||
let mut families = Vec::new();
|
||||
for &(a, b) in pairs {
|
||||
families.push(SurvivingFamily {
|
||||
family_idx: families.len(),
|
||||
mask: FamilyMask::EMPTY.with(a.trailing_zeros() as u8).with(b.trailing_zeros() as u8),
|
||||
genome_mask: vec![a, b],
|
||||
});
|
||||
}
|
||||
const BASES: [u8; 4] = [A, C, G, T];
|
||||
for i in 0..n_shared {
|
||||
let b = BASES[i % 4];
|
||||
let other = BASES[(i + 1) % 4];
|
||||
families.push(SurvivingFamily {
|
||||
family_idx: families.len(),
|
||||
mask: FamilyMask::EMPTY.with(b.trailing_zeros() as u8).with(other.trailing_zeros() as u8),
|
||||
genome_mask: vec![b, b],
|
||||
});
|
||||
}
|
||||
let mut tally = PairwiseTally::new(2);
|
||||
reduce_pairwise(&families, &mut tally);
|
||||
tally
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn categories_classify_each_substitution_type() {
|
||||
// A<->G (ts1), C<->T (ts2), A<->C (tv1), A<->T (tv2), + 1 shared.
|
||||
let tally = tally_2genomes(&[(A, G), (C, T), (A, C), (A, T)], 1);
|
||||
let c = tally.categories(0, 1);
|
||||
assert_eq!((c.ts1, c.ts2, c.tv1, c.tv2, c.shared), (1, 1, 1, 1, 1));
|
||||
assert_eq!(c.n_eligible(), 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identical_genomes_give_zero_distance_under_every_correction() {
|
||||
// No substitutions at all — every formula's "-ln(1)" term is 0.
|
||||
let tally = tally_2genomes(&[], 10);
|
||||
for kind in [
|
||||
SnpDistanceKind::Raw,
|
||||
SnpDistanceKind::Jc,
|
||||
SnpDistanceKind::K2p,
|
||||
SnpDistanceKind::K81,
|
||||
SnpDistanceKind::F81,
|
||||
SnpDistanceKind::T92,
|
||||
SnpDistanceKind::Tn93,
|
||||
SnpDistanceKind::Tv,
|
||||
] {
|
||||
let d = formula(kind)(&tally, 0, 1, None);
|
||||
assert!(d.abs() < 1e-9, "{kind:?} gave {d}, expected ~0");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn raw_matches_hand_computed_ratio() {
|
||||
// 1 substitution (A<->G) out of 5 eligible loci (1 subst + 4 shared).
|
||||
let tally = tally_2genomes(&[(A, G)], 4);
|
||||
assert!((raw(&tally, 0, 1) - 0.2).abs() < 1e-12);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn jc_matches_hand_computed_correction() {
|
||||
// p = 1/5 = 0.2 -> d = -0.75 * ln(1 - 4*0.2/3) = -0.75 * ln(1 - 4/15).
|
||||
let tally = tally_2genomes(&[(A, G)], 4);
|
||||
let expected = -0.75 * (1.0 - 4.0f64 * 0.2 / 3.0).ln();
|
||||
assert!((jc(&tally, 0, 1, None) - expected).abs() < 1e-12);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn jc_uncorrected_undershoots_raw_p_distance() {
|
||||
// JC always corrects *upward* from the raw p-distance (multiple-hit
|
||||
// correction only adds distance, never removes it) for any p in its
|
||||
// domain.
|
||||
let tally = tally_2genomes(&[(A, G), (C, T)], 20);
|
||||
let p = raw(&tally, 0, 1);
|
||||
let d = jc(&tally, 0, 1, None);
|
||||
assert!(d > p, "JC-corrected {d} should exceed raw p-distance {p}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gamma_correction_converges_to_uncorrected_as_alpha_grows() {
|
||||
// The Jin-Nei substitution alpha*(x^(-1/alpha) - 1) -> -ln(x) as
|
||||
// alpha -> infinity — a basic sanity check on `corrected_log`
|
||||
// independent of any hand-checked literature value.
|
||||
let tally = tally_2genomes(&[(A, G), (A, C)], 20);
|
||||
let uncorrected = jc(&tally, 0, 1, None);
|
||||
let large_alpha = jc(&tally, 0, 1, Some(1.0e6));
|
||||
assert!(
|
||||
(uncorrected - large_alpha).abs() < 1e-3,
|
||||
"uncorrected {uncorrected} vs large-alpha gamma {large_alpha}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gamma_shape_rejected_for_unsupported_kind() {
|
||||
assert!(!SnpDistanceKind::Raw.supports_gamma());
|
||||
assert!(!SnpDistanceKind::Tv.supports_gamma());
|
||||
assert!(SnpDistanceKind::Jc.supports_gamma());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn base_freq_sums_to_one_and_matches_pooled_counts() {
|
||||
// 2 A/G substitutions (genome0=A, genome1=G each time) + 1 shared
|
||||
// A/A family -> pooled bases across both genomes: A appears 4 times
|
||||
// (1 per substitution's genome0 side, 2 from the shared locus's
|
||||
// both genomes), G appears 2 times (1 per substitution's genome1
|
||||
// side), total 6 (2 genomes * 3 loci).
|
||||
let tally = tally_2genomes(&[(A, G), (A, G)], 1);
|
||||
let freq = tally.base_freq(0, 1);
|
||||
assert!((freq.iter().sum::<f64>() - 1.0).abs() < 1e-12);
|
||||
assert!((freq[0] - 4.0 / 6.0).abs() < 1e-12); // A
|
||||
assert!((freq[2] - 2.0 / 6.0).abs() < 1e-12); // G
|
||||
assert!((freq[1] - 0.0).abs() < 1e-12); // C
|
||||
}
|
||||
}
|
||||
@@ -9,15 +9,16 @@
|
||||
use std::io::{BufWriter, Write};
|
||||
use std::path::Path;
|
||||
|
||||
use ndarray::Array2;
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
use obisys::progress_bar;
|
||||
|
||||
use crate::siblings::algorithms::{
|
||||
EntropyBias, SankoffBundle, Selection, SiblingAnnexStats, SnpAlignment,
|
||||
EntropyBias, SankoffBundle, Selection, SiblingAnnexStats, SnpAlignment, SnpDistanceKind,
|
||||
build_layer_sibling_annex, family_entropy, family_entropy_4, is_fast_mode,
|
||||
sankoff_bundle, scan_layer_families, sibling_annex_stats, sibling_family_size_histogram,
|
||||
snp_pseudo_alignment,
|
||||
snp_distance, snp_pseudo_alignment,
|
||||
};
|
||||
use crate::siblings::extensions::SiblingBuilder;
|
||||
use crate::siblings::ENTROPY_ANNEX_FILE_NAME;
|
||||
@@ -121,6 +122,38 @@ pub trait SiblingExt {
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
ratio_ceiling: f64,
|
||||
) -> OKIResult<SankoffBundle>;
|
||||
|
||||
/// A `snp-*` `--distance` matrix — one of the closed-form corrections
|
||||
/// in [`SnpDistanceKind`], over the central-position SNP substitution
|
||||
/// spectrum (requires an already-built sibling annex, run
|
||||
/// [`build_sibling_annex`](Self::build_sibling_annex) first).
|
||||
///
|
||||
/// `n`: `Some(target)` samples proportionally per layer exactly like
|
||||
/// [`sankoff_bundle`](Self::sankoff_bundle)/
|
||||
/// [`snp_pseudo_alignment`](Self::snp_pseudo_alignment) (`--subsample`);
|
||||
/// `None` means exhaustive — every non-monomorphic minorant of the
|
||||
/// whole index, not an approximation (see
|
||||
/// `algorithms::snp_distance::snp_distance`'s own docs for how this
|
||||
/// reuses the same sampling machinery at `p = 1` rather than a second
|
||||
/// driver). `free_loss`/`no_ambiguity`/`excluded`/`entropy_bias` — same
|
||||
/// meaning as `snp_pseudo_alignment`'s own (the last forced to `None`
|
||||
/// when `n` is `None`: entropy-biasing only makes sense for a genuine
|
||||
/// subsample).
|
||||
///
|
||||
/// `gamma_shape`: `--gamma-shape`, the Jin-Nei rate-heterogeneity
|
||||
/// correction — `None` disables it, `Some(alpha)` is rejected with
|
||||
/// [`OKIError::InvalidInput`] for a `kind` that doesn't support it
|
||||
/// ([`SnpDistanceKind::supports_gamma`]).
|
||||
fn snp_distance(
|
||||
&self,
|
||||
kind: SnpDistanceKind,
|
||||
n: Option<usize>,
|
||||
free_loss: bool,
|
||||
no_ambiguity: bool,
|
||||
excluded: &[bool],
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
gamma_shape: Option<f64>,
|
||||
) -> OKIResult<Array2<f64>>;
|
||||
}
|
||||
|
||||
impl SiblingExt for IndexCache {
|
||||
@@ -265,4 +298,17 @@ impl SiblingExt for IndexCache {
|
||||
) -> OKIResult<SankoffBundle> {
|
||||
sankoff_bundle(self, n, free_loss, no_ambiguity, excluded, entropy_bias, ratio_ceiling)
|
||||
}
|
||||
|
||||
fn snp_distance(
|
||||
&self,
|
||||
kind: SnpDistanceKind,
|
||||
n: Option<usize>,
|
||||
free_loss: bool,
|
||||
no_ambiguity: bool,
|
||||
excluded: &[bool],
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
gamma_shape: Option<f64>,
|
||||
) -> OKIResult<Array2<f64>> {
|
||||
snp_distance(self, kind, n, free_loss, no_ambiguity, excluded, entropy_bias, gamma_shape)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -35,8 +35,8 @@ pub use siblingannex::{FamilyMask, SiblingAnnex, SiblingAnnexBuilder};
|
||||
|
||||
pub use algorithms::{
|
||||
BasePairTally, CardinalityTally, EntropyBias, RawSnpDistanceOutput, SankoffBundle,
|
||||
SiblingAnnexStats, SnpAlignment, cardinality_transition_probs, composition_transition_probs,
|
||||
pairwise_cost_matrix,
|
||||
SiblingAnnexStats, SnpAlignment, SnpDistanceKind, cardinality_transition_probs,
|
||||
composition_transition_probs, pairwise_cost_matrix,
|
||||
};
|
||||
pub use extensions::SiblingExt;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user