Implement SNP distance models with gamma correction and PHYLIP output
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
This commit is contained in:
Eric Coissac
@@ -1367,6 +1367,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1652,6 +1663,17 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#persistentcompactintmatrixsparse-implemented-2026-08-26" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -2582,6 +2604,98 @@ mismatches. The dense/sparse performance gap is gone — previously sparse
|
||||
systematic gap. <code>pack --sparse</code>'s claimed query win isn't confirmed
|
||||
outright by this (sparse should arguably now <em>beat</em> dense on truly sparse
|
||||
real data, not just tie), but the pathological regression is fixed.</p>
|
||||
<h2 id="persistentcompactintmatrixsparse-implemented-2026-08-26"><code>PersistentCompactIntMatrix::Sparse</code> — implemented (2026-08-26)</h2>
|
||||
<p>Closes the gap flagged throughout this document ("no sparse count format
|
||||
exists yet", <code>traits.rs:9-12</code>'s "Explicitly deferred"): <code>obicompactvec</code>
|
||||
already had <code>PersistentSparseCompactIntMatrix</code> (row-major, built on top of
|
||||
<code>PersistentSparseBitMatrix</code> as its "which columns are non-zero" support,
|
||||
values <em>not</em> deduplicated — see that struct's own doc comment), but it was
|
||||
never wired into <code>PersistentCompactIntMatrix</code>, the dense-dispatching enum
|
||||
every real consumer (<code>TypedLayer<PersistentCompactIntMatrix></code>,
|
||||
<code>KmerLayer::Count</code>) actually holds. Concretely: <code>kmer_index.rs::
|
||||
pack_matrices(sparse=true)</code> already called <code>pack_sparse_compact_int_matrix</code>
|
||||
on every layer's <code>counts/</code> — but <code>PersistentCompactIntMatrix::open</code> had no
|
||||
code path back to what that just wrote, so a <code>Count</code> layer became
|
||||
unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
|
||||
anyone ran <code>pack --sparse</code> on an index with count layers. Root cause, not a
|
||||
workaround: add the missing <code>Sparse</code> variant.</p>
|
||||
<ul>
|
||||
<li>
|
||||
<p><strong>Enum + dispatch</strong> (<code>intmatrix.rs</code>): <code>PersistentCompactIntMatrix::Sparse
|
||||
(PersistentSparseCompactIntMatrix)</code>, detected in <code>open</code>/<code>detect_storage</code>
|
||||
via a <code>singleton_values.pciv</code> marker (mirrors <code>PersistentBitMatrix</code>'s own
|
||||
<code>sparse_meta.json</code> check), reported via <code>storage_kind()</code>. <code>col</code>/
|
||||
<code>col_view</code>/<code>col_persist</code> panic/<code>Unsupported</code> on <code>Sparse</code>, same convention
|
||||
as the bit side. <code>sub_matrix</code>/<code>fill_sub_matrix</code> and <code>nonzero_iter</code>
|
||||
unified the same way <code>PersistentBitMatrix</code>'s already are (drain
|
||||
<code>nonzero_iter</code>, one traversal per format — see "Implemented
|
||||
(2026-08-20)" above); <code>nonzero_iter</code> had to become <code>Box<dyn Iterator<...>></code>
|
||||
for the same reason (<code>Columnar</code>/<code>Packed</code>/<code>Sparse</code> are different concrete
|
||||
types). No change needed in <code>obikindex</code> at all — <code>KmerLayer::Count</code>
|
||||
already only ever holds <code>TypedLayer<PersistentCompactIntMatrix></code>, so the
|
||||
enum absorbing <code>Sparse</code> fixes the unreadable-layer bug for free, same as
|
||||
<code>PersistentBitMatrix::Sparse</code> already did on the presence side.</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong><code>CountPartials</code>, non-naive</strong> (<code>sparse_intmatrix.rs</code>): unlike
|
||||
<code>PersistentSparseBitMatrix</code>'s dict-driven <code>col_weights_and_pair_counts</code>,
|
||||
values here aren't deduplicated (two rows can share the same non-zero
|
||||
column set via the same <code>dict_id</code> while carrying different counts), so
|
||||
the "weight by how many rows share a dict entry" shortcut doesn't carry
|
||||
over. What does: a single row-major pass (<code>row_major_pairwise</code>, decodes
|
||||
each row once via <code>for_each_cell_in_row</code>, nests over that row's own
|
||||
co-present columns) — <code>O(Σ k̄²)</code> over populated rows instead of the naive
|
||||
<code>O(n_cols² × n)</code> column-pair rescan, same complexity class as the bit
|
||||
side minus the dict multiplicity discount. Kernels used: <code>min(a,b)</code>
|
||||
(bray, relfreq-bray — both vanish when either side is absent, so no
|
||||
correction needed), <code>a·b</code> and <code>√(a·b)</code> (euclidean/relfreq-euclidean and
|
||||
hellinger — these <em>do</em> need a correction, reconstructed from per-column
|
||||
marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code>, since <code>(a-0)² = a² ≠ 0</code> unlike
|
||||
the <code>min</code>-based formulas). <code>threshold_jaccard(1)</code> shortcuts straight to
|
||||
<code>support</code>'s own <code>BitPartials::partial_jaccard</code> (threshold 1 is exactly
|
||||
presence); <code>threshold_jaccard(0)</code> is closed-form (every <code>u32</code> is <code>≥ 0</code>).</p>
|
||||
</li>
|
||||
<li>
|
||||
<p><strong>Two pre-existing bugs found and fixed while wiring the <code>threshold==1</code>
|
||||
shortcut</strong> (<code>bitmatrix/sparse.rs</code>, <code>BitPartials for
|
||||
PersistentSparseBitMatrix</code>, present since the 2026-08-15 implementation
|
||||
above, never caught because no test compared <code>Sparse</code>'s raw <code>partial_*</code>
|
||||
output against dense on real data — only the diagonal-blind
|
||||
<code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code> finalisations were tested):</p>
|
||||
</li>
|
||||
<li><code>partial_jaccard</code>'s diagonal was <code>(0, 2×col_weights[i])</code> instead of a
|
||||
genuine self-comparison <code>(col_weights[i], col_weights[i])</code> —
|
||||
<code>col_weights_and_pair_counts</code>'s <code>inter</code> never pairs a column with
|
||||
itself by construction.</li>
|
||||
<li><code>partial_hamming</code>'s off-diagonal formula itself was wrong: <code>total -
|
||||
union</code> (count of rows where <em>neither</em> column is present) instead of
|
||||
the actual Hamming distance <code>col_weights[i] + col_weights[j] -
|
||||
2×inter[i,j]</code> (symmetric-difference size). Only coincides with the
|
||||
correct value when <code>col_weights[i] + col_weights[j] == total</code>, so
|
||||
small/synthetic test data could easily have hidden it.</li>
|
||||
</ul>
|
||||
<p>Neither surfaced through <code>jaccard_dist_matrix</code>/<code>hamming_dist_matrix</code>
|
||||
(both explicitly zero their own diagonal at finalisation, and the
|
||||
off-diagonal <code>partial_hamming</code> bug had gone untested against dense
|
||||
entirely) — only visible to a caller of the raw <code>partial_*</code> methods
|
||||
directly, which is exactly what <code>partial_threshold_jaccard(1)</code>'s new
|
||||
shortcut became. Fixed at the source, not patched around at the call
|
||||
site; regression test added:
|
||||
<code>tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal</code>.</p>
|
||||
<ul>
|
||||
<li><strong>Tests</strong>: <code>tests::intmatrix::sparse_roundtrip_matches_columnar</code>/
|
||||
<code>sparse_roundtrip_from_packed</code> (the <code>open</code>-dispatch fix, both build
|
||||
paths); <code>tests::intmatrix::sparse_count_partials_match_dense</code> (all six
|
||||
<code>CountPartials</code> formulas, thresholds 0/1/2/3, against <code>Columnar</code> on
|
||||
asymmetric-presence data — this is what caught the diagonal gap in the
|
||||
int side's own new code before it shipped, the same way it exposed the
|
||||
two pre-existing bit-side bugs above); <code>obikindex</code>'s
|
||||
<code>count_layer_transparently_reads_sparse_after_pack</code> — the actual
|
||||
end-to-end regression test for the original "layer unreadable after
|
||||
<code>pack --sparse</code>" bug, built → packed sparse → reopened, compared against
|
||||
the pre-pack dense read. <code>cargo test -p obicompactvec -p obikindex</code>:
|
||||
green, no regressions (180 + 12 tests).</li>
|
||||
</ul>
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1168,7 +1168,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</code></pre></div>
|
||||
<hr/>
|
||||
<h2 id="matrix-types">Matrix types</h2>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has two variants (<code>Columnar</code>, <code>Packed</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -1185,12 +1185,12 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Packed</code></td>
|
||||
<td>single <code>matrix.pbmx</code> mmap file</td>
|
||||
<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
|
||||
<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Sparse</code> (bit only)</td>
|
||||
<td><code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major</td>
|
||||
<td><code>Sparse</code></td>
|
||||
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
||||
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -1200,7 +1200,8 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
|
||||
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||
|
||||
@@ -1213,6 +1213,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1719,6 +1835,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1853,6 +2085,21 @@ The underlying module boundary and its rationale (Layer tier / Partition
|
||||
tier / Index tier, each depending only downward) are unchanged; only the
|
||||
crate-vs-module packaging changed. See <a href="../layer_tier/">obikindex::layer</a>
|
||||
for the current module doc.</p>
|
||||
<p><strong>Superseded, second event, same day (2026-08-21):</strong> <code>obikpartitionner</code>
|
||||
and <code>obikderep</code>, the two algorithm crates, are also gone — but unlike
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> above, they were <strong>not</strong> folded into
|
||||
<code>obikindex</code>. They were first (mistakenly) merged into <code>obikindex</code> as an
|
||||
<code>algorithms</code> submodule, then corrected into a new sibling crate,
|
||||
<strong><code>obikindexer</code></strong>, holding <code>obikindexer::algorithms::{partitionner,
|
||||
dereplicator}</code> and depending on <code>obikindex</code> — never the reverse, same
|
||||
dependency direction <code>obikpartitionner</code>/<code>obikderep</code> already had. Read
|
||||
<code>obikpartitionner::X</code> as <code>obikindexer::algorithms::partitionner::X</code> and
|
||||
<code>obikderep::X</code> as <code>obikindexer::algorithms::dereplicator::X</code> throughout
|
||||
what follows. The distinction the mistake surfaced, worth keeping: data
|
||||
crates (<code>obikindex</code>, holding the <code>index</code>/<code>partition</code>/<code>layer</code> model) merge
|
||||
naturally into one crate as submodules; algorithm crates that operate on
|
||||
that model from outside stay separate, so the dependency only ever runs
|
||||
one way.</p>
|
||||
<p>Status (2026-08-20, latest pass): (1) done — <code>obilayeredmap::Layer</code>
|
||||
exists, <code>Mat</code> is gone. (1b) done — <code>Layer::Empty</code>, the first non-ready
|
||||
state, added (panics on every read method). (2a) done — the
|
||||
@@ -1872,7 +2119,24 @@ turned out to be bigger than <code>KmerPartition</code> alone: <code>Layer</code
|
||||
constructors don't self-name either. Full redesign of both, agreed in
|
||||
detail, session ended (budget) before implementation — see "(5) design
|
||||
agreed" below; <strong>read it before touching <code>KmerPartition</code>/<code>Layer</code>
|
||||
signatures</strong>, the shape is fully specified. Earlier mix-up, for
|
||||
signatures</strong>, the shape is fully specified. (6) done — <code>Counter</code>, a third
|
||||
algorithm, extracted from <code>PartitionRouter</code> the same way <code>Dereplicator</code>
|
||||
was in (4). (7) done — <code>LayerBuilder</code>, the fourth and last pipeline
|
||||
algorithm; the indexing pipeline is now fully decomposed into
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator, counter,
|
||||
layer_builder}</code>. (8) design agreed, item 1 done in (9) —
|
||||
<code>obikindexer::extensions::PrivateBuilder</code>, private, six construction-only
|
||||
<code>KmerIndex</code> methods moved out. (10) done, same session — <code>obikindex::
|
||||
IndexBuilder</code>, public, the four maintenance methods
|
||||
(<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>)
|
||||
shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>. Item 2 from (8)
|
||||
(<code>obikalgorithm::Algorithm</code>) done in (12) — new crate, <code>type Output</code> +
|
||||
<code>fn run(&mut self) -> SKResult<Self::Output></code>, <code>on_progress</code> moved off
|
||||
<code>run()</code>'s signature entirely into a per-algorithm <code>.on_progress(...)</code>
|
||||
setter. Note: <code>Layer</code> renamed
|
||||
<code>KmerLayer</code> (2026-08-21, outside this conversation). (5) itself still not
|
||||
implemented, still first on the "order of remaining work" list. Earlier
|
||||
mix-up, for
|
||||
context: an earlier
|
||||
version of this doc used the name <code>KmerPartition</code> (singular) for what was
|
||||
actually the <em>collection</em> type (later renamed <code>KmerPartitions</code>, later
|
||||
@@ -2515,6 +2779,622 @@ itself or a new crate.</p>
|
||||
examples were judged not enough to be sure of the shape (<code>Fn+Sync</code> vs
|
||||
<code>FnMut</code> callback bound already diverged between the two that exist).</li>
|
||||
</ol>
|
||||
<h2 id="6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator">(6) done (2026-08-21): <code>Counter</code> — third algorithm, extracted the same way as <code>Dereplicator</code></h2>
|
||||
<p>Between (5) and this, the user did a session of their own crate
|
||||
restructuring (see the two "Superseded" notes at the top of this file):
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> folded into <code>obikindex</code> as submodules
|
||||
(<code>obikindex::partition</code>, <code>obikindex::layer</code>), and <code>obikpartitionner</code>/
|
||||
<code>obikderep</code> merged into one sibling crate, <code>obikindexer</code>, holding
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator}</code>. (5)'s design
|
||||
(<code>Layer</code>/<code>KmerPartition</code> self-naming by number, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code> → <code>&</code> fix) was <strong>not</strong> part of that — pure crate/module packaging,
|
||||
confirmed by reading the actual code (<code>Layer::open</code>/<code>create</code> still take an
|
||||
external <code>dir: &Path</code>, <code>KmerPartition</code> still eagerly opens all layers,
|
||||
<code>PartitionRouter</code> still holds <code>&mut KmerIndex</code>). (5) remains exactly as
|
||||
specified, not yet implemented.</p>
|
||||
<p>This step: <code>count_kmer</code> (still living on <code>PartitionRouter</code>, per (4)'s own
|
||||
"still not done" note) extracted into <code>obikindexer::algorithms::counter::
|
||||
Counter</code>, mirroring <code>Dereplicator</code> exactly — third data point for the
|
||||
eventual <code>obikalgorithm</code> trait, still not extracted (still only 3 examples
|
||||
with 2 different callback bounds; holding off per (5)'s "order of
|
||||
remaining work").</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_partial</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_partial</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span><span class="w"> </span><span class="c1">// setter, mirrors PartitionRouter's style; defaults to false</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="n">KmerSpectrum</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Same shape as <code>Dereplicator</code> throughout: <code>Fn(Progress) + Sync</code> (not
|
||||
<code>FnMut</code>) since counting is also a parallel <code>par_iter</code> over partitions, an
|
||||
<code>AtomicU64</code> position counter incremented from inside the parallel closure
|
||||
so progress reports arrive in real time rather than bursting at the end
|
||||
once <code>.collect()</code> finishes, <code>total: Some(n_partitions)</code> (known up front).
|
||||
<code>KmerSpectrum</code> (the <code>{f0, f1, counts}</code> aggregate) moved from
|
||||
<code>partitionner::router</code> to <code>counter</code>, since it's <code>Counter::run</code>'s return
|
||||
value now, not <code>PartitionRouter</code>'s. <code>count.rs</code>/<code>kmer_sort.rs</code> moved
|
||||
verbatim from <code>partitionner/</code> to <code>counter/</code> (unchanged bodies — only
|
||||
<code>count_kmer</code> itself, <code>KmerSpectrum</code>, and the imports they pulled in were
|
||||
removed from <code>router.rs</code>).</p>
|
||||
<p>One divergence from <code>Dereplicator</code>: a <code>keep_partial</code> setter exists (no
|
||||
equivalent on <code>Dereplicator</code>, which has no setters at all) — a real,
|
||||
already-present parameter (<code>keep_intermediate</code> at the CLI), not a
|
||||
speculative addition.</p>
|
||||
<p><code>count_kmer</code>'s three former callers (<code>obikmer::cmd::index</code>, <code>obikphylo</code>'s
|
||||
test harness, <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test helper) all updated to <code>Counter::new(&idx).
|
||||
run(...)</code> — the last one simplified further: it used to read back
|
||||
<code>kmer_spectrum_raw.json</code> from disk after calling <code>count_partition</code>
|
||||
directly (white-box), now it just uses the <code>KmerSpectrum</code> <code>Counter::run</code>
|
||||
already returns.</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus an end-to-end CLI smoke test
|
||||
against real FASTA data (scatter → dereplicate → count → index-build →
|
||||
query) — required every time per (3)'s lesson, and it earned its keep
|
||||
again: the very first smoke-test query returned zero matches, which
|
||||
looked like a regression until traced to the query sequence itself being
|
||||
low-complexity ("GGCCCCCCACG", six same-base runs) and rejected by
|
||||
<em>query's own</em> default entropy threshold — nothing to do with this change.
|
||||
Re-tested with a different substring, confirmed working (kmer found,
|
||||
count matched the index).</p>
|
||||
<p>Still not done: (5) (<code>Layer</code>/<code>KmerPartition</code> redesign, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code>→<code>&</code>), the future cache crate, <code>build_layers</code> (still a <code>KmerIndex</code>
|
||||
inherent method, not an algorithm), and <code>obikalgorithm</code> itself.</p>
|
||||
<h2 id="7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm">(7) done (2026-08-21): <code>LayerBuilder</code> — fourth and last pipeline algorithm</h2>
|
||||
<p>Closes out the indexing pipeline: <code>build_layers</code>/<code>build_index_layer</code>
|
||||
(the last stage still living as <code>KmerIndex</code> inherent methods, flagged as
|
||||
inconsistent since (6)) extracted into <code>obikindexer::algorithms::
|
||||
layer_builder::LayerBuilder</code>, same two-phase shape as the other three.</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">min_abundance</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">max_abundance</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_intermediate</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">min_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">max_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_intermediate</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">FnMut</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="p">;</span><span class="w"> </span><span class="c1">// returns total kmers built</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p><strong>Different from all three prior extractions in one respect, deliberately
|
||||
not "fixed" to match them</strong>: the actual per-partition construction logic
|
||||
(De Bruijn graph from dereplicated superkmers + provisional counts →
|
||||
unitigs → MPHF → matrix) stayed put as <code>KmerIndex::build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> (both already <code>pub</code>) — not moved into
|
||||
<code>obikindexer</code>. Checked first: unlike <code>dereplicate_partition</code>/
|
||||
<code>count_partition</code> (which only ever had one caller), <code>build_index_layer</code>
|
||||
depends on several <code>obikindex</code>-internal helpers (<code>graph_pipeline::
|
||||
{write_graph_as_unitigs, materialize_layer}</code>, <code>common::olm_to_sk</code>) that
|
||||
are <code>pub(crate)</code> and shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>'s own
|
||||
layer-construction paths — moving <code>build_index_layer</code> out would have
|
||||
meant either exporting that internal surface just for this one algorithm
|
||||
or duplicating it. Neither was needed: <code>build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> were <em>already</em> public <code>KmerIndex</code> methods, so
|
||||
<code>LayerBuilder</code>'s job is purely the orchestration around them (scheduling,
|
||||
config, progress) — the exact same "algorithm calls already-public
|
||||
<code>KmerIndex</code> primitives" shape <code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>
|
||||
already have, just at a coarser grain for this one stage. This is the
|
||||
"is the producer's API actually deficient?" check from
|
||||
[[feedback_no_spaghetti_petits_pois]] applied and answered "no" — not
|
||||
skipped.</p>
|
||||
<p><strong>Two more real divergences, both forced by <code>PartitionRunner</code>, not
|
||||
arbitrary:</strong>
|
||||
- Uses <code>obikindex::PartitionRunner</code> (NUMA-aware scheduler, already
|
||||
<code>pub use</code>d from <code>obikindex</code>) instead of plain <code>rayon::into_par_iter</code>
|
||||
like <code>Dereplicator</code>/<code>Counter</code> — matches what <code>build_layers</code> already used
|
||||
before extraction; this stage is more CPU/memory-intensive per partition
|
||||
(graph construction) than scatter/dereplicate/count.
|
||||
- Callback bound is <code>FnMut(Progress) + Send</code> — a third variant, not
|
||||
matching either prior shape. <code>PartitionRunner::run</code>'s <code>on_done</code> is
|
||||
invoked from its own single controller thread (never concurrently, so
|
||||
no <code>Sync</code> needed, unlike <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn + Sync</code>), but
|
||||
that controller thread is itself <code>std::thread::scope</code>-spawned, so the
|
||||
closure still has to be <code>Send</code> to cross into it — caught immediately by
|
||||
the compiler (<code>cannot be sent between threads safely</code>) when <code>Send</code> was
|
||||
first omitted, not a design guess. <code>obikalgorithm</code>'s eventual shared
|
||||
trait now has three real callback-bound data points to reconcile
|
||||
(<code>FnMut</code> alone for <code>PartitionRouter::run</code>'s sequential loop, <code>FnMut +
|
||||
Send</code> here, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code>'s <code>rayon</code>
|
||||
<code>par_iter</code>), not two.</p>
|
||||
<p><code>KmerIndex::build_layers</code> deleted outright (<code>KmerIndex</code> stays a pure data
|
||||
structure — no compute orchestration methods, consistent with <code>dereplicate</code>/
|
||||
<code>count_kmer</code>'s removal in (4)/(6)). New <code>KmerIndex::mark_indexed()</code> added,
|
||||
symmetric to <code>mark_scattered</code>/<code>mark_counted</code>, replacing the inline
|
||||
<code>touch(SENTINEL_INDEXED)</code> that used to live inside <code>build_layers</code>.
|
||||
<code>Stage::start("index")</code>/<code>rep.push(...)</code> and the <code>progress_bar</code>/
|
||||
<code>"{n} total kmers indexed"</code> log line both moved to <code>cmd/index/mod.rs</code>,
|
||||
same pattern as (3)/(4)/(6) — <code>LayerBuilder</code> renders nothing itself, just
|
||||
reports <code>Progress</code>.</p>
|
||||
<p>All callers updated: <code>cmd/index/mod.rs</code> (Stage 3), <code>obikphylo</code>'s test
|
||||
harness (also gained a <code>mark_indexed()</code> call it was missing — harmless
|
||||
before since nothing checked <code>IndexState::Indexed</code> in that test, but now
|
||||
correct).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the end-to-end CLI smoke test
|
||||
(<code>scripts/smoke_test_index.sh</code>, built earlier specifically so this
|
||||
verification step is a one-liner from now on) — 870 kmers indexed, query
|
||||
round-trip confirmed, same numbers as (6).</p>
|
||||
<p><strong>The indexing pipeline is now fully decomposed</strong>: <code>obikindexer::
|
||||
algorithms::{partitionner, dereplicator, counter, layer_builder}</code>, each a
|
||||
<code>new</code>/(setters)/<code>run</code> algorithm operating on a <code>&KmerIndex</code> (or <code>&mut</code> for
|
||||
<code>PartitionRouter</code>, not yet fixed — see (5)), <code>KmerIndex</code> itself holding no
|
||||
pipeline-orchestration logic anymore. Still not done: (5), the future
|
||||
cache crate, <code>obikalgorithm</code> (now unblocked — three real callback-bound
|
||||
variants observed, worth revisiting whether a single trait can express
|
||||
all three or whether that's itself the answer: it can't, and the trait
|
||||
should not force it).</p>
|
||||
<h2 id="8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename">(8) design agreed, not yet implemented (2026-08-21): <code>obikalgorithm::Algorithm</code> trait + <code>obikindexer::extensions</code> — private/public extension-trait split, <code>KmerLayer</code> rename</h2>
|
||||
<p>Session note: <code>Layer</code> was renamed <code>KmerLayer</code> (user, outside this
|
||||
conversation, alongside other naming homogenisation with <code>KmerIndex</code>/
|
||||
<code>KmerPartition</code>) — every reference to <code>Layer</code> in this doc from before
|
||||
2026-08-21 means today's <code>obikindex::layer::KmerLayer</code>.</p>
|
||||
<h3 id="why-this-came-up">Why this came up</h3>
|
||||
<p>Verifying "does <code>cmd/index</code> now rest entirely on the algorithm structs"
|
||||
(it doesn't quite — see below) led to sorting <code>KmerIndex</code>'s own methods by
|
||||
a criterion the user was explicit is <strong>semantic, not mechanical</strong>: "les
|
||||
méthodes qui, sémantiquement, n'ont pas d'intérêt hors de la construction
|
||||
de l'index" (methods that have no semantic interest outside index
|
||||
construction) — not "methods only called from <code>cmd/index</code> today," which
|
||||
a grep could answer but would miss methods construction-adjacent code
|
||||
elsewhere (<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>) also depends on for the
|
||||
same reason.</p>
|
||||
<p><strong>Checked, not assumed</strong> (grepped every call site before classifying):</p>
|
||||
<ul>
|
||||
<li><strong>Construction-only, real candidates for a private extension trait</strong>:
|
||||
<code>KmerIndex::{mark_scattered, mark_counted, mark_indexed, write_spectrum,
|
||||
build_index_layer, remove_build_artifacts, clear_output_for_create,
|
||||
create_skeleton, finalize_indexed, state}</code>. The last four are called
|
||||
from <code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> too (as
|
||||
precondition checks — "is my source <code>Indexed</code>?" — or shared
|
||||
skeleton/finalize machinery), not just from the 4-stage pipeline — so
|
||||
this extension trait's scope is "construction of any kind," not
|
||||
narrowly "the initial build pipeline."</li>
|
||||
<li><strong>Looked construction-only by name, checked, and kept on <code>KmerIndex</code></strong>:
|
||||
<code>layer_unitigs_path</code> (unitigs are the only way to recover a built
|
||||
index's kmer sequences — read by <code>rebuild_layer.rs</code> and others, well
|
||||
beyond construction — see [[project_unitigs_always_kept]]),
|
||||
<code>pack_matrices</code> (re-runnable maintenance on an already-finished index
|
||||
via <code>obikmer pack</code>, not just a pipeline step), <code>upgrade_layer_meta</code>
|
||||
(migration, runnable on any existing index at any time).</li>
|
||||
</ul>
|
||||
<h3 id="the-general-pattern-not-obikindexer-specific">The general pattern (not obikindexer-specific)</h3>
|
||||
<p><code>KmerIndex</code>/<code>KmerPartition</code>/<code>KmerLayer</code> stay generic, in <code>obikindex</code> —
|
||||
every domain-specific consumer crate gets to attach its own extension
|
||||
trait(s), of two kinds:</p>
|
||||
<ul>
|
||||
<li><strong>Private</strong> (<code>pub(crate)</code>, invisible outside the defining crate) — for
|
||||
plumbing only that crate's own algorithms need. <code>obikindexer</code> gets
|
||||
exactly one of these (see below); no public counterpart makes sense for
|
||||
it — "l'index est tellement central que le second trait n'a pas
|
||||
vraiment d'intérêt" for construction specifically: nothing external
|
||||
should ever want to call <code>mark_scattered</code> or <code>build_index_layer</code>.</li>
|
||||
<li><strong>Public</strong> — for a genuinely reusable domain extension. The user's own
|
||||
example, found while discussing this, not hypothetical: <code>obikindex/src/
|
||||
index/distance.rs</code> (phylogenetic distance metrics) is currently an
|
||||
<code>impl KmerIndex</code> block <strong>inside <code>obikindex</code> itself</strong> — under this
|
||||
principle it should be a public extension trait owned by <code>obikphylo</code>
|
||||
instead (distance metrics are a phylo concept, <code>obikindex</code> has no more
|
||||
business defining them than <code>obikindex::layer</code> has defining
|
||||
"family"/"minorant", the reasoning <code>SiblingLayerExt</code> already followed
|
||||
for <code>KmerLayer</code> — see <code>obikphylo/src/siblings/iter.rs</code>). <strong>Explicitly
|
||||
deferred</strong> — noted here so it isn't lost, not part of this round.</li>
|
||||
<li>The future cache-manager crate (still blocked on (5), see above) will
|
||||
add its own <strong>public</strong> extension trait mirroring part of <code>KmerIndex</code>'s/
|
||||
<code>KmerPartition</code>'s own read API in cached form (e.g. a cached
|
||||
<code>.partition(i)</code> that doesn't re-touch disk) — same pattern, third data
|
||||
point once built.</li>
|
||||
</ul>
|
||||
<h3 id="concretely-next-to-implement-two-items-in-order">Concretely, next to implement (two items, in order)</h3>
|
||||
<ol>
|
||||
<li><strong><code>obikindexer::extensions</code></strong> — a private (<code>pub(crate)</code>) extension
|
||||
trait, most likely named something like <code>IndexBuildExt</code> (final name
|
||||
not yet chosen), implemented for <code>KmerIndex</code>, carrying the ten methods
|
||||
listed above, moved out of <code>obikindex::index::{kmer_index,
|
||||
index_layer}</code>. Every algorithm in <code>obikindexer::algorithms::*</code> that
|
||||
currently calls <code>idx.mark_scattered()</code>/etc. keeps the same call syntax
|
||||
(extension trait methods are called the same way as inherent ones,
|
||||
just need the trait in scope) — <code>cmd/index/mod.rs</code> itself would need
|
||||
<code>use obikindexer::extensions::IndexBuildExt;</code> (or the module re-exports
|
||||
it) to keep compiling, since it's the one place outside <code>obikindexer</code>'s
|
||||
own algorithms that currently calls <code>mark_scattered</code>/<code>write_spectrum</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code> directly. <strong>Not yet decided</strong>: exact
|
||||
trait name, whether it's one trait or split further (e.g. sentinel
|
||||
marking vs. skeleton/finalize machinery), and whether <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> (not yet extracted into algorithms themselves) move
|
||||
onto it now too or keep calling the soon-to-be-inherent-no-longer
|
||||
methods some other way in the meantime — <strong>ask before implementing</strong>,
|
||||
this changes the blast radius significantly (4 more <code>obikindex</code>
|
||||
internal files depend on <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code>).</li>
|
||||
<li><strong><code>obikalgorithm::Algorithm</code> trait</strong> — new crate, the shared trait
|
||||
<code>obikpartitionner</code>→<code>obikindexer</code> merge (session start of 2026-08-21)
|
||||
and (6)/(7) were deliberately building toward, now with four real
|
||||
<code>new</code>/(setters)/<code>run</code> examples and three distinct callback-bound
|
||||
shapes to reconcile (plain <code>FnMut</code> for <code>PartitionRouter</code>, <code>FnMut +
|
||||
Send</code> for <code>LayerBuilder</code>, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code> —
|
||||
see (7)). Exact shape not yet drafted in this doc — do that as its own
|
||||
design pass before coding, same discipline as everything above.</li>
|
||||
</ol>
|
||||
<p>Both items: <strong>design only, nothing implemented yet</strong> — this section is
|
||||
the record to resume from, not a plan already executed.</p>
|
||||
<h2 id="9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented">(9) done (2026-08-21): <code>obikindexer::extensions::PrivateBuilder</code> — item 1 above, implemented</h2>
|
||||
<p>Renamed from <code>IndexBuilder</code> to <code>PrivateBuilder</code> immediately after (same
|
||||
session), freeing the name <code>IndexBuilder</code> for (10)'s public trait — read
|
||||
<code>IndexBuilder</code> below as <code>PrivateBuilder</code> throughout this section.</p>
|
||||
<p>Scoped down from (8)'s six-method list to the concrete set that's
|
||||
genuinely movable without further ripple — checked, not assumed, before
|
||||
writing anything:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_scattered</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_counted</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_indexed</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">write_spectrum</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">f0</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">f1</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">counts</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">BTreeMap</span><span class="o"><</span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="kt">u64</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">build_index_layer</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">min_ab</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="n">max_ab</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">with_counts</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="n">mode</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMode</span><span class="p">,</span><span class="w"> </span><span class="n">block_bits</span><span class="p">:</span><span class="w"> </span><span class="kt">u8</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">SKError</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">remove_build_artifacts</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">);</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>All six moved bodily out of <code>obikindex::index::{kmer_index, index_layer}</code>
|
||||
into <code>obikindexer::extensions</code> (new module, <code>pub(crate)</code>) — <code>index_layer.rs</code>
|
||||
is now empty and deleted outright.
|
||||
<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>
|
||||
stayed inherent on <code>KmerIndex</code>, per (8)'s reasoning: <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> — living <em>inside</em> <code>obikindex</code> itself — call them too,
|
||||
and <code>obikindex</code> can never depend on <code>obikindexer</code> to reach a trait defined
|
||||
there. Moving those four is real future work (extract
|
||||
merge/select/rebuild/reindex into algorithms first), not part of this
|
||||
step.</p>
|
||||
<p><strong>One new, small, deliberate API widening in <code>obikindex</code></strong>: <code>build_index_layer</code>
|
||||
depends on three helpers that were <code>pub(crate)</code> to <code>obikindex</code>
|
||||
(<code>graph_pipeline::{write_graph_as_unitigs, materialize_layer}</code>,
|
||||
<code>common::olm_to_sk</code>) — widened to <code>pub</code> (re-exported from <code>obikindex</code>'s
|
||||
crate root) so <code>obikindexer</code> could reach them. This is exactly the
|
||||
"enrich shared/lower-level APIs instead of ad hoc local code" call the
|
||||
project's own rules ask for, made explicitly rather than routed around:
|
||||
three functions, already generically written (no rewrite needed), now
|
||||
serve a second caller instead of being duplicated.</p>
|
||||
<p><strong>Why the trait had to be defined in <code>obikindexer</code>, not <code>obikindex</code></strong>:
|
||||
Rust's orphan rule — implementing a trait for a foreign type requires
|
||||
either the trait or the type to be local to the current crate. <code>KmerIndex</code>
|
||||
is foreign to <code>obikindexer</code>, so the trait must be the local half; if it
|
||||
were defined in <code>obikindex</code> instead, <code>pub(crate)</code> there would make it
|
||||
invisible to <code>obikindexer</code> too (crate-private means private to <em>that</em>
|
||||
crate, not "private except to one named dependent") — the opposite of
|
||||
what was wanted.</p>
|
||||
<p><strong>A real design decision made while wiring callers up, not a mechanical
|
||||
rename</strong>: <code>PrivateBuilder</code> being genuinely <code>pub(crate)</code> to <code>obikindexer</code>
|
||||
means <code>obikmer::cmd::index</code> (a different crate) can no longer call
|
||||
<code>mark_scattered</code>/<code>mark_counted</code>/<code>mark_indexed</code>/<code>write_spectrum</code> directly —
|
||||
it never could have, once privacy was real rather than aspirational. Each
|
||||
algorithm now marks its own completion as part of <code>run()</code>/<code>close()</code>
|
||||
instead of leaving it to the caller:
|
||||
- <code>PartitionRouter::close()</code> (not <code>run()</code>) calls <code>mark_scattered()</code> —
|
||||
<code>close()</code>, not <code>run()</code>, is the actual shared completion point between
|
||||
the file-driven <code>run()</code> path and the manual <code>write</code>/<code>write_batch</code>+
|
||||
<code>close()</code> path low-level callers (tests) use; putting it in <code>run()</code>
|
||||
alone would have silently skipped marking for every caller that never
|
||||
calls <code>run()</code>. <code>run()</code> already calls <code>self.close()</code> at its own end, so
|
||||
this covers both paths through one line, not two.
|
||||
- <code>Counter::run</code> calls <code>write_spectrum</code> then <code>mark_counted</code> before
|
||||
returning.
|
||||
- <code>LayerBuilder::run</code> calls <code>mark_indexed</code> before returning.</p>
|
||||
<p><code>cmd/index/mod.rs</code> lost all four direct calls (<code>mark_scattered</code>/
|
||||
<code>write_spectrum</code>/<code>mark_counted</code>/<code>mark_indexed</code>) — each stage's <code>if
|
||||
idx.state() < IndexState::X { ... }</code> block is now purely "run the
|
||||
algorithm," no separate bookkeeping call after it. Confirms, precisely
|
||||
this time (checked by re-reading the whole file, not assumed): <code>cmd/index</code>
|
||||
now rests on the four algorithms for every read/write of pipeline state
|
||||
except <code>KmerIndex::{exists, create, state, n_partitions}</code>, which are
|
||||
genuinely index-identity concerns, not construction bookkeeping — the
|
||||
original question this whole design pass started from.</p>
|
||||
<p>Same fix applied to <code>obikphylo</code>'s test harness (its four explicit
|
||||
<code>mark_*</code>/<code>write_spectrum</code> calls removed, relying on the algorithms now
|
||||
doing it themselves) — <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test needed no change (never called <code>mark_*</code> directly).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the CLI smoke test — 870
|
||||
kmers, same as (6)/(7).</p>
|
||||
<p>Still not done at the time of writing: item 2 from (8) (<code>obikalgorithm::
|
||||
Algorithm</code>), (5), the future cache crate, the <code>distance.rs</code> →
|
||||
<code>obikphylo</code> relocation (noted in (8), explicitly deferred), and
|
||||
extracting <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code> into algorithms.</p>
|
||||
<h2 id="10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session">(10) done (2026-08-21): <code>obikindex::IndexBuilder</code> — the public counterpart, same session</h2>
|
||||
<p>Immediate correction to (9): the private trait built there was renamed
|
||||
<code>PrivateBuilder</code> (freeing the name), and the four methods (9) had left
|
||||
inherent on <code>KmerIndex</code> — <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code> — got their own trait after all: <strong><code>IndexBuilder</code></strong>,
|
||||
public, defined in <code>obikindex</code> itself (not <code>obikindexer</code>):</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="p">:</span><span class="w"> </span><span class="nb">Sized</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">clear_output_for_create</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">force</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">create_skeleton</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">meta</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMeta</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">finalize_indexed</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">rep</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">Reporter</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">state</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">IndexState</span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>User's framing: these four are "maintenance", not "scientific computation
|
||||
on an index" — a different kind of non-generic-ness than (9)'s six
|
||||
(<code>mark_*</code>/<code>write_spectrum</code>/<code>build_index_layer</code>/<code>remove_build_artifacts</code>,
|
||||
exclusive to the 4-stage pipeline). Maintenance is used more broadly
|
||||
(<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>), so it gets a real, public trait —
|
||||
not folded back into <code>KmerIndex</code>'s inherent surface, and not private
|
||||
either.</p>
|
||||
<p><strong>Where it lives, and why that's not arbitrary</strong>: (9) needed the orphan
|
||||
rule to force its trait into <code>obikindexer</code>, to achieve genuine
|
||||
crate-private visibility. Here the requirement is the opposite:
|
||||
<code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> — the trait's own
|
||||
heaviest users — live <em>inside</em> <code>obikindex</code>. A trait they need to reach
|
||||
must be local to <code>obikindex</code> (or a crate <code>obikindex</code> itself depends on,
|
||||
which doesn't exist for this). So <code>IndexBuilder</code> lives in a new
|
||||
<code>obikindex/src/index/builder.rs</code>, <code>pub trait</code> (no orphan-rule tension at
|
||||
all here — both trait and type are local to the same crate), re-exported
|
||||
from <code>obikindex</code>'s crate root alongside <code>PrivateBuilder</code>'s sibling
|
||||
<code>obikindexer::extensions::PrivateBuilder</code> staying where it is. Two
|
||||
traits, two crates, two different reasons, not a contradiction.</p>
|
||||
<p><strong>Blast radius, all inside <code>obikindex</code> plus one external crate</strong>: every
|
||||
internal caller of these four methods needs the trait imported now that
|
||||
they're no longer inherent — <code>merge.rs</code>, <code>select.rs</code>, <code>rebuild.rs</code>,
|
||||
<code>reindex.rs</code> (<code>use crate::index::builder::IndexBuilder;</code>) and, externally,
|
||||
<code>obikmer::cmd::index::mod</code> (<code>use obikindex::IndexBuilder;</code>, for the three
|
||||
<code>idx.state() < IndexState::X</code> resumability checks). Call syntax at every
|
||||
site is unchanged (<code>KmerIndex::create_skeleton(...)</code>,
|
||||
<code>self.state()</code>) — only trait-in-scope requirements are new, which is
|
||||
exactly the point: same ergonomics, less surface baked into <code>KmerIndex</code>
|
||||
itself.</p>
|
||||
<p>Verification went one step further than (9): beyond
|
||||
<code>cargo check --workspace --all-targets</code> + <code>cargo test --workspace</code> +
|
||||
<code>scripts/smoke_test_index.sh</code> (all green, 870 kmers again), ran
|
||||
<code>obikmer merge</code> end to end on two freshly built indexes (exercises
|
||||
<code>clear_output_for_create</code>/<code>finalize_indexed</code> directly, the two methods
|
||||
<code>scripts/smoke_test_index.sh</code> itself never touches) — exit 0, <code>pack</code>
|
||||
stage completed. Test suite alone would not have caught a regression
|
||||
here: no existing test builds two real indexes and merges them through
|
||||
the CLI.</p>
|
||||
<p><code>KmerIndex</code> itself now carries only: identity/config accessors
|
||||
(<code>root_path</code>/<code>meta</code>/<code>kmer_size</code>/<code>n_bits</code>/<code>evidence_mode</code>/<code>genomes</code>/...),
|
||||
path resolution (<code>partition_dir</code>/<code>index_dir</code>/<code>layer_dir</code>/
|
||||
<code>partition_meta</code>/<code>n_layers</code>), and a few index-maintenance operations not
|
||||
yet sorted into either trait (<code>layer_unitigs_path</code>, <code>pack_matrices</code>,
|
||||
<code>upgrade_layer_meta</code> — see (8)'s "tested and discarded" list; still
|
||||
correctly inherent, not construction-only by the semantic criterion) —
|
||||
<code>create</code>/<code>open</code>/<code>exists</code> (identity, can't be trait methods needing <code>Self</code>
|
||||
before one exists) round that out.</p>
|
||||
<h2 id="11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files">(11) done (2026-08-21): <code>KmerIndex</code>/<code>IndexMeta</code> made fully stateless, <code>IndexState</code> moved off sentinel files</h2>
|
||||
<p>Triggered mid-discussion of <code>obikalgorithm::Algorithm</code> (still not started —
|
||||
see "Still not done" below): user asked why <code>PartitionRouter::new</code> still
|
||||
took <code>&mut KmerIndex</code> at all, and questioned whether <code>mark_scattered</code>
|
||||
belonged in the algorithm or in <code>cmd/index</code>. Investigation found <code>&mut</code>
|
||||
had become <em>newly</em> necessary since (9) — <code>mark_scattered</code> was mutating
|
||||
<code>self.meta.genomes</code> in memory so <code>Counter</code>'s later <code>write_spectrum</code> call
|
||||
(same <code>idx</code> instance) would see the derived label. User's resolution: the
|
||||
"disk is truth, stateless" principle already agreed for <code>KmerPartition</code>/
|
||||
<code>Layer</code> in (5) (still unimplemented for those two) should extend to
|
||||
<code>KmerIndex</code> itself — move <code>IndexState</code> (<code>Empty</code>/<code>Scattered</code>/<code>Counted</code>/
|
||||
<code>Indexed</code>) off the three sentinel files (<code>scatter.done</code>/<code>count.done</code>/
|
||||
<code>index.done</code>, detected by existence) and into a field of <code>index.meta</code>'s
|
||||
own JSON, so the <code>mark_*</code> calls become plain disk writes an algorithm can
|
||||
legitimately make on <code>&self</code> — no in-memory mutation left to protect.</p>
|
||||
<p><strong>Shape of <code>IndexMeta</code>, per the user's explicit spec</strong>: one JSON file per
|
||||
index (<code>index.meta</code>), one <code>IndexMeta</code> instance per index, held and
|
||||
returned as <code>Arc<IndexMeta></code> (not <code>&IndexMeta</code>) by <code>KmerIndex::meta()</code>.
|
||||
<code>config</code> (<code>kmer_size</code>/<code>minimizer_size</code>/<code>n_bits</code>/<code>with_counts</code>/<code>evidence</code>/
|
||||
<code>block_bits</code>) is fixed at construction, cached as a <code>pub</code> field (getter
|
||||
kept alongside, for symmetry) — "les champs constants restent des champs
|
||||
de la structure", read once, never re-read from disk. <code>genomes</code> and
|
||||
<code>state</code> are the opposite: no in-memory cache at all, every accessor
|
||||
(<code>genomes()</code>, <code>state()</code>) re-reads <code>index.meta</code> from disk, every mutator
|
||||
(<code>push_genome</code>/<code>rename_genome</code>/<code>set_genomes</code>/<code>set_state</code>/<code>mark_scattered</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code>) does a full read-modify-write of the same
|
||||
file. An internal <code>std::sync::RwLock<()></code> is held across each
|
||||
read-modify-write sequence (not just the write) so two callers sharing
|
||||
the same <code>Arc<IndexMeta></code> can't lose an update to each other — this is
|
||||
<em>not</em> a cross-process lock (that's <code>obisys::DirLock</code>, already held by
|
||||
<code>cmd/index</code> for the whole build); it only serialises access through one
|
||||
shared in-process instance.</p>
|
||||
<p><strong>Construction, chain-of-responsibility style, matching (5)'s pattern</strong>:
|
||||
<code>IndexMeta::create(&KmerIndex, config, genomes)</code> / <code>IndexMeta::open(&KmerIndex)</code>
|
||||
ask the index for its own root path rather than taking one directly. Since
|
||||
<code>KmerIndex::create</code> doesn't have a complete <code>KmerIndex</code> yet to hand in
|
||||
(it's what's being built), added lower-level <code>pub(crate)</code> path-based
|
||||
primitives <code>create_at(&Path, ...)</code> / <code>open_at(&Path)</code> that <code>KmerIndex::create</code>/
|
||||
<code>open</code> and <code>builder.rs</code>'s <code>create_skeleton</code> call directly, bypassing the
|
||||
convenience wrappers for that one bootstrap case.</p>
|
||||
<p><strong>The one deliberate exception</strong>: <code>select_in_place</code> and <code>reindex</code>
|
||||
genuinely rewrite <code>config</code> after an index already exists (output
|
||||
type/evidence mode changes in place) — contradicting "config never
|
||||
changes" for the general case. Resolved with a separate, explicitly
|
||||
rare-labelled <code>IndexMeta::rewrite_config(config, genomes)</code> (preserves
|
||||
<code>state</code>, overwrites everything else); callers refresh their own cached
|
||||
<code>Arc<IndexMeta></code> afterward (<code>self.meta = Arc::new(IndexMeta::open(self)?)</code>)
|
||||
since <code>IndexMeta</code> has no way to reach back into whichever <code>KmerIndex</code>
|
||||
holds it.</p>
|
||||
<p><strong>Consequence confirmed, not just hoped for</strong>: with <code>mark_scattered</code> no
|
||||
longer touching anything in memory, <code>PartitionRouter</code> genuinely never
|
||||
needs <code>&mut KmerIndex</code> — <code>PartitionRouter<'a> { index: &'a KmerIndex }</code>,
|
||||
<code>new(&'a KmerIndex)</code>. This is effectively the <code>PartitionRouter</code> half of
|
||||
(5)'s "order of remaining work" item done as a side effect; <code>KmerPartition</code>/
|
||||
<code>Layer</code> themselves are still unimplemented for (5).</p>
|
||||
<p><strong>Blast radius — much larger than (9)/(10), touched nearly every crate</strong>:
|
||||
every <code>.meta().genomes</code>/<code>.meta.genomes</code> field access became a fallible
|
||||
<code>.genomes()?</code> method call (<code>genomes</code> reads <code>io::Result<Vec<GenomeInfo>></code>
|
||||
now, not a field), and <code>.meta_mut()</code> was removed outright (no more direct
|
||||
field mutation from outside <code>IndexMeta</code>). Fixed across:
|
||||
- <code>obikindex</code> internals: <code>meta.rs</code>/<code>state.rs</code>/<code>kmer_index.rs</code>/<code>builder.rs</code>
|
||||
(full rewrites), <code>reindex.rs</code>/<code>select.rs</code> (switched to <code>rewrite_config</code>),
|
||||
<code>merge.rs</code> (heaviest single file — genome counts precomputed once per
|
||||
source into a <code>Vec<Vec<GenomeInfo>></code> up front rather than re-reading
|
||||
<code>index.meta</code> from disk repeatedly through the function, sentinel write
|
||||
replaced with <code>dst2.meta.mark_indexed()</code>), <code>stats.rs</code>, <code>distance.rs</code>,
|
||||
<code>dump.rs</code>, <code>predicate.rs</code> (its <code>IndexMeta</code>-inherent <code>matching_genome_indices</code>/
|
||||
<code>build_group_filter</code> now read genomes fresh internally), <code>mod.rs</code>/<code>lib.rs</code>
|
||||
(sentinel constant re-exports removed — <code>IndexState</code> no longer has
|
||||
<code>SENTINEL_*</code>/<code>detect()</code> at all).
|
||||
- <code>obikindexer::extensions::PrivateBuilder</code>: <code>mark_scattered</code> signature
|
||||
dropped <code>&mut self</code> → <code>&self</code>; the four <code>mark_*</code>/<code>write_spectrum</code> bodies
|
||||
became one-line delegations to <code>self.meta().mark_*()</code>.
|
||||
- <code>obikphylo::siblings</code>: <code>alignment.rs</code>/<code>cardinality.rs</code>/<code>distance.rs</code>/
|
||||
<code>entropy.rs</code>/<code>sankoff_bundle.rs</code>/<code>stats.rs</code>/<code>tests.rs</code> — all had
|
||||
<code>self.meta().genomes.len()</code>-shaped reads, mechanically fixed to
|
||||
<code>.genomes().map_err(OKIError::Io)?.len()</code> (tests: <code>.unwrap()</code>).
|
||||
- <code>obikmer::cmd::*</code>: <code>annotate</code> (rewrote its rename path to load genomes
|
||||
once, mutate the in-memory <code>Vec</code>, then <code>idx.meta().set_genomes(...)</code>
|
||||
instead of <code>meta_mut()</code>), <code>filter</code>/<code>pack</code>/<code>dump</code>/<code>unitig</code>/<code>merge</code>/<code>select</code>/
|
||||
<code>phylo</code> (fetch-once-then-use pattern for genome counts/labels),
|
||||
<code>utils/maintenance.rs</code> (<code>run_rename</code> now calls the pre-existing
|
||||
<code>IndexMeta::rename_genome</code>, dropping its own hand-rolled field mutation
|
||||
entirely), <code>index/mod.rs</code> (three <code>idx.state() < IndexState::X</code>
|
||||
resumability checks needed a fallible read — factored into a small
|
||||
<code>current_state(&KmerIndex) -> IndexState</code> helper rather than repeating
|
||||
the same <code>unwrap_or_else</code> three times), <code>query/*</code> (<code>emit_batch</code>'s
|
||||
signature changed from <code>&IndexMeta</code> to <code>&[GenomeInfo]</code>, and <code>genomes</code> is
|
||||
now fetched once in <code>run()</code> and threaded down through <code>process_chunk</code>
|
||||
as <code>Arc<Vec<GenomeInfo>></code> rather than re-reading <code>index.meta</code> from disk
|
||||
on every chunk — a deliberate deviation from the "always re-read"
|
||||
default, justified because this is a genuine per-chunk hot path, unlike
|
||||
every other call site touched in this pass).
|
||||
- One <code>&IndexMeta</code>-vs-<code>Arc<IndexMeta></code> argument-type mismatch pattern
|
||||
recurred at several CLI call sites (<code>build_filters</code>/<code>build_specs</code>/
|
||||
<code>emit_batch</code>'s original signature) — resolved via <code>Arc</code>'s deref
|
||||
coercion (<code>&idx.meta()</code> coerces to <code>&IndexMeta</code>) rather than changing
|
||||
every downstream signature to accept <code>Arc<IndexMeta></code>.</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and
|
||||
<code>cargo test --workspace</code> both green (0 failures) after the full
|
||||
propagation, <code>scripts/smoke_test_index.sh</code> green (870 kmers, same as every
|
||||
prior round), plus a manual CLI run of <code>index</code> (×2) → <code>merge</code> → <code>select</code>
|
||||
→ <code>reindex</code> → <code>utils --new-label</code> (rename) → <code>utils --stats</code>, all exit 0,
|
||||
confirming the four most-affected commands (the ones (10)'s verification
|
||||
already flagged as under-covered by the automated test suite) still work
|
||||
end to end against the new <code>Arc<IndexMeta></code>/on-disk-<code>IndexState</code> shape.</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign itself
|
||||
(only the <code>PartitionRouter</code>-<code>&mut</code>-removal piece landed, as a side
|
||||
effect); the <code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly
|
||||
deferred); the future cache-manager crate. (8)'s <code>obikalgorithm::
|
||||
Algorithm</code> trait, resumed and closed in (12) below.</p>
|
||||
<h2 id="12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session">(12) done (2026-08-21): <code>obikalgorithm::Algorithm</code> — the shared trait, resumed and closed in one session</h2>
|
||||
<p>Resumed (8)'s point 2 through a point-by-point discussion of what's
|
||||
actually common across the four pipeline algorithms, now that (11) made
|
||||
<code>KmerIndex</code> itself immutable everywhere. Four sub-points, each closed
|
||||
before moving to the next:</p>
|
||||
<p><strong>1. Receiver (<code>&self</code> vs <code>&mut self</code>)</strong> — investigated whether (11)'s
|
||||
removal of <code>&mut KmerIndex</code> also removed the need for <code>PartitionRouter::
|
||||
run</code> to take <code>&mut self</code>. It didn't: <code>PartitionRouter</code> holds real
|
||||
per-run state of its own (<code>writers: Vec<Option<SKFileWriter>></code>, open file
|
||||
handles, purely in-process RAM — confirmed by checking where <code>writers</code> is
|
||||
stored, nothing to do with <code>KmerIndex</code>/disk truth), unrelated to the
|
||||
index. First proposal (wrap <code>writers</code> in <code>RefCell</code> so all four could
|
||||
share a uniform <code>&self</code>) was retracted on pushback: manufacturing
|
||||
interior mutability with runtime borrow checks to satisfy a cosmetic
|
||||
uniformity that Rust doesn't even require is over-engineering — a trait
|
||||
method's receiver must match exactly across implementors, but nothing
|
||||
stops that shared receiver from being <code>&mut self</code> with three of the four
|
||||
implementations simply not using the mutability. Settled: trait declares
|
||||
<code>&mut self</code>; <code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> (previously <code>&self</code>)
|
||||
now also take <code>&mut self</code>, unused.</p>
|
||||
<p><strong>2. <code>path_source</code> as a <code>PartitionRouter</code> setter, not a <code>run()</code> param</strong> —
|
||||
added a <code>files: Option<Box<dyn Iterator<Item = PathBuf> + Send>></code> field +
|
||||
<code>.files(impl Iterator<Item = PathBuf> + Send + 'static) -> Self</code> setter
|
||||
(boxed rather than a generic type parameter on <code>PartitionRouter<'a></code>:
|
||||
negligible cost — one <code>PathBuf</code> per input <em>file</em>, not per k-mer — for a
|
||||
much more usable type when passing the builder around). <code>run</code> now does
|
||||
<code>self.files.take().ok_or_else(...)</code>, erroring if <code>.files(...)</code> was never
|
||||
called, instead of taking <code>path_source</code> as a parameter.</p>
|
||||
<p><strong>3. Unifying the three progress-callback bound shapes</strong> — reopened, then
|
||||
resolved differently than (8) originally framed it. First proposal
|
||||
(force everything to <code>FnMut(Progress) + Send</code>) was rejected on the same
|
||||
principle as point 1: <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn(Progress) + Sync</code>
|
||||
isn't arbitrary — their callback is invoked concurrently from multiple
|
||||
rayon worker threads, and <code>FnMut</code> requires exclusive access, so forcing
|
||||
it would mean wrapping the callback in a <code>Mutex</code> for zero benefit at the
|
||||
one real call site (<code>pb.inc(1)</code>, already thread-safe). The actual
|
||||
resolution: move <code>on_progress</code> off <code>run()</code>'s signature entirely, onto a
|
||||
per-algorithm <code>.on_progress(...)</code> setter — same treatment as point 2's
|
||||
<code>path_source</code> — so each algorithm keeps its own bound (<code>PartitionRouter</code>:
|
||||
<code>FnMut(Progress) + 'a</code>, sequential, no <code>Send</code> needed; <code>LayerBuilder</code>:
|
||||
<code>FnMut(Progress) + Send + 'a</code>, crosses into <code>PartitionRunner</code>'s
|
||||
<code>thread::scope</code>-spawned controller thread once; <code>Dereplicator</code>/<code>Counter</code>:
|
||||
<code>Fn(Progress) + Sync + 'a</code>, invoked concurrently from rayon workers).
|
||||
This <em>also</em> dissolves the original problem <code>run()</code> had: once the
|
||||
callback isn't part of <code>run</code>'s signature at all, there's nothing left to
|
||||
unify there, and point 4 (below) becomes trivial.</p>
|
||||
<p><strong>4. <code>Output</code> as an associated type, <code>Error</code> fixed</strong> — trivial once (3)
|
||||
moved the callback out: <code>Error</code> was already uniform (all four return
|
||||
<code>obiskio::SKResult<T></code> = <code>Result<T, SKError></code>), only <code>Output</code> varied
|
||||
(<code>()</code>/<code>()</code>/<code>KmerSpectrum</code>/<code>usize</code>). First cut reused <code>obiskio::SKResult</code>
|
||||
directly as the trait's return type — <strong>caught and corrected the same
|
||||
session</strong>: <code>SKError</code> enumerates I/O-specific cases (<code>BadMagic</code>/
|
||||
<code>Truncated</code>/<code>Compression</code>/...), meaningless at the level of a generic
|
||||
"algorithm" abstraction, and borrowing it made <code>obikalgorithm</code> — meant to
|
||||
be minimal and neutral — depend on a low-level I/O crate purely to reuse
|
||||
its error type. Textbook instance of the "petits pois" failure mode
|
||||
(patch around a convenient existing type instead of asking what this
|
||||
crate should actually own). Fixed to a genuinely generic, boxed error
|
||||
type owned by <code>obikalgorithm</code> itself:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// obikalgorithm — no dependency on obiskio or any other crate</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">error</span><span class="p">::</span><span class="n">Error</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">result</span><span class="p">::</span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">Error</span><span class="o">></span><span class="p">;</span>
|
||||
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Algorithm</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Output</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="bp">Self</span><span class="p">::</span><span class="n">Output</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Any concrete error (<code>SKError</code>, <code>std::io::Error</code>, ...) converts
|
||||
automatically via <code>?</code>, through <code>std</code>'s own blanket <code>impl<E: Error + Send
|
||||
+ Sync> From<E> for Box<dyn Error + Send + Sync></code> — no custom <code>From</code> impl
|
||||
needed, no dependency on the crate that defines the concrete error type.
|
||||
The four algorithms' <code>run</code> bodies needed no change beyond the signature's
|
||||
return type (every existing <code>?</code> on an <code>SKError</code>-returning subcall keeps
|
||||
compiling, converting through the same blanket impl at the boundary).</p>
|
||||
<p><code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> each <code>impl
|
||||
Algorithm for X<'_> { type Output = ...; fn run(&mut self) -> obikalgorithm::Result<...> { ... } }</code>
|
||||
— the old inherent <code>run</code> methods were removed outright (not kept as
|
||||
duplicates), so callers now <code>use obikalgorithm::Algorithm;</code> to call
|
||||
<code>.run()</code>. Every field-lifetime-bound boxed callback (<code>Box<dyn
|
||||
FnMut(Progress) + 'a></code> etc.) is tied to the algorithm's own <code>'a</code> (the
|
||||
<code>&'a KmerIndex</code> lifetime already on the struct), not <code>'static</code> — avoids
|
||||
forcing callers' progress closures to <code>move</code>-capture (and therefore clone
|
||||
or <code>Arc</code>-wrap) local state like <code>TracedBar</code>/EMA-rate accumulators that
|
||||
they'd otherwise want to keep using by reference after <code>run()</code> returns.</p>
|
||||
<p><strong>Why a new crate, not a submodule of <code>obikindexer</code></strong>: <code>obikmer::cmd::
|
||||
index::mod</code> and <code>obikphylo</code>'s own test helpers both need to call <code>.run()</code>
|
||||
on these algorithms, so the trait has to be reachable from outside
|
||||
<code>obikindexer</code> — putting it in <code>obikindexer</code> itself would work file-wise
|
||||
but conflates "the trait every algorithm implements" with "one crate's
|
||||
particular four implementations of it", the same reasoning that already
|
||||
separated <code>obikindex</code> (data model) from <code>obikindexer</code> (algorithms
|
||||
operating on it). <code>obikalgorithm</code> has <strong>no dependencies at all</strong> (see
|
||||
above); <code>obikindexer</code>, <code>obikmer</code>, and <code>obikphylo</code> (dev-dependency, for
|
||||
its test helper) all depend on it.</p>
|
||||
<p><strong>Blast radius</strong>: <code>obikindexer</code>'s four algorithm modules (struct field +
|
||||
setter + trait impl each); <code>obikmer::cmd::index::mod</code> (three call sites:
|
||||
<code>.on_progress(cb)</code> before <code>.run()</code>, unqualified now that the trait is in
|
||||
scope); <code>obikindexer::algorithms::partitionner::tests</code> and <code>obikphylo::
|
||||
siblings::tests</code> (both had direct <code>.run(None::<fn(Progress)>)</code>-shaped
|
||||
calls needing the same treatment). New <code>obikalgorithm</code> crate registered
|
||||
in the workspace <code>Cargo.toml</code>, depended on by <code>obikindexer</code>/<code>obikmer</code>
|
||||
(regular) and <code>obikphylo</code> (dev).</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and <code>cargo test
|
||||
--workspace</code> both green (0 failures), <code>scripts/smoke_test_index.sh</code> green
|
||||
(870 kmers, same as every prior round) — this round didn't repeat the
|
||||
manual <code>merge</code>/<code>select</code>/<code>reindex</code> CLI exercise from (11), since nothing
|
||||
in this pass touched those commands' code paths (only the four pipeline
|
||||
algorithms and <code>cmd::index</code>, already covered by the smoke test). Reverified
|
||||
after the <code>obiskio</code>-dependency fix above (same three checks, still green,
|
||||
<code>obikalgorithm/Cargo.toml</code> now has zero <code>[dependencies]</code>).</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign; the
|
||||
<code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly deferred); the
|
||||
future cache-manager crate (mentioned in (8) as a later, mirrored
|
||||
extension-trait exercise, not started).</p>
|
||||
<h2 id="the-problem">The problem</h2>
|
||||
<p>Reading a layer's data (MPHF + matrix) is not free: <code>MphfLayer::open</code> mmaps
|
||||
<code>mphf.bin</code> plus (<code>evidence.bin</code>/<code>fingerprint.bin</code> + <code>unitigs.bin</code>), and the
|
||||
|
||||
Binary file not shown.
@@ -909,6 +909,56 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -2195,6 +2245,56 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -4261,6 +4361,249 @@ among the survivors) — a single extra pass is sufficient.</p>
|
||||
<code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
|
||||
from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
|
||||
workspace <code>cargo test</code> green.</p>
|
||||
<h2 id="-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28"><code>--distance</code> unification: SNP corrections as first-class metrics (implemented, 2026-08-28)</h2>
|
||||
<p><strong>Implemented.</strong> <code>--metric</code> (renamed <code>--distance</code> — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
<code>--metric</code> was a misnomer) gains a family of <code>snp-*</code> values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
machinery (<code>PairwiseTally</code>, <code>obikphylo::siblings::algorithms::pairwise</code>)
|
||||
instead of <code>cache.distance(...)</code>'s existing per-layer traversal — a different
|
||||
code path behind the same CLI surface, not just another branch of one
|
||||
formula function.</p>
|
||||
<p><strong>Why unify at the CLI level despite the implementation split</strong>: phylogenetically
|
||||
a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
|
||||
to how the matrix was produced, so exposing it as a special-cased subcommand
|
||||
instead of a <code>--distance</code> value would misrepresent its role. The
|
||||
implementation divergence (sibling-annex-based vs. plain index scan) is real
|
||||
but belongs at the routing layer, invisible to the CLI's own vocabulary.</p>
|
||||
<p><strong><code>--subsample</code> becomes optional for <code>snp-*</code> distances</strong> (it stays mandatory
|
||||
for <code>--sankoff</code>/<code>--pseudo-alignment</code>, unrelated commands): absent means
|
||||
exhaustive, achieved for free by reusing <code>sample_index</code>'s existing
|
||||
proportional-per-layer-quota mechanism with <code>n</code> set to the index-wide total
|
||||
non-monomorphic-minorant count (already available from the sibling-annex
|
||||
stats) — every layer's quota then equals its own full count, giving Bernoulli
|
||||
<code>p = 1</code> everywhere, i.e. every eligible family is drawn. No second,
|
||||
exhaustive-only driver needed. Present means sampled, exactly as <code>--sankoff</code>
|
||||
already behaves.</p>
|
||||
<p><strong>One shared tally, many derived formulas.</strong> <code>PairwiseTally</code>'s <code>subst[4][4]</code>
|
||||
per-pair substitution counts (plus marginal base frequencies derived from it)
|
||||
are the sufficient statistic for every closed-form correction below — each
|
||||
is a small pure function <code>PairwiseTally -> Array2<f64></code>, at the same level as
|
||||
the already-implemented <code>raw_snp_distance</code>/<code>base_pair_tally</code>/
|
||||
<code>cardinality_tally</code>. No new full scan per formula, whether the tally itself
|
||||
was built exhaustively or from a subsample.</p>
|
||||
<p><strong><code>--raw-snp-counts</code> stays a separate, unrelated flag</strong> — same underlying
|
||||
tally, but a diagnostic (<code>n_snp</code>/<code>n_shared</code>/<code>n_eligible</code> per genome pair, one
|
||||
row per pair) rather than a distance value, and its long-table shape doesn't
|
||||
fold into a single N×N matrix the way a distance does. No change to its
|
||||
existing CSV format.</p>
|
||||
<h3 id="snp-distance-catalog"><code>snp-*</code> distance catalog</h3>
|
||||
<p>All closed-form (method-of-moments / direct formula), none requiring
|
||||
per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
|
||||
<em>tree</em>-ML usage but not its <em>pairwise</em> estimator, which is closed-form like
|
||||
F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>value</th>
|
||||
<th>corrects for</th>
|
||||
<th>inputs beyond raw counts</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>snp-raw</code></td>
|
||||
<td>nothing (uncorrected p-distance)</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-jc</code> (Jukes-Cantor, JC69)</td>
|
||||
<td>multiple substitutions per site</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k2p</code> (Kimura 2-parameter, K80)</td>
|
||||
<td>+ transition/transversion rate bias</td>
|
||||
<td>ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k81</code> (Kimura 3-parameter, K3ST)</td>
|
||||
<td>+ splits transversions into 2 categories</td>
|
||||
<td>ts/tv split, by category</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f81</code> (Felsenstein 81)</td>
|
||||
<td>+ unequal base frequencies (no ts/tv split)</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tajima-nei</code> (Tajima-Nei 1984)</td>
|
||||
<td>same goal as F81 (equal-input model), different formula, better small-sample behavior</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-t92</code> (Tamura 3-parameter)</td>
|
||||
<td>K2P + GC-content bias</td>
|
||||
<td>ts/tv split, GC content</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f84</code> (Felsenstein 84)</td>
|
||||
<td>full empirical base freqs + single ts/tv rate</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-hky85</code> (Hasegawa-Kishino-Yano, pairwise estimator)</td>
|
||||
<td>same inputs as F84, different formula</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tn93</code> (Tamura-Nei)</td>
|
||||
<td>full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate</td>
|
||||
<td>empirical base freqs, purine-ts/pyrimidine-ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-logdet</code> (LogDet / paralinear)</td>
|
||||
<td>no shared-model or stationarity assumption at all — general divergence-matrix determinant</td>
|
||||
<td>full empirical 4×4 divergence matrix (already <code>subst[4][4]</code>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tv</code> (transversions-only p-distance)</td>
|
||||
<td>diagnostic/deep-divergence variant — drops transitions entirely (they saturate first)</td>
|
||||
<td>tv-only counts</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
||||
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
||||
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
|
||||
by the user (<code>--gamma-shape <alpha></code>), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.</p>
|
||||
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
||||
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
||||
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
||||
<code>snp-logdet</code> are catalogued above but <strong>not implemented</strong>: <code>snp-logdet</code>
|
||||
needs the true <em>directional</em> per-pair base co-occurrence matrix
|
||||
(<code>PairwiseTally</code> only keeps the symmetrised substitution counts
|
||||
<code>BasePairTally</code> itself wants — see <code>snp_distance.rs</code>'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), <code>snp-tajima-nei</code> needs each genome's <em>own</em> base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
<code>snp-f84</code>/<code>snp-hky85</code> had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against <a href="https://github.com/emmanuelparadis/ape">ape</a>'s own
|
||||
<code>src/dist_dna.c</code>, not re-derived from memory). Adding any of these later is
|
||||
a new function in <code>obikphylo::siblings::algorithms::snp_distance</code>, plus for
|
||||
<code>snp-logdet</code>/<code>snp-tajima-nei</code> a new field on <code>PairStats</code>/a per-genome
|
||||
accumulator — not an architecture change.</p>
|
||||
<h3 id="exact-formulas-implemented-2026-08-28">Exact formulas (implemented, 2026-08-28)</h3>
|
||||
<p>Sufficient statistic, per genome pair <code>(i, j)</code>, from
|
||||
<code>PairwiseTally::categories</code>/<code>PairwiseTally::base_freq</code> (base order always
|
||||
<code>0=A, 1=C, 2=G, 3=T</code>, matching <code>FamilyMask</code>/<code>STATE_SYMBOL</code>):</p>
|
||||
<ul>
|
||||
<li><span class="arithmatex">\(n_{ts1}\)</span>: A↔G substitutions (purine transitions), <span class="arithmatex">\(n_{ts2}\)</span>: C↔T
|
||||
(pyrimidine transitions)</li>
|
||||
<li><span class="arithmatex">\(n_{tv1}\)</span>: A↔C and G↔T substitutions, <span class="arithmatex">\(n_{tv2}\)</span>: A↔T and C↔G
|
||||
(Kimura's two transversion categories)</li>
|
||||
<li><span class="arithmatex">\(n_{shared}\)</span>: loci where both genomes agree</li>
|
||||
<li><span class="arithmatex">\(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\)</span> (total eligible
|
||||
loci for the pair)</li>
|
||||
<li><span class="arithmatex">\(\pi_A, \pi_C, \pi_G, \pi_T\)</span>: pair-pooled base frequencies,
|
||||
<span class="arithmatex">\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)</span>
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)</li>
|
||||
</ul>
|
||||
<p>Derived proportions used below:</p>
|
||||
<div class="arithmatex">\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]</div>
|
||||
<p>Every formula below was checked term-by-term against <code>ape</code>'s own
|
||||
<code>src/dist_dna.c</code> (not re-derived from memory) before being ported to
|
||||
<code>obikphylo::siblings::algorithms::snp_distance</code>.</p>
|
||||
<p><strong><code>snp-raw</code></strong> — uncorrected p-distance:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{raw} = p
|
||||
\]</div>
|
||||
<p><strong><code>snp-tv</code></strong> — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, <em>is</em> the correction):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{tv} = Q
|
||||
\]</div>
|
||||
<p><strong><code>snp-jc</code></strong> (Jukes-Cantor, JC69):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-k2p</code></strong> (Kimura 2-parameter, K80), with <span class="arithmatex">\(a_1 = 1-2P-Q\)</span>, <span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-k81</code></strong> (Kimura 3-parameter, K3ST), with <span class="arithmatex">\(a_1 = 1-2P-2Q_1\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2P-2Q_2\)</span>, <span class="arithmatex">\(a_3 = 1-2Q_1-2Q_2\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-f81</code></strong> (Felsenstein 81), with <span class="arithmatex">\(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-t92</code></strong> (Tamura 3-parameter), with GC content
|
||||
<span class="arithmatex">\(g = \pi_C+\pi_G\)</span>, <span class="arithmatex">\(w = 2g(1-g)\)</span>, <span class="arithmatex">\(a_1 = 1 - \dfrac{P}{w} - Q\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-tn93</code></strong> (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
<span class="arithmatex">\(g_R = \pi_A+\pi_G\)</span>, <span class="arithmatex">\(g_Y = \pi_C+\pi_T\)</span>, and</p>
|
||||
<div class="arithmatex">\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]</div>
|
||||
<p><strong><code>+Γ</code> gamma correction</strong> (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of <span class="arithmatex">\(-\ln(x)\)</span> terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
<span class="arithmatex">\(\alpha\left(x^{-1/\alpha} - 1\right)\)</span> instead — the standard mechanical
|
||||
substitution (as <span class="arithmatex">\(\alpha \to \infty\)</span>, this expression → <span class="arithmatex">\(-\ln(x)\)</span>,
|
||||
recovering the uncorrected formula exactly). E.g. for JC:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]</div>
|
||||
<p>Verified term-by-term against <code>ape</code>'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to <code>snp-k2p</code>'s own <span class="arithmatex">\(a_1\)</span>/<span class="arithmatex">\(a_2\)</span> terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one <code>corrected_log</code> helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.</p>
|
||||
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
||||
<p><strong>Implemented.</strong> The primary distance-matrix output
|
||||
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
||||
default</strong> (widely read by external NJ tools — PHYLIP <code>neighbor</code>, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
label truncation, since genome labels here routinely exceed it), a <code>--csv</code>
|
||||
flag opts back into the current CSV format, PHYLIP-strict is a possible
|
||||
future addition (not now). This changes the <em>default</em> output of every
|
||||
existing <code>--distance</code> value (jaccard, hamming, bray-curtis, ...), not just
|
||||
the new <code>snp-*</code> ones — accepted explicitly (pre-release, single developer
|
||||
user, no external consumers to break). Scoped to the distance matrix only:
|
||||
<code>--shared-kmers</code> and <code>--raw-snp-counts</code> are counts, not distances, and keep
|
||||
their existing CSV-only format.</p>
|
||||
<h2 id="references">References</h2>
|
||||
<p>The Mash mutation-rate model this discussion contrasts with:
|
||||
(Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>
|
||||
|
||||
@@ -2182,9 +2182,9 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
|
||||
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
|
||||
workspace `cargo test` green.
|
||||
|
||||
## `--distance` unification: SNP corrections as first-class metrics (discussion, 2026-08-28)
|
||||
## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
**Decided, not yet implemented.** `--metric` (renamed `--distance` — several of
|
||||
**Implemented.** `--metric` (renamed `--distance` — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
`--metric` was a misnomer) gains a family of `snp-*` values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
@@ -2253,9 +2253,147 @@ formula as the base correction, weighted by a shape parameter `α` supplied
|
||||
by the user (`--gamma-shape <alpha>`), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.
|
||||
|
||||
**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
|
||||
`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
|
||||
"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
|
||||
`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
|
||||
needs the true *directional* per-pair base co-occurrence matrix
|
||||
(`PairwiseTally` only keeps the symmetrised substitution counts
|
||||
`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), `snp-tajima-nei` needs each genome's *own* base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
`snp-f84`/`snp-hky85` had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
|
||||
`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
|
||||
a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
|
||||
`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
|
||||
accumulator — not an architecture change.
|
||||
|
||||
### Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
Sufficient statistic, per genome pair `(i, j)`, from
|
||||
`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
|
||||
`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
|
||||
|
||||
- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
|
||||
(pyrimidine transitions)
|
||||
- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
|
||||
(Kimura's two transversion categories)
|
||||
- \(n_{shared}\): loci where both genomes agree
|
||||
- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
|
||||
loci for the pair)
|
||||
- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
|
||||
\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)
|
||||
|
||||
Derived proportions used below:
|
||||
|
||||
\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]
|
||||
|
||||
Every formula below was checked term-by-term against `ape`'s own
|
||||
`src/dist_dna.c` (not re-derived from memory) before being ported to
|
||||
`obikphylo::siblings::algorithms::snp_distance`.
|
||||
|
||||
**`snp-raw`** — uncorrected p-distance:
|
||||
|
||||
\[
|
||||
d_{raw} = p
|
||||
\]
|
||||
|
||||
**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, *is* the correction):
|
||||
|
||||
\[
|
||||
d_{tv} = Q
|
||||
\]
|
||||
|
||||
**`snp-jc`** (Jukes-Cantor, JC69):
|
||||
|
||||
\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]
|
||||
|
||||
**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
|
||||
\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
|
||||
|
||||
\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]
|
||||
|
||||
**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
|
||||
|
||||
\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]
|
||||
|
||||
**`snp-t92`** (Tamura 3-parameter), with GC content
|
||||
\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
|
||||
\(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
|
||||
|
||||
\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]
|
||||
|
||||
\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]
|
||||
|
||||
\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]
|
||||
|
||||
**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
|
||||
substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
|
||||
recovering the uncorrected formula exactly). E.g. for JC:
|
||||
|
||||
\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]
|
||||
|
||||
Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one `corrected_log` helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
`ape`-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.
|
||||
|
||||
### Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
**Decided, not yet implemented.** The primary distance-matrix output
|
||||
**Implemented.** The primary distance-matrix output
|
||||
(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
|
||||
default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
|
||||
Generated
+1
@@ -1670,6 +1670,7 @@ version = "1.2.2"
|
||||
dependencies = [
|
||||
"clap",
|
||||
"csv",
|
||||
"ndarray",
|
||||
"obifastwrite",
|
||||
"obikalgorithm",
|
||||
"obikdump",
|
||||
|
||||
@@ -29,6 +29,7 @@ obifastwrite = { path = "../obifastwrite" }
|
||||
obiskbuilder = { path = "../obiskbuilder" }
|
||||
clap = { version = "4", features = ["derive"] }
|
||||
csv = "1"
|
||||
ndarray = "0.17"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_yaml = "0.9"
|
||||
tracing = "0.1.44"
|
||||
|
||||
@@ -1,10 +1,16 @@
|
||||
use std::path::PathBuf;
|
||||
|
||||
use clap::Args;
|
||||
use obikphylo::DistanceMetric;
|
||||
use obikphylo::{DistanceMetric, SnpDistanceKind};
|
||||
|
||||
/// `--distance` value — either one of `obikphylo::DistanceMetric`'s
|
||||
/// whole-index metrics (routed to `IndexCache::distance`) or one of
|
||||
/// `obikphylo::SnpDistanceKind`'s `snp-*` corrections (routed to
|
||||
/// `SiblingExt::snp_distance`, the sibling-annex pipeline) — two genuinely
|
||||
/// different code paths behind one CLI vocabulary, see
|
||||
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance` unification".
|
||||
#[derive(clap::ValueEnum, Clone, Copy, Debug)]
|
||||
pub enum MetricArg {
|
||||
pub enum DistanceArg {
|
||||
Jaccard,
|
||||
Mash,
|
||||
Hamming,
|
||||
@@ -17,35 +23,69 @@ pub enum MetricArg {
|
||||
Hellinger,
|
||||
#[value(name = "hellinger-euclidean")]
|
||||
HellingerEuclidean,
|
||||
#[value(name = "snp-raw")]
|
||||
SnpRaw,
|
||||
#[value(name = "snp-jc")]
|
||||
SnpJc,
|
||||
#[value(name = "snp-k2p")]
|
||||
SnpK2p,
|
||||
#[value(name = "snp-k81")]
|
||||
SnpK81,
|
||||
#[value(name = "snp-f81")]
|
||||
SnpF81,
|
||||
#[value(name = "snp-t92")]
|
||||
SnpT92,
|
||||
#[value(name = "snp-tn93")]
|
||||
SnpTn93,
|
||||
#[value(name = "snp-tv")]
|
||||
SnpTv,
|
||||
}
|
||||
|
||||
impl From<MetricArg> for DistanceMetric {
|
||||
fn from(m: MetricArg) -> Self {
|
||||
match m {
|
||||
MetricArg::Jaccard => DistanceMetric::Jaccard,
|
||||
MetricArg::Mash => DistanceMetric::Mash,
|
||||
MetricArg::Hamming => DistanceMetric::Hamming,
|
||||
MetricArg::BrayCurtis => DistanceMetric::BrayCurtis,
|
||||
MetricArg::RelfreqBrayCurtis => DistanceMetric::RelfreqBrayCurtis,
|
||||
MetricArg::Euclidean => DistanceMetric::Euclidean,
|
||||
MetricArg::RelfreqEuclidean => DistanceMetric::RelfreqEuclidean,
|
||||
MetricArg::Hellinger => DistanceMetric::Hellinger,
|
||||
MetricArg::HellingerEuclidean => DistanceMetric::HellingerEuclidean,
|
||||
}
|
||||
impl DistanceArg {
|
||||
/// `Some` for the whole-index metrics, `None` for `snp-*` values.
|
||||
pub fn as_classic(self) -> Option<DistanceMetric> {
|
||||
Some(match self {
|
||||
DistanceArg::Jaccard => DistanceMetric::Jaccard,
|
||||
DistanceArg::Mash => DistanceMetric::Mash,
|
||||
DistanceArg::Hamming => DistanceMetric::Hamming,
|
||||
DistanceArg::BrayCurtis => DistanceMetric::BrayCurtis,
|
||||
DistanceArg::RelfreqBrayCurtis => DistanceMetric::RelfreqBrayCurtis,
|
||||
DistanceArg::Euclidean => DistanceMetric::Euclidean,
|
||||
DistanceArg::RelfreqEuclidean => DistanceMetric::RelfreqEuclidean,
|
||||
DistanceArg::Hellinger => DistanceMetric::Hellinger,
|
||||
DistanceArg::HellingerEuclidean => DistanceMetric::HellingerEuclidean,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
|
||||
/// `Some` for the `snp-*` values, `None` for the whole-index metrics.
|
||||
pub fn as_snp(self) -> Option<SnpDistanceKind> {
|
||||
Some(match self {
|
||||
DistanceArg::SnpRaw => SnpDistanceKind::Raw,
|
||||
DistanceArg::SnpJc => SnpDistanceKind::Jc,
|
||||
DistanceArg::SnpK2p => SnpDistanceKind::K2p,
|
||||
DistanceArg::SnpK81 => SnpDistanceKind::K81,
|
||||
DistanceArg::SnpF81 => SnpDistanceKind::F81,
|
||||
DistanceArg::SnpT92 => SnpDistanceKind::T92,
|
||||
DistanceArg::SnpTn93 => SnpDistanceKind::Tn93,
|
||||
DistanceArg::SnpTv => SnpDistanceKind::Tv,
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Partial transfer of `obikmer`'s `phylo` command: the plain distance-metric
|
||||
/// path (`--metric`/NJ/UPGMA), annex construction (`--sibling-annex`),
|
||||
/// annex diagnostics (`--sibling-stats`, `--sibling-hist`), entropy
|
||||
/// reporting (`--shannon`), SNP pseudo-alignment sampling
|
||||
/// (`--pseudo-alignment`, `--subsample`, `--free-loss`, `--no-ambiguity`,
|
||||
/// `--entropy`/`--entropy-sd`), Sankoff cost-matrix calibration
|
||||
/// (`--sankoff`, `--sankoff-ratio-ceiling`) and its TNT/PhyG/IQ-TREE exports
|
||||
/// (`--tnt`, `--phyg`, `--iqtree`/`--iqtree-min-freq`,
|
||||
/// `--sankoff-cost-scale`) — everything else sibling-annex-based (raw SNP
|
||||
/// distance, family overlap, ...) stays in `obikmer` until the rest of
|
||||
/// `obikphylo::siblings` is reconnected (see the project memory on this).
|
||||
/// Partial transfer of `obikmer`'s `phylo` command: the whole-index
|
||||
/// `--distance` path (classic metrics + `snp-*` corrections/NJ/UPGMA),
|
||||
/// annex construction (`--sibling-annex`), annex diagnostics
|
||||
/// (`--sibling-stats`, `--sibling-hist`), entropy reporting (`--shannon`),
|
||||
/// SNP pseudo-alignment sampling (`--pseudo-alignment`, `--subsample`,
|
||||
/// `--free-loss`, `--no-ambiguity`, `--entropy`/`--entropy-sd`), Sankoff
|
||||
/// cost-matrix calibration (`--sankoff`, `--sankoff-ratio-ceiling`) and its
|
||||
/// TNT/PhyG/IQ-TREE exports (`--tnt`, `--phyg`, `--iqtree`/
|
||||
/// `--iqtree-min-freq`, `--sankoff-cost-scale`) — everything else
|
||||
/// sibling-annex-based (family overlap, ...) stays in `obikmer` until the
|
||||
/// rest of `obikphylo::siblings` is reconnected (see the project memory on
|
||||
/// this).
|
||||
#[derive(Args)]
|
||||
pub struct PhyloArgs {
|
||||
/// Index directory
|
||||
@@ -97,9 +137,13 @@ pub struct PhyloArgs {
|
||||
pub pseudo_alignment: bool,
|
||||
|
||||
/// Target number of variable sites to sample index-wide for
|
||||
/// `--pseudo-alignment` — a target, not a guarantee (proportional
|
||||
/// per-layer sampling; see `obikphylo::siblings::SiblingExt::snp_pseudo_alignment`'s
|
||||
/// own docs).
|
||||
/// `--pseudo-alignment`/`--sankoff` (mandatory for both) and for a
|
||||
/// `snp-*` `--distance` value (optional there: omitted means exhaustive
|
||||
/// — every non-monomorphic minorant of the whole index, not an
|
||||
/// approximation, see `obikphylo::siblings::SiblingExt::snp_distance`'s
|
||||
/// own docs) — a target, not a guarantee when given (proportional
|
||||
/// per-layer sampling; see
|
||||
/// `obikphylo::siblings::SiblingExt::snp_pseudo_alignment`'s own docs).
|
||||
#[arg(long)]
|
||||
pub subsample: Option<usize>,
|
||||
|
||||
@@ -205,14 +249,43 @@ pub struct PhyloArgs {
|
||||
#[arg(long, default_value = "100")]
|
||||
pub sankoff_cost_scale: f64,
|
||||
|
||||
/// Distance metric to compute
|
||||
/// Distance to compute — either a whole-index metric (`jaccard`,
|
||||
/// `mash`, `hamming`, `bray-curtis`, ...) or a `snp-*` correction over
|
||||
/// the central-position SNP substitution spectrum (`snp-raw`, `snp-jc`,
|
||||
/// `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`) —
|
||||
/// the latter route to a different computation entirely
|
||||
/// (`SiblingExt::snp_distance`, requires `--sibling-annex` first; see
|
||||
/// `DevDocMD/theory/evolutionary_distances.md`, "`--distance`
|
||||
/// unification" for the full catalog and why LogDet/Tajima-Nei/F84/
|
||||
/// HKY85 aren't offered yet).
|
||||
#[arg(long, value_enum, default_value = "jaccard")]
|
||||
pub metric: MetricArg,
|
||||
pub distance: DistanceArg,
|
||||
|
||||
/// Rate-heterogeneity correction (Jin-Nei gamma shape parameter `α`)
|
||||
/// for `snp-*` `--distance` values that support it
|
||||
/// (`obikphylo::SnpDistanceKind::supports_gamma`: every one except
|
||||
/// `snp-raw`/`snp-tv`, which have nothing to correct/are deliberately
|
||||
/// uncorrected). Has no effect on the whole-index metrics. Rejected at
|
||||
/// runtime if given alongside an unsupported `--distance` value.
|
||||
#[arg(long, value_name = "ALPHA")]
|
||||
pub gamma_shape: Option<f64>,
|
||||
|
||||
/// Minimum count to consider a kmer present when computing Jaccard on count indexes
|
||||
#[arg(long, default_value = "1")]
|
||||
pub presence_threshold: u32,
|
||||
|
||||
/// Write the primary distance matrix as plain CSV instead of the
|
||||
/// default relaxed-PHYLIP format (`n` on the first line, then one
|
||||
/// `label<TAB>value...` row per genome — no 10-character label
|
||||
/// truncation, unlike strict PHYLIP, not yet offered here). PHYLIP is
|
||||
/// the default because it's what external NJ tools (PHYLIP `neighbor`,
|
||||
/// FastME, T-REX, SplitsTree) actually read; CSV stays available for
|
||||
/// scripting/inspection. Only affects the primary distance matrix —
|
||||
/// `--shared-kmers` keeps its own CSV-only format regardless of this
|
||||
/// flag.
|
||||
#[arg(long)]
|
||||
pub csv: bool,
|
||||
|
||||
/// Also output the shared-kmer count matrix (CSV)
|
||||
#[arg(long)]
|
||||
pub shared_kmers: bool,
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
mod args;
|
||||
mod iqtree;
|
||||
mod phyg;
|
||||
mod phylip;
|
||||
mod sankoff;
|
||||
mod tnt;
|
||||
|
||||
@@ -19,6 +20,7 @@ use tracing::info;
|
||||
|
||||
use iqtree::write_iqtree;
|
||||
use phyg::write_sankoff_phyg;
|
||||
use phylip::write_phylip_relaxed;
|
||||
use sankoff::{write_sankoff_alignment_fasta, write_sankoff_matrix_csv, write_sankoff_params};
|
||||
use tnt::write_sankoff_tnt;
|
||||
|
||||
@@ -271,57 +273,100 @@ pub fn run(args: PhyloArgs) {
|
||||
}
|
||||
}
|
||||
|
||||
info!("computing {:?} distances for {} genome(s)", args.metric, n);
|
||||
// ── Distance computation: classic whole-index metric vs. `snp-*` ───────────
|
||||
// Two genuinely different code paths behind one `--distance` value — see
|
||||
// `args::DistanceArg`'s own docs.
|
||||
let (matrix, shared_kmers) = match args.distance.as_classic() {
|
||||
Some(metric) => {
|
||||
info!("computing {metric:?} distances for {n} genome(s)");
|
||||
let need_shared = args.shared_kmers || args.nj || args.upgma;
|
||||
let t = Stage::start("distance");
|
||||
let result = cache
|
||||
.distance(metric, need_shared, args.presence_threshold)
|
||||
.unwrap_or_else(|e| {
|
||||
eprintln!("error computing distances: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
rep.push(t.stop());
|
||||
(result.matrix, result.shared_kmers)
|
||||
}
|
||||
None => {
|
||||
if args.shared_kmers {
|
||||
eprintln!("error: --shared-kmers has no meaning for a snp-* --distance value");
|
||||
std::process::exit(1);
|
||||
}
|
||||
let kind = args.distance.as_snp().expect("DistanceArg is always classic or snp");
|
||||
info!(
|
||||
"computing {kind:?} SNP distance for {n} genome(s){}",
|
||||
match args.subsample {
|
||||
Some(n) => format!(" (subsampled, target {n} site(s))"),
|
||||
None => " (exhaustive)".into(),
|
||||
}
|
||||
);
|
||||
let t = Stage::start("snp_distance");
|
||||
let matrix = cache
|
||||
.snp_distance(
|
||||
kind,
|
||||
args.subsample,
|
||||
args.free_loss,
|
||||
args.no_ambiguity,
|
||||
&exclude_mask,
|
||||
entropy_bias,
|
||||
args.gamma_shape,
|
||||
)
|
||||
.unwrap_or_else(|e| {
|
||||
eprintln!("error computing SNP distance: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
rep.push(t.stop());
|
||||
(matrix, None)
|
||||
}
|
||||
};
|
||||
|
||||
let need_shared = args.shared_kmers || args.nj || args.upgma;
|
||||
let t = Stage::start("distance");
|
||||
let result = cache
|
||||
.distance(args.metric.into(), need_shared, args.presence_threshold)
|
||||
.unwrap_or_else(|e| {
|
||||
eprintln!("error computing distances: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
rep.push(t.stop());
|
||||
|
||||
// Rows/columns kept in every matrix CSV below — `cache.distance(...)`
|
||||
// above is computed over every genome regardless; only the writers skip
|
||||
// Rows/columns kept in every matrix output below — the computation
|
||||
// above runs over every genome regardless; only the writers skip
|
||||
// excluded ones.
|
||||
let kept: Vec<usize> = (0..n).filter(|&i| !exclude_mask[i]).collect();
|
||||
|
||||
// ── Distance matrix → CSV ─────────────────────────────────────────────────
|
||||
let write_dist_csv = |w: &mut dyn Write| {
|
||||
write!(w, "genome").unwrap();
|
||||
for &j in &kept { write!(w, ",{}", labels[j]).unwrap(); }
|
||||
writeln!(w).unwrap();
|
||||
for &i in &kept {
|
||||
write!(w, "{}", labels[i]).unwrap();
|
||||
for &j in &kept {
|
||||
write!(w, ",{:.6}", result.matrix[[i, j]]).unwrap();
|
||||
}
|
||||
// ── Distance matrix → relaxed PHYLIP (default) or CSV (`--csv`) ────────────
|
||||
let write_dist = |w: &mut dyn Write| {
|
||||
if args.csv {
|
||||
write!(w, "genome").unwrap();
|
||||
for &j in &kept { write!(w, ",{}", labels[j]).unwrap(); }
|
||||
writeln!(w).unwrap();
|
||||
for &i in &kept {
|
||||
write!(w, "{}", labels[i]).unwrap();
|
||||
for &j in &kept {
|
||||
write!(w, ",{:.6}", matrix[[i, j]]).unwrap();
|
||||
}
|
||||
writeln!(w).unwrap();
|
||||
}
|
||||
} else {
|
||||
write_phylip_relaxed(w, &labels, &kept, &matrix);
|
||||
}
|
||||
};
|
||||
|
||||
match &args.output {
|
||||
Some(prefix) => {
|
||||
let path = format!("{}_dist.csv", prefix.display());
|
||||
let suffix = if args.csv { "_dist.csv" } else { "_dist.phy" };
|
||||
let path = format!("{}{suffix}", prefix.display());
|
||||
let mut f = BufWriter::new(std::fs::File::create(&path).unwrap_or_else(|e| {
|
||||
eprintln!("error creating {path}: {e}");
|
||||
std::process::exit(1);
|
||||
}));
|
||||
write_dist_csv(&mut f);
|
||||
write_dist(&mut f);
|
||||
info!("distance matrix → {path}");
|
||||
}
|
||||
None => {
|
||||
let stdout = io::stdout();
|
||||
let mut out = BufWriter::new(stdout.lock());
|
||||
write_dist_csv(&mut out);
|
||||
write_dist(&mut out);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Shared-kmer matrix → CSV ──────────────────────────────────────────────
|
||||
if args.shared_kmers {
|
||||
if let Some(shared) = &result.shared_kmers {
|
||||
if let Some(shared) = &shared_kmers {
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_shared.csv", p.display()))
|
||||
.unwrap_or_else(|| "shared.csv".into());
|
||||
@@ -343,7 +388,7 @@ pub fn run(args: PhyloArgs) {
|
||||
|
||||
// ── NJ tree ────────────────────────────────────────────────────────────────
|
||||
if args.nj {
|
||||
let tree = neighbor_joining(&result.matrix, &labels).unwrap_or_else(|e| {
|
||||
let tree = neighbor_joining(&matrix, &labels).unwrap_or_else(|e| {
|
||||
eprintln!("error computing NJ tree: {e}");
|
||||
std::process::exit(1);
|
||||
});
|
||||
@@ -360,7 +405,7 @@ pub fn run(args: PhyloArgs) {
|
||||
|
||||
// ── UPGMA tree ───────────────────────────────────────────────────────────────
|
||||
if args.upgma {
|
||||
let newick = upgma(&result.matrix, &labels).to_newick();
|
||||
let newick = upgma(&matrix, &labels).to_newick();
|
||||
let path = args.output.as_ref()
|
||||
.map(|p| format!("{}_upgma.nwk", p.display()))
|
||||
.unwrap_or_else(|| "upgma.nwk".into());
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
//! Relaxed-PHYLIP distance-matrix writer — the default output format for
|
||||
//! `--distance` (see `args::PhyloArgs::csv`'s own docs for why): `n` on the
|
||||
//! first line, then one `label<TAB>value...` row per genome. "Relaxed"
|
||||
//! (unlike strict PHYLIP) means no 10-character label truncation/padding —
|
||||
//! just whitespace-separated fields, which every external NJ tool this
|
||||
//! targets (PHYLIP `neighbor`, FastME, T-REX, SplitsTree) already reads,
|
||||
//! and genome labels here routinely exceed strict PHYLIP's 10 characters.
|
||||
|
||||
use std::io::Write;
|
||||
|
||||
use ndarray::Array2;
|
||||
|
||||
pub(super) fn write_phylip_relaxed(w: &mut dyn Write, labels: &[String], indices: &[usize], matrix: &Array2<f64>) {
|
||||
writeln!(w, "{}", indices.len()).unwrap();
|
||||
for &i in indices {
|
||||
write!(w, "{}", labels[i]).unwrap();
|
||||
for &j in indices {
|
||||
write!(w, "\t{:.6}", matrix[[i, j]]).unwrap();
|
||||
}
|
||||
writeln!(w).unwrap();
|
||||
}
|
||||
}
|
||||
@@ -19,4 +19,5 @@ mod tree;
|
||||
pub mod siblings;
|
||||
|
||||
pub use distance::{DistanceMetric, DistanceOutput, Metrics};
|
||||
pub use siblings::SnpDistanceKind;
|
||||
pub use tree::{Tree, neighbor_joining, upgma};
|
||||
|
||||
@@ -22,6 +22,7 @@ mod masking;
|
||||
mod minorant_selection;
|
||||
mod pairwise;
|
||||
mod sankoff;
|
||||
mod snp_distance;
|
||||
mod stats;
|
||||
mod subsample;
|
||||
|
||||
@@ -33,6 +34,7 @@ pub use cardcomp::{
|
||||
};
|
||||
pub use pairwise::{BasePairTally, CardinalityTally, RawSnpDistanceOutput};
|
||||
pub use sankoff::SankoffBundle;
|
||||
pub use snp_distance::SnpDistanceKind;
|
||||
pub use stats::SiblingAnnexStats;
|
||||
pub use subsample::{EntropyBias, SurvivingFamily};
|
||||
|
||||
@@ -41,6 +43,7 @@ pub(crate) use annex::build_layer_sibling_annex;
|
||||
pub(crate) use entropy::{ensure_layer_entropy_annex, family_entropy, family_entropy_4};
|
||||
pub(crate) use family_scan::{Selection, scan_layer_families};
|
||||
pub(crate) use sankoff::sankoff_bundle;
|
||||
pub(crate) use snp_distance::snp_distance;
|
||||
pub(crate) use stats::{sibling_annex_stats, sibling_family_size_histogram};
|
||||
|
||||
/// Whether every layer number in `cache` fits in a `FamilyMask` field
|
||||
|
||||
@@ -175,6 +175,66 @@ impl PairwiseTally {
|
||||
}
|
||||
CardinalityTally { counts }
|
||||
}
|
||||
|
||||
/// Categorised substitution counts for pair `(i, j)`, plus its eligible
|
||||
/// ("shared") locus count — the sufficient statistic every closed-form
|
||||
/// `algorithms::snp_distance` correction is built from. Base order
|
||||
/// `0=A,1=C,2=G,3=T` (same convention as `FamilyMask`/`STATE_SYMBOL`).
|
||||
/// `ts1`/`ts2` split transitions by purine (A↔G) vs pyrimidine (C↔T)
|
||||
/// pair — required for TN93, collapse to a single `ts1+ts2` for
|
||||
/// K80/K81/T92, which don't distinguish them. `tv1`/`tv2` split
|
||||
/// transversions the way Kimura's 3-parameter model does (A↔C/G↔T vs
|
||||
/// A↔T/C↔G) — collapse to `tv1+tv2` for every model that doesn't need
|
||||
/// the distinction (K80, F81, T92, TN93).
|
||||
pub(crate) fn categories(&self, i: usize, j: usize) -> PairCategories {
|
||||
let stats = self.pair(i, j);
|
||||
PairCategories {
|
||||
ts1: stats.subst[0][2],
|
||||
ts2: stats.subst[1][3],
|
||||
tv1: stats.subst[0][1] + stats.subst[2][3],
|
||||
tv2: stats.subst[0][3] + stats.subst[1][2],
|
||||
shared: stats.same_all.iter().sum(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Pooled base composition for pair `(i, j)`, estimated from both
|
||||
/// genomes' observed calls at their `shared`/differing eligible loci —
|
||||
/// `2 * same_all[a]` (each agreement locus contributes `a` to both
|
||||
/// genomes) plus `Σ_b subst[a][b]` (each differing locus contributes
|
||||
/// `a` to whichever of the two genomes carried it — `subst` is
|
||||
/// symmetric by construction, so summing one row already counts each
|
||||
/// such event exactly once, see `reduce_pairwise`'s own docs), over
|
||||
/// `2 * n_eligible_loci` total base observations. Feeds F81/T92/TN93,
|
||||
/// which all correct for base-composition bias using exactly this kind
|
||||
/// of pooled empirical frequency (Nei & Kumar's standard estimator for
|
||||
/// a pairwise comparison, not a whole-index average).
|
||||
pub(crate) fn base_freq(&self, i: usize, j: usize) -> [f64; 4] {
|
||||
let stats = self.pair(i, j);
|
||||
let mut counts = [0u64; 4];
|
||||
for a in 0..4 {
|
||||
counts[a] = 2 * stats.same_all[a] + (0..4).map(|b| stats.subst[a][b]).sum::<u64>();
|
||||
}
|
||||
let total: u64 = counts.iter().sum();
|
||||
if total == 0 {
|
||||
return [0.25; 4];
|
||||
}
|
||||
counts.map(|c| c as f64 / total as f64)
|
||||
}
|
||||
}
|
||||
|
||||
/// See [`PairwiseTally::categories`]'s own docs.
|
||||
pub(crate) struct PairCategories {
|
||||
pub ts1: u64,
|
||||
pub ts2: u64,
|
||||
pub tv1: u64,
|
||||
pub tv2: u64,
|
||||
pub shared: u64,
|
||||
}
|
||||
|
||||
impl PairCategories {
|
||||
pub(crate) fn n_eligible(&self) -> u64 {
|
||||
self.ts1 + self.ts2 + self.tv1 + self.tv2 + self.shared
|
||||
}
|
||||
}
|
||||
|
||||
/// Raw p-distance restricted to loci that are single-copy in **both**
|
||||
|
||||
@@ -0,0 +1,398 @@
|
||||
//! `snp-*` `--distance` values — closed-form (method-of-moments) pairwise
|
||||
//! corrections over the central-position SNP substitution spectrum, all
|
||||
//! derived from one shared [`PairwiseTally`] (see its own module docs: a
|
||||
//! single `O(n²)` structure, built once, from which every correction below
|
||||
//! is cheap post-processing, never a second index scan). See
|
||||
//! `DevDocMD/theory/evolutionary_distances.md`, "`--distance` unification"
|
||||
//! for the design discussion and the literature this implements
|
||||
//! ([`ape`](https://github.com/emmanuelparadis/ape)'s `dist.dna` C source,
|
||||
//! `src/dist_dna.c`, verified formula-by-formula against the upstream
|
||||
//! implementation rather than re-derived from memory).
|
||||
//!
|
||||
//! Deliberately **not** included here: LogDet/paralinear (needs the true
|
||||
//! *directional* per-pair base co-occurrence matrix — `PairwiseTally`
|
||||
//! only keeps the symmetrised substitution counts `BasePairTally` itself
|
||||
//! wants, which loses exactly the compositional-asymmetry information
|
||||
//! LogDet exists to detect), Tajima-Nei (needs each genome's *own* base
|
||||
//! composition, not the pair-pooled estimate `base_freq` below provides),
|
||||
//! F84 and the ML-fit HKY85 tree usage (no formula independently verified
|
||||
//! against a primary source at implementation time). Adding any of these
|
||||
//! later is a new function in this module plus, for LogDet/Tajima-Nei, a
|
||||
//! new field on `PairStats`/a per-genome accumulator — not an architecture
|
||||
//! change.
|
||||
|
||||
use ndarray::Array2;
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
use obikindex::OKIResult;
|
||||
|
||||
use super::pairwise::PairwiseTally;
|
||||
use super::sibling_family_size_histogram;
|
||||
use super::subsample::{EntropyBias, sample_index};
|
||||
|
||||
/// One `snp-*` `--distance` value. `pub`: part of
|
||||
/// [`crate::siblings::extensions::SiblingExt::snp_distance`]'s public
|
||||
/// signature.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum SnpDistanceKind {
|
||||
/// Uncorrected p-distance (`snp / (snp + shared)`).
|
||||
Raw,
|
||||
/// Jukes-Cantor (JC69): corrects for multiple substitutions per site,
|
||||
/// equal rates, equal base frequencies.
|
||||
Jc,
|
||||
/// Kimura 2-parameter (K80): + transition/transversion rate bias.
|
||||
K2p,
|
||||
/// Kimura 3-parameter (K81/K3ST): splits transversions into the two
|
||||
/// categories A↔C/G↔T and A↔T/C↔G, each its own rate.
|
||||
K81,
|
||||
/// Felsenstein 81 (F81): JC69 + unequal (pair-pooled empirical) base
|
||||
/// frequencies, no ts/tv distinction.
|
||||
F81,
|
||||
/// Tamura 3-parameter (T92): K80 + GC-content bias.
|
||||
T92,
|
||||
/// Tamura-Nei (TN93): unequal base frequencies + separate purine
|
||||
/// (A↔G) / pyrimidine (C↔T) transition rates + a transversion rate —
|
||||
/// the richest closed-form correction implemented here.
|
||||
Tn93,
|
||||
/// Transversions-only p-distance — diagnostic/deep-divergence variant,
|
||||
/// deliberately uncorrected (dropping transitions, which saturate
|
||||
/// first, is itself the correction).
|
||||
Tv,
|
||||
}
|
||||
|
||||
impl SnpDistanceKind {
|
||||
/// Whether `--gamma-shape` applies to this correction — every
|
||||
/// rate-based model here except the two with no "-ln(x)" term to
|
||||
/// reinterpret as a gamma mixture (`Raw`, nothing to correct; `Tv`,
|
||||
/// deliberately uncorrected).
|
||||
pub fn supports_gamma(self) -> bool {
|
||||
!matches!(self, SnpDistanceKind::Raw | SnpDistanceKind::Tv)
|
||||
}
|
||||
}
|
||||
|
||||
/// `-ln(x)`, gamma-mixture corrected when `alpha` is given: replaces the
|
||||
/// single-rate `-ln(x)` term with the standard Jin-Nei (1990) gamma
|
||||
/// substitution `alpha * (x^(-1/alpha) - 1)` — the mechanical term-wise
|
||||
/// rewrite every gamma-corrected distance in the literature (JC+Γ, K80+Γ,
|
||||
/// F81+Γ, ...) applies to each log term of its base formula; verified here
|
||||
/// against `ape`'s own JC69/K80/F81 gamma branches (`dist_dna.c`), then
|
||||
/// generalised the same way to every other log term this module uses
|
||||
/// (K81/T92/TN93 each being multi-term sums of exactly this shape).
|
||||
fn corrected_log(x: f64, alpha: Option<f64>) -> f64 {
|
||||
match alpha {
|
||||
Some(alpha) => alpha * (x.powf(-1.0 / alpha) - 1.0),
|
||||
None => -x.ln(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn raw(tally: &PairwiseTally, i: usize, j: usize) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
(c.ts1 + c.ts2 + c.tv1 + c.tv2) as f64 / l as f64
|
||||
}
|
||||
|
||||
pub(crate) fn tv_only(tally: &PairwiseTally, i: usize, j: usize) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
(c.tv1 + c.tv2) as f64 / l as f64
|
||||
}
|
||||
|
||||
pub(crate) fn jc(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let p = raw(tally, i, j);
|
||||
0.75 * corrected_log(1.0 - 4.0 * p / 3.0, alpha)
|
||||
}
|
||||
|
||||
pub(crate) fn k2p(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let p = (c.ts1 + c.ts2) as f64 / l as f64;
|
||||
let q = (c.tv1 + c.tv2) as f64 / l as f64;
|
||||
let a1 = 1.0 - 2.0 * p - q;
|
||||
let a2 = 1.0 - 2.0 * q;
|
||||
0.5 * corrected_log(a1, alpha) + 0.25 * corrected_log(a2, alpha)
|
||||
}
|
||||
|
||||
pub(crate) fn k81(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let p = (c.ts1 + c.ts2) as f64 / l as f64;
|
||||
let q = c.tv1 as f64 / l as f64;
|
||||
let r = c.tv2 as f64 / l as f64;
|
||||
let a1 = 1.0 - 2.0 * p - 2.0 * q;
|
||||
let a2 = 1.0 - 2.0 * p - 2.0 * r;
|
||||
let a3 = 1.0 - 2.0 * q - 2.0 * r;
|
||||
0.25 * (corrected_log(a1, alpha) + corrected_log(a2, alpha) + corrected_log(a3, alpha))
|
||||
}
|
||||
|
||||
pub(crate) fn f81(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let p = raw(tally, i, j);
|
||||
let freq = tally.base_freq(i, j);
|
||||
let e = 1.0 - freq.iter().map(|f| f * f).sum::<f64>();
|
||||
match alpha {
|
||||
Some(a) => e * a * ((1.0 - p / e).powf(-1.0 / a) - 1.0),
|
||||
None => -e * (1.0 - p / e).ln(),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn t92(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let p = (c.ts1 + c.ts2) as f64 / l as f64;
|
||||
let q = (c.tv1 + c.tv2) as f64 / l as f64;
|
||||
let freq = tally.base_freq(i, j);
|
||||
let gc = freq[1] + freq[2]; // C + G
|
||||
let wg = 2.0 * gc * (1.0 - gc);
|
||||
let a1 = 1.0 - p / wg - q;
|
||||
let a2 = 1.0 - 2.0 * q;
|
||||
wg * corrected_log(a1, alpha) + 0.5 * (1.0 - wg) * corrected_log(a2, alpha)
|
||||
}
|
||||
|
||||
pub(crate) fn tn93(tally: &PairwiseTally, i: usize, j: usize, alpha: Option<f64>) -> f64 {
|
||||
let c = tally.categories(i, j);
|
||||
let l = c.n_eligible();
|
||||
if l == 0 {
|
||||
return f64::NAN;
|
||||
}
|
||||
let freq = tally.base_freq(i, j);
|
||||
let g_r = freq[0] + freq[2]; // A + G (purines)
|
||||
let g_y = freq[1] + freq[3]; // C + T (pyrimidines)
|
||||
let k1 = 2.0 * freq[0] * freq[2] / g_r;
|
||||
let k2 = 2.0 * freq[1] * freq[3] / g_y;
|
||||
let k3 = 2.0 * (g_r * g_y - freq[0] * freq[2] * g_y / g_r - freq[1] * freq[3] * g_r / g_y);
|
||||
let p1 = c.ts1 as f64 / l as f64; // A<->G
|
||||
let p2 = c.ts2 as f64 / l as f64; // C<->T
|
||||
let q = (c.tv1 + c.tv2) as f64 / l as f64;
|
||||
let w1 = 1.0 - p1 / k1 - q / (2.0 * g_r);
|
||||
let w2 = 1.0 - p2 / k2 - q / (2.0 * g_y);
|
||||
let w3 = 1.0 - q / (2.0 * g_r * g_y);
|
||||
k1 * corrected_log(w1, alpha) + k2 * corrected_log(w2, alpha) + k3 * corrected_log(w3, alpha)
|
||||
}
|
||||
|
||||
fn formula(kind: SnpDistanceKind) -> fn(&PairwiseTally, usize, usize, Option<f64>) -> f64 {
|
||||
match kind {
|
||||
SnpDistanceKind::Raw => |t, i, j, _| raw(t, i, j),
|
||||
SnpDistanceKind::Tv => |t, i, j, _| tv_only(t, i, j),
|
||||
SnpDistanceKind::Jc => jc,
|
||||
SnpDistanceKind::K2p => k2p,
|
||||
SnpDistanceKind::K81 => k81,
|
||||
SnpDistanceKind::F81 => f81,
|
||||
SnpDistanceKind::T92 => t92,
|
||||
SnpDistanceKind::Tn93 => tn93,
|
||||
}
|
||||
}
|
||||
|
||||
/// Build (or reuse) a [`PairwiseTally`] and reduce it to one `n×n` distance
|
||||
/// matrix under `kind`. `n`: `Some(target)` samples proportionally per
|
||||
/// layer exactly like `--sankoff`/`--pseudo-alignment` (see
|
||||
/// `algorithms::subsample::sample_index`'s own docs); `None` means
|
||||
/// exhaustive — every non-monomorphic minorant of the whole index, achieved
|
||||
/// by handing `sample_index` a quota equal to the index-wide total (from
|
||||
/// [`sibling_family_size_histogram`], annex-bits-only, no cross-partition
|
||||
/// resolution), which makes every per-layer quota equal to that layer's own
|
||||
/// full eligible count, i.e. Bernoulli `p = 1` everywhere — no separate
|
||||
/// exhaustive-only driver needed. Entropy-biased sampling
|
||||
/// (`entropy_bias`) only makes sense for a genuine subsample, so it's
|
||||
/// ignored (forced to `None`) when `n` is `None`, regardless of what the
|
||||
/// caller passes.
|
||||
///
|
||||
/// `gamma_shape`: `--gamma-shape`, `None` disables the correction. Rejected
|
||||
/// with [`obikindex::OKIError::InvalidInput`] if given alongside a `kind`
|
||||
/// that doesn't support it ([`SnpDistanceKind::supports_gamma`]) — checked
|
||||
/// here rather than left to silently no-op.
|
||||
pub(crate) fn snp_distance(
|
||||
cache: &IndexCache,
|
||||
kind: SnpDistanceKind,
|
||||
n: Option<usize>,
|
||||
free_loss: bool,
|
||||
no_ambiguity: bool,
|
||||
excluded: &[bool],
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
gamma_shape: Option<f64>,
|
||||
) -> OKIResult<Array2<f64>> {
|
||||
if gamma_shape.is_some() && !kind.supports_gamma() {
|
||||
return Err(obikindex::OKIError::InvalidInput(
|
||||
"--gamma-shape has no effect on this --distance value".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let n_genomes = cache.meta().genomes().len();
|
||||
let mut tally = PairwiseTally::new(n_genomes);
|
||||
|
||||
let (target, entropy_bias) = match n {
|
||||
Some(target) => (target, entropy_bias),
|
||||
None => {
|
||||
let counts = sibling_family_size_histogram(cache)?;
|
||||
let total_eligible = (counts[1] + counts[2] + counts[3]) as usize;
|
||||
(total_eligible, None)
|
||||
}
|
||||
};
|
||||
|
||||
if target > 0 {
|
||||
sample_index(
|
||||
cache,
|
||||
target,
|
||||
free_loss,
|
||||
no_ambiguity,
|
||||
excluded,
|
||||
entropy_bias,
|
||||
|_partition, _layer, survivors| {
|
||||
super::pairwise::reduce_pairwise(&survivors, &mut tally);
|
||||
},
|
||||
)?;
|
||||
}
|
||||
|
||||
let f = formula(kind);
|
||||
Ok(Array2::from_shape_fn((n_genomes, n_genomes), |(i, j)| {
|
||||
if i == j { 0.0 } else { f(&tally, i, j, gamma_shape) }
|
||||
}))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::siblings::FamilyMask;
|
||||
use crate::siblings::algorithms::SurvivingFamily;
|
||||
use super::super::pairwise::reduce_pairwise;
|
||||
|
||||
// Base bit positions, matching `PairwiseTally::categories`' convention
|
||||
// (0=A, 1=C, 2=G, 3=T).
|
||||
const A: u8 = 1 << 0;
|
||||
const C: u8 = 1 << 1;
|
||||
const G: u8 = 1 << 2;
|
||||
const T: u8 = 1 << 3;
|
||||
|
||||
/// Two-genome synthetic tally: one variable family per `(genome0,
|
||||
/// genome1)` pair in `pairs`, plus `n_shared` fully-agreeing families to
|
||||
/// pad the eligible-locus denominator — cycled over all 4 bases (not a
|
||||
/// single one) so the pooled base composition stays non-degenerate
|
||||
/// (`F81`/`T92`/`TN93`'s denominators — `E`/`wg` — are exactly `0` for a
|
||||
/// single-base composition, an edge case irrelevant to real data, not
|
||||
/// something these tests need to exercise).
|
||||
fn tally_2genomes(pairs: &[(u8, u8)], n_shared: usize) -> PairwiseTally {
|
||||
let mut families = Vec::new();
|
||||
for &(a, b) in pairs {
|
||||
families.push(SurvivingFamily {
|
||||
family_idx: families.len(),
|
||||
mask: FamilyMask::EMPTY.with(a.trailing_zeros() as u8).with(b.trailing_zeros() as u8),
|
||||
genome_mask: vec![a, b],
|
||||
});
|
||||
}
|
||||
const BASES: [u8; 4] = [A, C, G, T];
|
||||
for i in 0..n_shared {
|
||||
let b = BASES[i % 4];
|
||||
let other = BASES[(i + 1) % 4];
|
||||
families.push(SurvivingFamily {
|
||||
family_idx: families.len(),
|
||||
mask: FamilyMask::EMPTY.with(b.trailing_zeros() as u8).with(other.trailing_zeros() as u8),
|
||||
genome_mask: vec![b, b],
|
||||
});
|
||||
}
|
||||
let mut tally = PairwiseTally::new(2);
|
||||
reduce_pairwise(&families, &mut tally);
|
||||
tally
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn categories_classify_each_substitution_type() {
|
||||
// A<->G (ts1), C<->T (ts2), A<->C (tv1), A<->T (tv2), + 1 shared.
|
||||
let tally = tally_2genomes(&[(A, G), (C, T), (A, C), (A, T)], 1);
|
||||
let c = tally.categories(0, 1);
|
||||
assert_eq!((c.ts1, c.ts2, c.tv1, c.tv2, c.shared), (1, 1, 1, 1, 1));
|
||||
assert_eq!(c.n_eligible(), 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identical_genomes_give_zero_distance_under_every_correction() {
|
||||
// No substitutions at all — every formula's "-ln(1)" term is 0.
|
||||
let tally = tally_2genomes(&[], 10);
|
||||
for kind in [
|
||||
SnpDistanceKind::Raw,
|
||||
SnpDistanceKind::Jc,
|
||||
SnpDistanceKind::K2p,
|
||||
SnpDistanceKind::K81,
|
||||
SnpDistanceKind::F81,
|
||||
SnpDistanceKind::T92,
|
||||
SnpDistanceKind::Tn93,
|
||||
SnpDistanceKind::Tv,
|
||||
] {
|
||||
let d = formula(kind)(&tally, 0, 1, None);
|
||||
assert!(d.abs() < 1e-9, "{kind:?} gave {d}, expected ~0");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn raw_matches_hand_computed_ratio() {
|
||||
// 1 substitution (A<->G) out of 5 eligible loci (1 subst + 4 shared).
|
||||
let tally = tally_2genomes(&[(A, G)], 4);
|
||||
assert!((raw(&tally, 0, 1) - 0.2).abs() < 1e-12);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn jc_matches_hand_computed_correction() {
|
||||
// p = 1/5 = 0.2 -> d = -0.75 * ln(1 - 4*0.2/3) = -0.75 * ln(1 - 4/15).
|
||||
let tally = tally_2genomes(&[(A, G)], 4);
|
||||
let expected = -0.75 * (1.0 - 4.0f64 * 0.2 / 3.0).ln();
|
||||
assert!((jc(&tally, 0, 1, None) - expected).abs() < 1e-12);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn jc_uncorrected_undershoots_raw_p_distance() {
|
||||
// JC always corrects *upward* from the raw p-distance (multiple-hit
|
||||
// correction only adds distance, never removes it) for any p in its
|
||||
// domain.
|
||||
let tally = tally_2genomes(&[(A, G), (C, T)], 20);
|
||||
let p = raw(&tally, 0, 1);
|
||||
let d = jc(&tally, 0, 1, None);
|
||||
assert!(d > p, "JC-corrected {d} should exceed raw p-distance {p}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gamma_correction_converges_to_uncorrected_as_alpha_grows() {
|
||||
// The Jin-Nei substitution alpha*(x^(-1/alpha) - 1) -> -ln(x) as
|
||||
// alpha -> infinity — a basic sanity check on `corrected_log`
|
||||
// independent of any hand-checked literature value.
|
||||
let tally = tally_2genomes(&[(A, G), (A, C)], 20);
|
||||
let uncorrected = jc(&tally, 0, 1, None);
|
||||
let large_alpha = jc(&tally, 0, 1, Some(1.0e6));
|
||||
assert!(
|
||||
(uncorrected - large_alpha).abs() < 1e-3,
|
||||
"uncorrected {uncorrected} vs large-alpha gamma {large_alpha}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn gamma_shape_rejected_for_unsupported_kind() {
|
||||
assert!(!SnpDistanceKind::Raw.supports_gamma());
|
||||
assert!(!SnpDistanceKind::Tv.supports_gamma());
|
||||
assert!(SnpDistanceKind::Jc.supports_gamma());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn base_freq_sums_to_one_and_matches_pooled_counts() {
|
||||
// 2 A/G substitutions (genome0=A, genome1=G each time) + 1 shared
|
||||
// A/A family -> pooled bases across both genomes: A appears 4 times
|
||||
// (1 per substitution's genome0 side, 2 from the shared locus's
|
||||
// both genomes), G appears 2 times (1 per substitution's genome1
|
||||
// side), total 6 (2 genomes * 3 loci).
|
||||
let tally = tally_2genomes(&[(A, G), (A, G)], 1);
|
||||
let freq = tally.base_freq(0, 1);
|
||||
assert!((freq.iter().sum::<f64>() - 1.0).abs() < 1e-12);
|
||||
assert!((freq[0] - 4.0 / 6.0).abs() < 1e-12); // A
|
||||
assert!((freq[2] - 2.0 / 6.0).abs() < 1e-12); // G
|
||||
assert!((freq[1] - 0.0).abs() < 1e-12); // C
|
||||
}
|
||||
}
|
||||
@@ -9,15 +9,16 @@
|
||||
use std::io::{BufWriter, Write};
|
||||
use std::path::Path;
|
||||
|
||||
use ndarray::Array2;
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
use obisys::progress_bar;
|
||||
|
||||
use crate::siblings::algorithms::{
|
||||
EntropyBias, SankoffBundle, Selection, SiblingAnnexStats, SnpAlignment,
|
||||
EntropyBias, SankoffBundle, Selection, SiblingAnnexStats, SnpAlignment, SnpDistanceKind,
|
||||
build_layer_sibling_annex, family_entropy, family_entropy_4, is_fast_mode,
|
||||
sankoff_bundle, scan_layer_families, sibling_annex_stats, sibling_family_size_histogram,
|
||||
snp_pseudo_alignment,
|
||||
snp_distance, snp_pseudo_alignment,
|
||||
};
|
||||
use crate::siblings::extensions::SiblingBuilder;
|
||||
use crate::siblings::ENTROPY_ANNEX_FILE_NAME;
|
||||
@@ -121,6 +122,38 @@ pub trait SiblingExt {
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
ratio_ceiling: f64,
|
||||
) -> OKIResult<SankoffBundle>;
|
||||
|
||||
/// A `snp-*` `--distance` matrix — one of the closed-form corrections
|
||||
/// in [`SnpDistanceKind`], over the central-position SNP substitution
|
||||
/// spectrum (requires an already-built sibling annex, run
|
||||
/// [`build_sibling_annex`](Self::build_sibling_annex) first).
|
||||
///
|
||||
/// `n`: `Some(target)` samples proportionally per layer exactly like
|
||||
/// [`sankoff_bundle`](Self::sankoff_bundle)/
|
||||
/// [`snp_pseudo_alignment`](Self::snp_pseudo_alignment) (`--subsample`);
|
||||
/// `None` means exhaustive — every non-monomorphic minorant of the
|
||||
/// whole index, not an approximation (see
|
||||
/// `algorithms::snp_distance::snp_distance`'s own docs for how this
|
||||
/// reuses the same sampling machinery at `p = 1` rather than a second
|
||||
/// driver). `free_loss`/`no_ambiguity`/`excluded`/`entropy_bias` — same
|
||||
/// meaning as `snp_pseudo_alignment`'s own (the last forced to `None`
|
||||
/// when `n` is `None`: entropy-biasing only makes sense for a genuine
|
||||
/// subsample).
|
||||
///
|
||||
/// `gamma_shape`: `--gamma-shape`, the Jin-Nei rate-heterogeneity
|
||||
/// correction — `None` disables it, `Some(alpha)` is rejected with
|
||||
/// [`OKIError::InvalidInput`] for a `kind` that doesn't support it
|
||||
/// ([`SnpDistanceKind::supports_gamma`]).
|
||||
fn snp_distance(
|
||||
&self,
|
||||
kind: SnpDistanceKind,
|
||||
n: Option<usize>,
|
||||
free_loss: bool,
|
||||
no_ambiguity: bool,
|
||||
excluded: &[bool],
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
gamma_shape: Option<f64>,
|
||||
) -> OKIResult<Array2<f64>>;
|
||||
}
|
||||
|
||||
impl SiblingExt for IndexCache {
|
||||
@@ -265,4 +298,17 @@ impl SiblingExt for IndexCache {
|
||||
) -> OKIResult<SankoffBundle> {
|
||||
sankoff_bundle(self, n, free_loss, no_ambiguity, excluded, entropy_bias, ratio_ceiling)
|
||||
}
|
||||
|
||||
fn snp_distance(
|
||||
&self,
|
||||
kind: SnpDistanceKind,
|
||||
n: Option<usize>,
|
||||
free_loss: bool,
|
||||
no_ambiguity: bool,
|
||||
excluded: &[bool],
|
||||
entropy_bias: Option<EntropyBias>,
|
||||
gamma_shape: Option<f64>,
|
||||
) -> OKIResult<Array2<f64>> {
|
||||
snp_distance(self, kind, n, free_loss, no_ambiguity, excluded, entropy_bias, gamma_shape)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -35,8 +35,8 @@ pub use siblingannex::{FamilyMask, SiblingAnnex, SiblingAnnexBuilder};
|
||||
|
||||
pub use algorithms::{
|
||||
BasePairTally, CardinalityTally, EntropyBias, RawSnpDistanceOutput, SankoffBundle,
|
||||
SiblingAnnexStats, SnpAlignment, cardinality_transition_probs, composition_transition_probs,
|
||||
pairwise_cost_matrix,
|
||||
SiblingAnnexStats, SnpAlignment, SnpDistanceKind, cardinality_transition_probs,
|
||||
composition_transition_probs, pairwise_cost_matrix,
|
||||
};
|
||||
pub use extensions::SiblingExt;
|
||||
|
||||
|
||||
Reference in New Issue
Block a user