Implement SNP distance models with gamma correction and PHYLIP output
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
This commit is contained in:
Eric Coissac
@@ -1168,7 +1168,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</code></pre></div>
|
||||
<hr/>
|
||||
<h2 id="matrix-types">Matrix types</h2>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has two variants (<code>Columnar</code>, <code>Packed</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<p>Both matrix types are enums behind a transparent API — the caller never matches on the variant. <code>PersistentCompactIntMatrix</code> has three variants (<code>Columnar</code>, <code>Packed</code>, <code>Sparse</code>). <code>PersistentBitMatrix</code> has four:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
@@ -1185,12 +1185,12 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Packed</code></td>
|
||||
<td>single <code>matrix.pbmx</code> mmap file</td>
|
||||
<td>single <code>matrix.pbmx</code>/<code>matrix.pcmx</code> mmap file</td>
|
||||
<td>query-optimised, produced by <code>pack_bit_matrix</code>/<code>pack_compact_int_matrix</code></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>Sparse</code> (bit only)</td>
|
||||
<td><code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major</td>
|
||||
<td><code>Sparse</code></td>
|
||||
<td>bit: <code>sparse_meta.json</code> + PFIV/Elias-Fano component files, row-major. Int: same support files (built on <code>PersistentSparseBitMatrix</code> internally) plus <code>singleton_values.pciv</code>/<code>multi_values.pciv</code>/<code>multi_offsets</code> for the per-row, non-deduplicated values</td>
|
||||
<td><code>pack --sparse</code>; see <a href="../../architecture/siblings/">siblings.md</a> for the sparse-vs-dense access-pattern trade-off</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -1200,7 +1200,8 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p><code>PersistentBitMatrix::open(layer_dir)</code> auto-detects the variant, in order: <code>matrix.pbmx</code> → Packed, <code>presence/meta.json</code> → Columnar, <code>presence/sparse_meta.json</code> → Sparse, <code>layer_meta.json</code> (no presence dir at all) → Implicit. <code>PersistentCompactIntMatrix::open(layer_dir)</code> mirrors the same priority order minus <code>Implicit</code> (there's no implicit count matrix — counts always have at least one on-disk column): <code>matrix.pcmx</code> → Packed, <code>counts/meta.json</code> → Columnar, <code>counts/singleton_values.pciv</code> → Sparse. <code>col_view</code>/<code>col</code>/<code>sub_matrix</code> panic on <code>Sparse</code>/<code>Implicit</code> where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through <code>row</code>/<code>fill_row</code>.</p>
|
||||
<p>Unlike the bit side, <code>PersistentSparseCompactIntMatrix</code>'s values are <em>not</em> deduplicated across rows — two rows can share the same non-zero column set (same <code>dict_id</code> in the shared support) while carrying different counts — so its <code>CountPartials</code> impl can't reuse the support's dict-multiplicity shortcut the way <code>BitPartials for PersistentSparseBitMatrix</code> does. It still avoids the naive <code>O(n_cols² × n)</code> column-pair scan via a single row-major pass (<code>row_major_pairwise</code> in <code>sparse_intmatrix.rs</code>), reconstructing the squared-difference formulas (<code>euclidean</code>/<code>relfreq_euclidean</code>/<code>hellinger</code>) from per-column marginals via <code>Σ(a-b)² = Σa²+Σb²-2Σab</code> — see <a href="../../architecture/siblings/">siblings.md</a>'s "<code>PersistentCompactIntMatrix::Sparse</code> — implemented" entry for the full derivation.</p>
|
||||
<p><code>col_view(c)</code> returns the appropriate view directly:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// PersistentBitMatrix</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">col_view</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">c</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">BitSliceView</span><span class="o"><'</span><span class="nb">_</span><span class="o">></span>
|
||||
|
||||
@@ -1213,6 +1213,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1719,6 +1835,122 @@
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(6) done (2026-08-21): Counter — third algorithm, extracted the same way as Dereplicator
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(7) done (2026-08-21): LayerBuilder — fourth and last pipeline algorithm
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="(8) design agreed, not yet implemented (2026-08-21): obikalgorithm::Algorithm trait + obikindexer::extensions — private/public extension-trait split, KmerLayer rename">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#why-this-came-up" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Why this came up
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#the-general-pattern-not-obikindexer-specific" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
The general pattern (not obikindexer-specific)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#concretely-next-to-implement-two-items-in-order" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Concretely, next to implement (two items, in order)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(9) done (2026-08-21): obikindexer::extensions::PrivateBuilder — item 1 above, implemented
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(10) done (2026-08-21): obikindex::IndexBuilder — the public counterpart, same session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(11) done (2026-08-21): KmerIndex/IndexMeta made fully stateless, IndexState moved off sentinel files
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
(12) done (2026-08-21): obikalgorithm::Algorithm — the shared trait, resumed and closed in one session
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -1853,6 +2085,21 @@ The underlying module boundary and its rationale (Layer tier / Partition
|
||||
tier / Index tier, each depending only downward) are unchanged; only the
|
||||
crate-vs-module packaging changed. See <a href="../layer_tier/">obikindex::layer</a>
|
||||
for the current module doc.</p>
|
||||
<p><strong>Superseded, second event, same day (2026-08-21):</strong> <code>obikpartitionner</code>
|
||||
and <code>obikderep</code>, the two algorithm crates, are also gone — but unlike
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> above, they were <strong>not</strong> folded into
|
||||
<code>obikindex</code>. They were first (mistakenly) merged into <code>obikindex</code> as an
|
||||
<code>algorithms</code> submodule, then corrected into a new sibling crate,
|
||||
<strong><code>obikindexer</code></strong>, holding <code>obikindexer::algorithms::{partitionner,
|
||||
dereplicator}</code> and depending on <code>obikindex</code> — never the reverse, same
|
||||
dependency direction <code>obikpartitionner</code>/<code>obikderep</code> already had. Read
|
||||
<code>obikpartitionner::X</code> as <code>obikindexer::algorithms::partitionner::X</code> and
|
||||
<code>obikderep::X</code> as <code>obikindexer::algorithms::dereplicator::X</code> throughout
|
||||
what follows. The distinction the mistake surfaced, worth keeping: data
|
||||
crates (<code>obikindex</code>, holding the <code>index</code>/<code>partition</code>/<code>layer</code> model) merge
|
||||
naturally into one crate as submodules; algorithm crates that operate on
|
||||
that model from outside stay separate, so the dependency only ever runs
|
||||
one way.</p>
|
||||
<p>Status (2026-08-20, latest pass): (1) done — <code>obilayeredmap::Layer</code>
|
||||
exists, <code>Mat</code> is gone. (1b) done — <code>Layer::Empty</code>, the first non-ready
|
||||
state, added (panics on every read method). (2a) done — the
|
||||
@@ -1872,7 +2119,24 @@ turned out to be bigger than <code>KmerPartition</code> alone: <code>Layer</code
|
||||
constructors don't self-name either. Full redesign of both, agreed in
|
||||
detail, session ended (budget) before implementation — see "(5) design
|
||||
agreed" below; <strong>read it before touching <code>KmerPartition</code>/<code>Layer</code>
|
||||
signatures</strong>, the shape is fully specified. Earlier mix-up, for
|
||||
signatures</strong>, the shape is fully specified. (6) done — <code>Counter</code>, a third
|
||||
algorithm, extracted from <code>PartitionRouter</code> the same way <code>Dereplicator</code>
|
||||
was in (4). (7) done — <code>LayerBuilder</code>, the fourth and last pipeline
|
||||
algorithm; the indexing pipeline is now fully decomposed into
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator, counter,
|
||||
layer_builder}</code>. (8) design agreed, item 1 done in (9) —
|
||||
<code>obikindexer::extensions::PrivateBuilder</code>, private, six construction-only
|
||||
<code>KmerIndex</code> methods moved out. (10) done, same session — <code>obikindex::
|
||||
IndexBuilder</code>, public, the four maintenance methods
|
||||
(<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>)
|
||||
shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>. Item 2 from (8)
|
||||
(<code>obikalgorithm::Algorithm</code>) done in (12) — new crate, <code>type Output</code> +
|
||||
<code>fn run(&mut self) -> SKResult<Self::Output></code>, <code>on_progress</code> moved off
|
||||
<code>run()</code>'s signature entirely into a per-algorithm <code>.on_progress(...)</code>
|
||||
setter. Note: <code>Layer</code> renamed
|
||||
<code>KmerLayer</code> (2026-08-21, outside this conversation). (5) itself still not
|
||||
implemented, still first on the "order of remaining work" list. Earlier
|
||||
mix-up, for
|
||||
context: an earlier
|
||||
version of this doc used the name <code>KmerPartition</code> (singular) for what was
|
||||
actually the <em>collection</em> type (later renamed <code>KmerPartitions</code>, later
|
||||
@@ -2515,6 +2779,622 @@ itself or a new crate.</p>
|
||||
examples were judged not enough to be sure of the shape (<code>Fn+Sync</code> vs
|
||||
<code>FnMut</code> callback bound already diverged between the two that exist).</li>
|
||||
</ol>
|
||||
<h2 id="6-done-2026-08-21-counter-third-algorithm-extracted-the-same-way-as-dereplicator">(6) done (2026-08-21): <code>Counter</code> — third algorithm, extracted the same way as <code>Dereplicator</code></h2>
|
||||
<p>Between (5) and this, the user did a session of their own crate
|
||||
restructuring (see the two "Superseded" notes at the top of this file):
|
||||
<code>obikpartition</code>/<code>obilayeredmap</code> folded into <code>obikindex</code> as submodules
|
||||
(<code>obikindex::partition</code>, <code>obikindex::layer</code>), and <code>obikpartitionner</code>/
|
||||
<code>obikderep</code> merged into one sibling crate, <code>obikindexer</code>, holding
|
||||
<code>obikindexer::algorithms::{partitionner, dereplicator}</code>. (5)'s design
|
||||
(<code>Layer</code>/<code>KmerPartition</code> self-naming by number, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code> → <code>&</code> fix) was <strong>not</strong> part of that — pure crate/module packaging,
|
||||
confirmed by reading the actual code (<code>Layer::open</code>/<code>create</code> still take an
|
||||
external <code>dir: &Path</code>, <code>KmerPartition</code> still eagerly opens all layers,
|
||||
<code>PartitionRouter</code> still holds <code>&mut KmerIndex</code>). (5) remains exactly as
|
||||
specified, not yet implemented.</p>
|
||||
<p>This step: <code>count_kmer</code> (still living on <code>PartitionRouter</code>, per (4)'s own
|
||||
"still not done" note) extracted into <code>obikindexer::algorithms::counter::
|
||||
Counter</code>, mirroring <code>Dereplicator</code> exactly — third data point for the
|
||||
eventual <code>obikalgorithm</code> trait, still not extracted (still only 3 examples
|
||||
with 2 different callback bounds; holding off per (5)'s "order of
|
||||
remaining work").</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_partial</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">Counter</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_partial</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span><span class="w"> </span><span class="c1">// setter, mirrors PartitionRouter's style; defaults to false</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">Fn</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="n">KmerSpectrum</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Same shape as <code>Dereplicator</code> throughout: <code>Fn(Progress) + Sync</code> (not
|
||||
<code>FnMut</code>) since counting is also a parallel <code>par_iter</code> over partitions, an
|
||||
<code>AtomicU64</code> position counter incremented from inside the parallel closure
|
||||
so progress reports arrive in real time rather than bursting at the end
|
||||
once <code>.collect()</code> finishes, <code>total: Some(n_partitions)</code> (known up front).
|
||||
<code>KmerSpectrum</code> (the <code>{f0, f1, counts}</code> aggregate) moved from
|
||||
<code>partitionner::router</code> to <code>counter</code>, since it's <code>Counter::run</code>'s return
|
||||
value now, not <code>PartitionRouter</code>'s. <code>count.rs</code>/<code>kmer_sort.rs</code> moved
|
||||
verbatim from <code>partitionner/</code> to <code>counter/</code> (unchanged bodies — only
|
||||
<code>count_kmer</code> itself, <code>KmerSpectrum</code>, and the imports they pulled in were
|
||||
removed from <code>router.rs</code>).</p>
|
||||
<p>One divergence from <code>Dereplicator</code>: a <code>keep_partial</code> setter exists (no
|
||||
equivalent on <code>Dereplicator</code>, which has no setters at all) — a real,
|
||||
already-present parameter (<code>keep_intermediate</code> at the CLI), not a
|
||||
speculative addition.</p>
|
||||
<p><code>count_kmer</code>'s three former callers (<code>obikmer::cmd::index</code>, <code>obikphylo</code>'s
|
||||
test harness, <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test helper) all updated to <code>Counter::new(&idx).
|
||||
run(...)</code> — the last one simplified further: it used to read back
|
||||
<code>kmer_spectrum_raw.json</code> from disk after calling <code>count_partition</code>
|
||||
directly (white-box), now it just uses the <code>KmerSpectrum</code> <code>Counter::run</code>
|
||||
already returns.</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus an end-to-end CLI smoke test
|
||||
against real FASTA data (scatter → dereplicate → count → index-build →
|
||||
query) — required every time per (3)'s lesson, and it earned its keep
|
||||
again: the very first smoke-test query returned zero matches, which
|
||||
looked like a regression until traced to the query sequence itself being
|
||||
low-complexity ("GGCCCCCCACG", six same-base runs) and rejected by
|
||||
<em>query's own</em> default entropy threshold — nothing to do with this change.
|
||||
Re-tested with a different substring, confirmed working (kmer found,
|
||||
count matched the index).</p>
|
||||
<p>Still not done: (5) (<code>Layer</code>/<code>KmerPartition</code> redesign, <code>PartitionRouter</code>'s
|
||||
<code>&mut</code>→<code>&</code>), the future cache crate, <code>build_layers</code> (still a <code>KmerIndex</code>
|
||||
inherent method, not an algorithm), and <code>obikalgorithm</code> itself.</p>
|
||||
<h2 id="7-done-2026-08-21-layerbuilder-fourth-and-last-pipeline-algorithm">(7) done (2026-08-21): <code>LayerBuilder</code> — fourth and last pipeline algorithm</h2>
|
||||
<p>Closes out the indexing pipeline: <code>build_layers</code>/<code>build_index_layer</code>
|
||||
(the last stage still living as <code>KmerIndex</code> inherent methods, flagged as
|
||||
inconsistent since (6)) extracted into <code>obikindexer::algorithms::
|
||||
layer_builder::LayerBuilder</code>, same two-phase shape as the other three.</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">struct</span><span class="w"> </span><span class="nc">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">n_partitions</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">min_abundance</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">max_abundance</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span>
|
||||
<span class="w"> </span><span class="n">keep_intermediate</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span>
|
||||
<span class="p">}</span>
|
||||
|
||||
<span class="k">impl</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="n">LayerBuilder</span><span class="o"><'</span><span class="na">a</span><span class="o">></span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">new</span><span class="p">(</span><span class="n">index</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="o">'</span><span class="na">a</span><span class="w"> </span><span class="nc">KmerIndex</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">min_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">max_abundance</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">keep_intermediate</span><span class="p">(</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">v</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">Self</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">pub</span><span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">on_progress</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="k">impl</span><span class="w"> </span><span class="nb">FnMut</span><span class="p">(</span><span class="n">Progress</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">SKResult</span><span class="o"><</span><span class="kt">usize</span><span class="o">></span><span class="p">;</span><span class="w"> </span><span class="c1">// returns total kmers built</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p><strong>Different from all three prior extractions in one respect, deliberately
|
||||
not "fixed" to match them</strong>: the actual per-partition construction logic
|
||||
(De Bruijn graph from dereplicated superkmers + provisional counts →
|
||||
unitigs → MPHF → matrix) stayed put as <code>KmerIndex::build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> (both already <code>pub</code>) — not moved into
|
||||
<code>obikindexer</code>. Checked first: unlike <code>dereplicate_partition</code>/
|
||||
<code>count_partition</code> (which only ever had one caller), <code>build_index_layer</code>
|
||||
depends on several <code>obikindex</code>-internal helpers (<code>graph_pipeline::
|
||||
{write_graph_as_unitigs, materialize_layer}</code>, <code>common::olm_to_sk</code>) that
|
||||
are <code>pub(crate)</code> and shared with <code>merge</code>/<code>select</code>/<code>rebuild</code>'s own
|
||||
layer-construction paths — moving <code>build_index_layer</code> out would have
|
||||
meant either exporting that internal surface just for this one algorithm
|
||||
or duplicating it. Neither was needed: <code>build_index_layer</code>/
|
||||
<code>remove_build_artifacts</code> were <em>already</em> public <code>KmerIndex</code> methods, so
|
||||
<code>LayerBuilder</code>'s job is purely the orchestration around them (scheduling,
|
||||
config, progress) — the exact same "algorithm calls already-public
|
||||
<code>KmerIndex</code> primitives" shape <code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>
|
||||
already have, just at a coarser grain for this one stage. This is the
|
||||
"is the producer's API actually deficient?" check from
|
||||
[[feedback_no_spaghetti_petits_pois]] applied and answered "no" — not
|
||||
skipped.</p>
|
||||
<p><strong>Two more real divergences, both forced by <code>PartitionRunner</code>, not
|
||||
arbitrary:</strong>
|
||||
- Uses <code>obikindex::PartitionRunner</code> (NUMA-aware scheduler, already
|
||||
<code>pub use</code>d from <code>obikindex</code>) instead of plain <code>rayon::into_par_iter</code>
|
||||
like <code>Dereplicator</code>/<code>Counter</code> — matches what <code>build_layers</code> already used
|
||||
before extraction; this stage is more CPU/memory-intensive per partition
|
||||
(graph construction) than scatter/dereplicate/count.
|
||||
- Callback bound is <code>FnMut(Progress) + Send</code> — a third variant, not
|
||||
matching either prior shape. <code>PartitionRunner::run</code>'s <code>on_done</code> is
|
||||
invoked from its own single controller thread (never concurrently, so
|
||||
no <code>Sync</code> needed, unlike <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn + Sync</code>), but
|
||||
that controller thread is itself <code>std::thread::scope</code>-spawned, so the
|
||||
closure still has to be <code>Send</code> to cross into it — caught immediately by
|
||||
the compiler (<code>cannot be sent between threads safely</code>) when <code>Send</code> was
|
||||
first omitted, not a design guess. <code>obikalgorithm</code>'s eventual shared
|
||||
trait now has three real callback-bound data points to reconcile
|
||||
(<code>FnMut</code> alone for <code>PartitionRouter::run</code>'s sequential loop, <code>FnMut +
|
||||
Send</code> here, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code>'s <code>rayon</code>
|
||||
<code>par_iter</code>), not two.</p>
|
||||
<p><code>KmerIndex::build_layers</code> deleted outright (<code>KmerIndex</code> stays a pure data
|
||||
structure — no compute orchestration methods, consistent with <code>dereplicate</code>/
|
||||
<code>count_kmer</code>'s removal in (4)/(6)). New <code>KmerIndex::mark_indexed()</code> added,
|
||||
symmetric to <code>mark_scattered</code>/<code>mark_counted</code>, replacing the inline
|
||||
<code>touch(SENTINEL_INDEXED)</code> that used to live inside <code>build_layers</code>.
|
||||
<code>Stage::start("index")</code>/<code>rep.push(...)</code> and the <code>progress_bar</code>/
|
||||
<code>"{n} total kmers indexed"</code> log line both moved to <code>cmd/index/mod.rs</code>,
|
||||
same pattern as (3)/(4)/(6) — <code>LayerBuilder</code> renders nothing itself, just
|
||||
reports <code>Progress</code>.</p>
|
||||
<p>All callers updated: <code>cmd/index/mod.rs</code> (Stage 3), <code>obikphylo</code>'s test
|
||||
harness (also gained a <code>mark_indexed()</code> call it was missing — harmless
|
||||
before since nothing checked <code>IndexState::Indexed</code> in that test, but now
|
||||
correct).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the end-to-end CLI smoke test
|
||||
(<code>scripts/smoke_test_index.sh</code>, built earlier specifically so this
|
||||
verification step is a one-liner from now on) — 870 kmers indexed, query
|
||||
round-trip confirmed, same numbers as (6).</p>
|
||||
<p><strong>The indexing pipeline is now fully decomposed</strong>: <code>obikindexer::
|
||||
algorithms::{partitionner, dereplicator, counter, layer_builder}</code>, each a
|
||||
<code>new</code>/(setters)/<code>run</code> algorithm operating on a <code>&KmerIndex</code> (or <code>&mut</code> for
|
||||
<code>PartitionRouter</code>, not yet fixed — see (5)), <code>KmerIndex</code> itself holding no
|
||||
pipeline-orchestration logic anymore. Still not done: (5), the future
|
||||
cache crate, <code>obikalgorithm</code> (now unblocked — three real callback-bound
|
||||
variants observed, worth revisiting whether a single trait can express
|
||||
all three or whether that's itself the answer: it can't, and the trait
|
||||
should not force it).</p>
|
||||
<h2 id="8-design-agreed-not-yet-implemented-2026-08-21-obikalgorithmalgorithm-trait-obikindexerextensions-privatepublic-extension-trait-split-kmerlayer-rename">(8) design agreed, not yet implemented (2026-08-21): <code>obikalgorithm::Algorithm</code> trait + <code>obikindexer::extensions</code> — private/public extension-trait split, <code>KmerLayer</code> rename</h2>
|
||||
<p>Session note: <code>Layer</code> was renamed <code>KmerLayer</code> (user, outside this
|
||||
conversation, alongside other naming homogenisation with <code>KmerIndex</code>/
|
||||
<code>KmerPartition</code>) — every reference to <code>Layer</code> in this doc from before
|
||||
2026-08-21 means today's <code>obikindex::layer::KmerLayer</code>.</p>
|
||||
<h3 id="why-this-came-up">Why this came up</h3>
|
||||
<p>Verifying "does <code>cmd/index</code> now rest entirely on the algorithm structs"
|
||||
(it doesn't quite — see below) led to sorting <code>KmerIndex</code>'s own methods by
|
||||
a criterion the user was explicit is <strong>semantic, not mechanical</strong>: "les
|
||||
méthodes qui, sémantiquement, n'ont pas d'intérêt hors de la construction
|
||||
de l'index" (methods that have no semantic interest outside index
|
||||
construction) — not "methods only called from <code>cmd/index</code> today," which
|
||||
a grep could answer but would miss methods construction-adjacent code
|
||||
elsewhere (<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>) also depends on for the
|
||||
same reason.</p>
|
||||
<p><strong>Checked, not assumed</strong> (grepped every call site before classifying):</p>
|
||||
<ul>
|
||||
<li><strong>Construction-only, real candidates for a private extension trait</strong>:
|
||||
<code>KmerIndex::{mark_scattered, mark_counted, mark_indexed, write_spectrum,
|
||||
build_index_layer, remove_build_artifacts, clear_output_for_create,
|
||||
create_skeleton, finalize_indexed, state}</code>. The last four are called
|
||||
from <code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> too (as
|
||||
precondition checks — "is my source <code>Indexed</code>?" — or shared
|
||||
skeleton/finalize machinery), not just from the 4-stage pipeline — so
|
||||
this extension trait's scope is "construction of any kind," not
|
||||
narrowly "the initial build pipeline."</li>
|
||||
<li><strong>Looked construction-only by name, checked, and kept on <code>KmerIndex</code></strong>:
|
||||
<code>layer_unitigs_path</code> (unitigs are the only way to recover a built
|
||||
index's kmer sequences — read by <code>rebuild_layer.rs</code> and others, well
|
||||
beyond construction — see [[project_unitigs_always_kept]]),
|
||||
<code>pack_matrices</code> (re-runnable maintenance on an already-finished index
|
||||
via <code>obikmer pack</code>, not just a pipeline step), <code>upgrade_layer_meta</code>
|
||||
(migration, runnable on any existing index at any time).</li>
|
||||
</ul>
|
||||
<h3 id="the-general-pattern-not-obikindexer-specific">The general pattern (not obikindexer-specific)</h3>
|
||||
<p><code>KmerIndex</code>/<code>KmerPartition</code>/<code>KmerLayer</code> stay generic, in <code>obikindex</code> —
|
||||
every domain-specific consumer crate gets to attach its own extension
|
||||
trait(s), of two kinds:</p>
|
||||
<ul>
|
||||
<li><strong>Private</strong> (<code>pub(crate)</code>, invisible outside the defining crate) — for
|
||||
plumbing only that crate's own algorithms need. <code>obikindexer</code> gets
|
||||
exactly one of these (see below); no public counterpart makes sense for
|
||||
it — "l'index est tellement central que le second trait n'a pas
|
||||
vraiment d'intérêt" for construction specifically: nothing external
|
||||
should ever want to call <code>mark_scattered</code> or <code>build_index_layer</code>.</li>
|
||||
<li><strong>Public</strong> — for a genuinely reusable domain extension. The user's own
|
||||
example, found while discussing this, not hypothetical: <code>obikindex/src/
|
||||
index/distance.rs</code> (phylogenetic distance metrics) is currently an
|
||||
<code>impl KmerIndex</code> block <strong>inside <code>obikindex</code> itself</strong> — under this
|
||||
principle it should be a public extension trait owned by <code>obikphylo</code>
|
||||
instead (distance metrics are a phylo concept, <code>obikindex</code> has no more
|
||||
business defining them than <code>obikindex::layer</code> has defining
|
||||
"family"/"minorant", the reasoning <code>SiblingLayerExt</code> already followed
|
||||
for <code>KmerLayer</code> — see <code>obikphylo/src/siblings/iter.rs</code>). <strong>Explicitly
|
||||
deferred</strong> — noted here so it isn't lost, not part of this round.</li>
|
||||
<li>The future cache-manager crate (still blocked on (5), see above) will
|
||||
add its own <strong>public</strong> extension trait mirroring part of <code>KmerIndex</code>'s/
|
||||
<code>KmerPartition</code>'s own read API in cached form (e.g. a cached
|
||||
<code>.partition(i)</code> that doesn't re-touch disk) — same pattern, third data
|
||||
point once built.</li>
|
||||
</ul>
|
||||
<h3 id="concretely-next-to-implement-two-items-in-order">Concretely, next to implement (two items, in order)</h3>
|
||||
<ol>
|
||||
<li><strong><code>obikindexer::extensions</code></strong> — a private (<code>pub(crate)</code>) extension
|
||||
trait, most likely named something like <code>IndexBuildExt</code> (final name
|
||||
not yet chosen), implemented for <code>KmerIndex</code>, carrying the ten methods
|
||||
listed above, moved out of <code>obikindex::index::{kmer_index,
|
||||
index_layer}</code>. Every algorithm in <code>obikindexer::algorithms::*</code> that
|
||||
currently calls <code>idx.mark_scattered()</code>/etc. keeps the same call syntax
|
||||
(extension trait methods are called the same way as inherent ones,
|
||||
just need the trait in scope) — <code>cmd/index/mod.rs</code> itself would need
|
||||
<code>use obikindexer::extensions::IndexBuildExt;</code> (or the module re-exports
|
||||
it) to keep compiling, since it's the one place outside <code>obikindexer</code>'s
|
||||
own algorithms that currently calls <code>mark_scattered</code>/<code>write_spectrum</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code> directly. <strong>Not yet decided</strong>: exact
|
||||
trait name, whether it's one trait or split further (e.g. sentinel
|
||||
marking vs. skeleton/finalize machinery), and whether <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> (not yet extracted into algorithms themselves) move
|
||||
onto it now too or keep calling the soon-to-be-inherent-no-longer
|
||||
methods some other way in the meantime — <strong>ask before implementing</strong>,
|
||||
this changes the blast radius significantly (4 more <code>obikindex</code>
|
||||
internal files depend on <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code>).</li>
|
||||
<li><strong><code>obikalgorithm::Algorithm</code> trait</strong> — new crate, the shared trait
|
||||
<code>obikpartitionner</code>→<code>obikindexer</code> merge (session start of 2026-08-21)
|
||||
and (6)/(7) were deliberately building toward, now with four real
|
||||
<code>new</code>/(setters)/<code>run</code> examples and three distinct callback-bound
|
||||
shapes to reconcile (plain <code>FnMut</code> for <code>PartitionRouter</code>, <code>FnMut +
|
||||
Send</code> for <code>LayerBuilder</code>, <code>Fn + Sync</code> for <code>Dereplicator</code>/<code>Counter</code> —
|
||||
see (7)). Exact shape not yet drafted in this doc — do that as its own
|
||||
design pass before coding, same discipline as everything above.</li>
|
||||
</ol>
|
||||
<p>Both items: <strong>design only, nothing implemented yet</strong> — this section is
|
||||
the record to resume from, not a plan already executed.</p>
|
||||
<h2 id="9-done-2026-08-21-obikindexerextensionsprivatebuilder-item-1-above-implemented">(9) done (2026-08-21): <code>obikindexer::extensions::PrivateBuilder</code> — item 1 above, implemented</h2>
|
||||
<p>Renamed from <code>IndexBuilder</code> to <code>PrivateBuilder</code> immediately after (same
|
||||
session), freeing the name <code>IndexBuilder</code> for (10)'s public trait — read
|
||||
<code>IndexBuilder</code> below as <code>PrivateBuilder</code> throughout this section.</p>
|
||||
<p>Scoped down from (8)'s six-method list to the concrete set that's
|
||||
genuinely movable without further ripple — checked, not assumed, before
|
||||
writing anything:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="p">(</span><span class="k">crate</span><span class="p">)</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_scattered</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_counted</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">mark_indexed</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">write_spectrum</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">f0</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">f1</span><span class="p">:</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">counts</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">BTreeMap</span><span class="o"><</span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="kt">u64</span><span class="o">></span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">build_index_layer</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">min_ab</span><span class="p">:</span><span class="w"> </span><span class="kt">u32</span><span class="p">,</span><span class="w"> </span><span class="n">max_ab</span><span class="p">:</span><span class="w"> </span><span class="nb">Option</span><span class="o"><</span><span class="kt">u32</span><span class="o">></span><span class="p">,</span><span class="w"> </span><span class="n">with_counts</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">,</span><span class="w"> </span><span class="n">mode</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMode</span><span class="p">,</span><span class="w"> </span><span class="n">block_bits</span><span class="p">:</span><span class="w"> </span><span class="kt">u8</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="kt">usize</span><span class="p">,</span><span class="w"> </span><span class="n">SKError</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">remove_build_artifacts</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">,</span><span class="w"> </span><span class="n">i</span><span class="p">:</span><span class="w"> </span><span class="kt">usize</span><span class="p">);</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">PrivateBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>All six moved bodily out of <code>obikindex::index::{kmer_index, index_layer}</code>
|
||||
into <code>obikindexer::extensions</code> (new module, <code>pub(crate)</code>) — <code>index_layer.rs</code>
|
||||
is now empty and deleted outright.
|
||||
<code>clear_output_for_create</code>/<code>create_skeleton</code>/<code>finalize_indexed</code>/<code>state</code>
|
||||
stayed inherent on <code>KmerIndex</code>, per (8)'s reasoning: <code>merge</code>/<code>select</code>/
|
||||
<code>rebuild</code>/<code>reindex</code> — living <em>inside</em> <code>obikindex</code> itself — call them too,
|
||||
and <code>obikindex</code> can never depend on <code>obikindexer</code> to reach a trait defined
|
||||
there. Moving those four is real future work (extract
|
||||
merge/select/rebuild/reindex into algorithms first), not part of this
|
||||
step.</p>
|
||||
<p><strong>One new, small, deliberate API widening in <code>obikindex</code></strong>: <code>build_index_layer</code>
|
||||
depends on three helpers that were <code>pub(crate)</code> to <code>obikindex</code>
|
||||
(<code>graph_pipeline::{write_graph_as_unitigs, materialize_layer}</code>,
|
||||
<code>common::olm_to_sk</code>) — widened to <code>pub</code> (re-exported from <code>obikindex</code>'s
|
||||
crate root) so <code>obikindexer</code> could reach them. This is exactly the
|
||||
"enrich shared/lower-level APIs instead of ad hoc local code" call the
|
||||
project's own rules ask for, made explicitly rather than routed around:
|
||||
three functions, already generically written (no rewrite needed), now
|
||||
serve a second caller instead of being duplicated.</p>
|
||||
<p><strong>Why the trait had to be defined in <code>obikindexer</code>, not <code>obikindex</code></strong>:
|
||||
Rust's orphan rule — implementing a trait for a foreign type requires
|
||||
either the trait or the type to be local to the current crate. <code>KmerIndex</code>
|
||||
is foreign to <code>obikindexer</code>, so the trait must be the local half; if it
|
||||
were defined in <code>obikindex</code> instead, <code>pub(crate)</code> there would make it
|
||||
invisible to <code>obikindexer</code> too (crate-private means private to <em>that</em>
|
||||
crate, not "private except to one named dependent") — the opposite of
|
||||
what was wanted.</p>
|
||||
<p><strong>A real design decision made while wiring callers up, not a mechanical
|
||||
rename</strong>: <code>PrivateBuilder</code> being genuinely <code>pub(crate)</code> to <code>obikindexer</code>
|
||||
means <code>obikmer::cmd::index</code> (a different crate) can no longer call
|
||||
<code>mark_scattered</code>/<code>mark_counted</code>/<code>mark_indexed</code>/<code>write_spectrum</code> directly —
|
||||
it never could have, once privacy was real rather than aspirational. Each
|
||||
algorithm now marks its own completion as part of <code>run()</code>/<code>close()</code>
|
||||
instead of leaving it to the caller:
|
||||
- <code>PartitionRouter::close()</code> (not <code>run()</code>) calls <code>mark_scattered()</code> —
|
||||
<code>close()</code>, not <code>run()</code>, is the actual shared completion point between
|
||||
the file-driven <code>run()</code> path and the manual <code>write</code>/<code>write_batch</code>+
|
||||
<code>close()</code> path low-level callers (tests) use; putting it in <code>run()</code>
|
||||
alone would have silently skipped marking for every caller that never
|
||||
calls <code>run()</code>. <code>run()</code> already calls <code>self.close()</code> at its own end, so
|
||||
this covers both paths through one line, not two.
|
||||
- <code>Counter::run</code> calls <code>write_spectrum</code> then <code>mark_counted</code> before
|
||||
returning.
|
||||
- <code>LayerBuilder::run</code> calls <code>mark_indexed</code> before returning.</p>
|
||||
<p><code>cmd/index/mod.rs</code> lost all four direct calls (<code>mark_scattered</code>/
|
||||
<code>write_spectrum</code>/<code>mark_counted</code>/<code>mark_indexed</code>) — each stage's <code>if
|
||||
idx.state() < IndexState::X { ... }</code> block is now purely "run the
|
||||
algorithm," no separate bookkeeping call after it. Confirms, precisely
|
||||
this time (checked by re-reading the whole file, not assumed): <code>cmd/index</code>
|
||||
now rests on the four algorithms for every read/write of pipeline state
|
||||
except <code>KmerIndex::{exists, create, state, n_partitions}</code>, which are
|
||||
genuinely index-identity concerns, not construction bookkeeping — the
|
||||
original question this whole design pass started from.</p>
|
||||
<p>Same fix applied to <code>obikphylo</code>'s test harness (its four explicit
|
||||
<code>mark_*</code>/<code>write_spectrum</code> calls removed, relying on the algorithms now
|
||||
doing it themselves) — <code>obikindexer::algorithms::partitionner</code>'s own
|
||||
<code>pipeline_counts</code> test needed no change (never called <code>mark_*</code> directly).</p>
|
||||
<p>Full workspace suite green (<code>cargo check --workspace --all-targets</code> +
|
||||
<code>cargo test --workspace</code>, exit code 0), plus the CLI smoke test — 870
|
||||
kmers, same as (6)/(7).</p>
|
||||
<p>Still not done at the time of writing: item 2 from (8) (<code>obikalgorithm::
|
||||
Algorithm</code>), (5), the future cache crate, the <code>distance.rs</code> →
|
||||
<code>obikphylo</code> relocation (noted in (8), explicitly deferred), and
|
||||
extracting <code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code> into algorithms.</p>
|
||||
<h2 id="10-done-2026-08-21-obikindexindexbuilder-the-public-counterpart-same-session">(10) done (2026-08-21): <code>obikindex::IndexBuilder</code> — the public counterpart, same session</h2>
|
||||
<p>Immediate correction to (9): the private trait built there was renamed
|
||||
<code>PrivateBuilder</code> (freeing the name), and the four methods (9) had left
|
||||
inherent on <code>KmerIndex</code> — <code>clear_output_for_create</code>/<code>create_skeleton</code>/
|
||||
<code>finalize_indexed</code>/<code>state</code> — got their own trait after all: <strong><code>IndexBuilder</code></strong>,
|
||||
public, defined in <code>obikindex</code> itself (not <code>obikindexer</code>):</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="p">:</span><span class="w"> </span><span class="nb">Sized</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">clear_output_for_create</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">force</span><span class="p">:</span><span class="w"> </span><span class="kt">bool</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">create_skeleton</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">meta</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">IndexMeta</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">finalize_indexed</span><span class="o"><</span><span class="n">P</span><span class="p">:</span><span class="w"> </span><span class="nb">AsRef</span><span class="o"><</span><span class="n">Path</span><span class="o">>></span><span class="p">(</span><span class="n">output</span><span class="p">:</span><span class="w"> </span><span class="nc">P</span><span class="p">,</span><span class="w"> </span><span class="n">rep</span><span class="p">:</span><span class="w"> </span><span class="kp">&</span><span class="nc">mut</span><span class="w"> </span><span class="n">Reporter</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">OKIResult</span><span class="o"><</span><span class="bp">Self</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">state</span><span class="p">(</span><span class="o">&</span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nc">IndexState</span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
<span class="k">impl</span><span class="w"> </span><span class="n">IndexBuilder</span><span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">KmerIndex</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="o">..</span><span class="p">.</span><span class="w"> </span><span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>User's framing: these four are "maintenance", not "scientific computation
|
||||
on an index" — a different kind of non-generic-ness than (9)'s six
|
||||
(<code>mark_*</code>/<code>write_spectrum</code>/<code>build_index_layer</code>/<code>remove_build_artifacts</code>,
|
||||
exclusive to the 4-stage pipeline). Maintenance is used more broadly
|
||||
(<code>merge</code>/<code>select</code>/<code>rebuild</code>/<code>reindex</code>), so it gets a real, public trait —
|
||||
not folded back into <code>KmerIndex</code>'s inherent surface, and not private
|
||||
either.</p>
|
||||
<p><strong>Where it lives, and why that's not arbitrary</strong>: (9) needed the orphan
|
||||
rule to force its trait into <code>obikindexer</code>, to achieve genuine
|
||||
crate-private visibility. Here the requirement is the opposite:
|
||||
<code>merge.rs</code>/<code>select.rs</code>/<code>rebuild.rs</code>/<code>reindex.rs</code> — the trait's own
|
||||
heaviest users — live <em>inside</em> <code>obikindex</code>. A trait they need to reach
|
||||
must be local to <code>obikindex</code> (or a crate <code>obikindex</code> itself depends on,
|
||||
which doesn't exist for this). So <code>IndexBuilder</code> lives in a new
|
||||
<code>obikindex/src/index/builder.rs</code>, <code>pub trait</code> (no orphan-rule tension at
|
||||
all here — both trait and type are local to the same crate), re-exported
|
||||
from <code>obikindex</code>'s crate root alongside <code>PrivateBuilder</code>'s sibling
|
||||
<code>obikindexer::extensions::PrivateBuilder</code> staying where it is. Two
|
||||
traits, two crates, two different reasons, not a contradiction.</p>
|
||||
<p><strong>Blast radius, all inside <code>obikindex</code> plus one external crate</strong>: every
|
||||
internal caller of these four methods needs the trait imported now that
|
||||
they're no longer inherent — <code>merge.rs</code>, <code>select.rs</code>, <code>rebuild.rs</code>,
|
||||
<code>reindex.rs</code> (<code>use crate::index::builder::IndexBuilder;</code>) and, externally,
|
||||
<code>obikmer::cmd::index::mod</code> (<code>use obikindex::IndexBuilder;</code>, for the three
|
||||
<code>idx.state() < IndexState::X</code> resumability checks). Call syntax at every
|
||||
site is unchanged (<code>KmerIndex::create_skeleton(...)</code>,
|
||||
<code>self.state()</code>) — only trait-in-scope requirements are new, which is
|
||||
exactly the point: same ergonomics, less surface baked into <code>KmerIndex</code>
|
||||
itself.</p>
|
||||
<p>Verification went one step further than (9): beyond
|
||||
<code>cargo check --workspace --all-targets</code> + <code>cargo test --workspace</code> +
|
||||
<code>scripts/smoke_test_index.sh</code> (all green, 870 kmers again), ran
|
||||
<code>obikmer merge</code> end to end on two freshly built indexes (exercises
|
||||
<code>clear_output_for_create</code>/<code>finalize_indexed</code> directly, the two methods
|
||||
<code>scripts/smoke_test_index.sh</code> itself never touches) — exit 0, <code>pack</code>
|
||||
stage completed. Test suite alone would not have caught a regression
|
||||
here: no existing test builds two real indexes and merges them through
|
||||
the CLI.</p>
|
||||
<p><code>KmerIndex</code> itself now carries only: identity/config accessors
|
||||
(<code>root_path</code>/<code>meta</code>/<code>kmer_size</code>/<code>n_bits</code>/<code>evidence_mode</code>/<code>genomes</code>/...),
|
||||
path resolution (<code>partition_dir</code>/<code>index_dir</code>/<code>layer_dir</code>/
|
||||
<code>partition_meta</code>/<code>n_layers</code>), and a few index-maintenance operations not
|
||||
yet sorted into either trait (<code>layer_unitigs_path</code>, <code>pack_matrices</code>,
|
||||
<code>upgrade_layer_meta</code> — see (8)'s "tested and discarded" list; still
|
||||
correctly inherent, not construction-only by the semantic criterion) —
|
||||
<code>create</code>/<code>open</code>/<code>exists</code> (identity, can't be trait methods needing <code>Self</code>
|
||||
before one exists) round that out.</p>
|
||||
<h2 id="11-done-2026-08-21-kmerindexindexmeta-made-fully-stateless-indexstate-moved-off-sentinel-files">(11) done (2026-08-21): <code>KmerIndex</code>/<code>IndexMeta</code> made fully stateless, <code>IndexState</code> moved off sentinel files</h2>
|
||||
<p>Triggered mid-discussion of <code>obikalgorithm::Algorithm</code> (still not started —
|
||||
see "Still not done" below): user asked why <code>PartitionRouter::new</code> still
|
||||
took <code>&mut KmerIndex</code> at all, and questioned whether <code>mark_scattered</code>
|
||||
belonged in the algorithm or in <code>cmd/index</code>. Investigation found <code>&mut</code>
|
||||
had become <em>newly</em> necessary since (9) — <code>mark_scattered</code> was mutating
|
||||
<code>self.meta.genomes</code> in memory so <code>Counter</code>'s later <code>write_spectrum</code> call
|
||||
(same <code>idx</code> instance) would see the derived label. User's resolution: the
|
||||
"disk is truth, stateless" principle already agreed for <code>KmerPartition</code>/
|
||||
<code>Layer</code> in (5) (still unimplemented for those two) should extend to
|
||||
<code>KmerIndex</code> itself — move <code>IndexState</code> (<code>Empty</code>/<code>Scattered</code>/<code>Counted</code>/
|
||||
<code>Indexed</code>) off the three sentinel files (<code>scatter.done</code>/<code>count.done</code>/
|
||||
<code>index.done</code>, detected by existence) and into a field of <code>index.meta</code>'s
|
||||
own JSON, so the <code>mark_*</code> calls become plain disk writes an algorithm can
|
||||
legitimately make on <code>&self</code> — no in-memory mutation left to protect.</p>
|
||||
<p><strong>Shape of <code>IndexMeta</code>, per the user's explicit spec</strong>: one JSON file per
|
||||
index (<code>index.meta</code>), one <code>IndexMeta</code> instance per index, held and
|
||||
returned as <code>Arc<IndexMeta></code> (not <code>&IndexMeta</code>) by <code>KmerIndex::meta()</code>.
|
||||
<code>config</code> (<code>kmer_size</code>/<code>minimizer_size</code>/<code>n_bits</code>/<code>with_counts</code>/<code>evidence</code>/
|
||||
<code>block_bits</code>) is fixed at construction, cached as a <code>pub</code> field (getter
|
||||
kept alongside, for symmetry) — "les champs constants restent des champs
|
||||
de la structure", read once, never re-read from disk. <code>genomes</code> and
|
||||
<code>state</code> are the opposite: no in-memory cache at all, every accessor
|
||||
(<code>genomes()</code>, <code>state()</code>) re-reads <code>index.meta</code> from disk, every mutator
|
||||
(<code>push_genome</code>/<code>rename_genome</code>/<code>set_genomes</code>/<code>set_state</code>/<code>mark_scattered</code>/
|
||||
<code>mark_counted</code>/<code>mark_indexed</code>) does a full read-modify-write of the same
|
||||
file. An internal <code>std::sync::RwLock<()></code> is held across each
|
||||
read-modify-write sequence (not just the write) so two callers sharing
|
||||
the same <code>Arc<IndexMeta></code> can't lose an update to each other — this is
|
||||
<em>not</em> a cross-process lock (that's <code>obisys::DirLock</code>, already held by
|
||||
<code>cmd/index</code> for the whole build); it only serialises access through one
|
||||
shared in-process instance.</p>
|
||||
<p><strong>Construction, chain-of-responsibility style, matching (5)'s pattern</strong>:
|
||||
<code>IndexMeta::create(&KmerIndex, config, genomes)</code> / <code>IndexMeta::open(&KmerIndex)</code>
|
||||
ask the index for its own root path rather than taking one directly. Since
|
||||
<code>KmerIndex::create</code> doesn't have a complete <code>KmerIndex</code> yet to hand in
|
||||
(it's what's being built), added lower-level <code>pub(crate)</code> path-based
|
||||
primitives <code>create_at(&Path, ...)</code> / <code>open_at(&Path)</code> that <code>KmerIndex::create</code>/
|
||||
<code>open</code> and <code>builder.rs</code>'s <code>create_skeleton</code> call directly, bypassing the
|
||||
convenience wrappers for that one bootstrap case.</p>
|
||||
<p><strong>The one deliberate exception</strong>: <code>select_in_place</code> and <code>reindex</code>
|
||||
genuinely rewrite <code>config</code> after an index already exists (output
|
||||
type/evidence mode changes in place) — contradicting "config never
|
||||
changes" for the general case. Resolved with a separate, explicitly
|
||||
rare-labelled <code>IndexMeta::rewrite_config(config, genomes)</code> (preserves
|
||||
<code>state</code>, overwrites everything else); callers refresh their own cached
|
||||
<code>Arc<IndexMeta></code> afterward (<code>self.meta = Arc::new(IndexMeta::open(self)?)</code>)
|
||||
since <code>IndexMeta</code> has no way to reach back into whichever <code>KmerIndex</code>
|
||||
holds it.</p>
|
||||
<p><strong>Consequence confirmed, not just hoped for</strong>: with <code>mark_scattered</code> no
|
||||
longer touching anything in memory, <code>PartitionRouter</code> genuinely never
|
||||
needs <code>&mut KmerIndex</code> — <code>PartitionRouter<'a> { index: &'a KmerIndex }</code>,
|
||||
<code>new(&'a KmerIndex)</code>. This is effectively the <code>PartitionRouter</code> half of
|
||||
(5)'s "order of remaining work" item done as a side effect; <code>KmerPartition</code>/
|
||||
<code>Layer</code> themselves are still unimplemented for (5).</p>
|
||||
<p><strong>Blast radius — much larger than (9)/(10), touched nearly every crate</strong>:
|
||||
every <code>.meta().genomes</code>/<code>.meta.genomes</code> field access became a fallible
|
||||
<code>.genomes()?</code> method call (<code>genomes</code> reads <code>io::Result<Vec<GenomeInfo>></code>
|
||||
now, not a field), and <code>.meta_mut()</code> was removed outright (no more direct
|
||||
field mutation from outside <code>IndexMeta</code>). Fixed across:
|
||||
- <code>obikindex</code> internals: <code>meta.rs</code>/<code>state.rs</code>/<code>kmer_index.rs</code>/<code>builder.rs</code>
|
||||
(full rewrites), <code>reindex.rs</code>/<code>select.rs</code> (switched to <code>rewrite_config</code>),
|
||||
<code>merge.rs</code> (heaviest single file — genome counts precomputed once per
|
||||
source into a <code>Vec<Vec<GenomeInfo>></code> up front rather than re-reading
|
||||
<code>index.meta</code> from disk repeatedly through the function, sentinel write
|
||||
replaced with <code>dst2.meta.mark_indexed()</code>), <code>stats.rs</code>, <code>distance.rs</code>,
|
||||
<code>dump.rs</code>, <code>predicate.rs</code> (its <code>IndexMeta</code>-inherent <code>matching_genome_indices</code>/
|
||||
<code>build_group_filter</code> now read genomes fresh internally), <code>mod.rs</code>/<code>lib.rs</code>
|
||||
(sentinel constant re-exports removed — <code>IndexState</code> no longer has
|
||||
<code>SENTINEL_*</code>/<code>detect()</code> at all).
|
||||
- <code>obikindexer::extensions::PrivateBuilder</code>: <code>mark_scattered</code> signature
|
||||
dropped <code>&mut self</code> → <code>&self</code>; the four <code>mark_*</code>/<code>write_spectrum</code> bodies
|
||||
became one-line delegations to <code>self.meta().mark_*()</code>.
|
||||
- <code>obikphylo::siblings</code>: <code>alignment.rs</code>/<code>cardinality.rs</code>/<code>distance.rs</code>/
|
||||
<code>entropy.rs</code>/<code>sankoff_bundle.rs</code>/<code>stats.rs</code>/<code>tests.rs</code> — all had
|
||||
<code>self.meta().genomes.len()</code>-shaped reads, mechanically fixed to
|
||||
<code>.genomes().map_err(OKIError::Io)?.len()</code> (tests: <code>.unwrap()</code>).
|
||||
- <code>obikmer::cmd::*</code>: <code>annotate</code> (rewrote its rename path to load genomes
|
||||
once, mutate the in-memory <code>Vec</code>, then <code>idx.meta().set_genomes(...)</code>
|
||||
instead of <code>meta_mut()</code>), <code>filter</code>/<code>pack</code>/<code>dump</code>/<code>unitig</code>/<code>merge</code>/<code>select</code>/
|
||||
<code>phylo</code> (fetch-once-then-use pattern for genome counts/labels),
|
||||
<code>utils/maintenance.rs</code> (<code>run_rename</code> now calls the pre-existing
|
||||
<code>IndexMeta::rename_genome</code>, dropping its own hand-rolled field mutation
|
||||
entirely), <code>index/mod.rs</code> (three <code>idx.state() < IndexState::X</code>
|
||||
resumability checks needed a fallible read — factored into a small
|
||||
<code>current_state(&KmerIndex) -> IndexState</code> helper rather than repeating
|
||||
the same <code>unwrap_or_else</code> three times), <code>query/*</code> (<code>emit_batch</code>'s
|
||||
signature changed from <code>&IndexMeta</code> to <code>&[GenomeInfo]</code>, and <code>genomes</code> is
|
||||
now fetched once in <code>run()</code> and threaded down through <code>process_chunk</code>
|
||||
as <code>Arc<Vec<GenomeInfo>></code> rather than re-reading <code>index.meta</code> from disk
|
||||
on every chunk — a deliberate deviation from the "always re-read"
|
||||
default, justified because this is a genuine per-chunk hot path, unlike
|
||||
every other call site touched in this pass).
|
||||
- One <code>&IndexMeta</code>-vs-<code>Arc<IndexMeta></code> argument-type mismatch pattern
|
||||
recurred at several CLI call sites (<code>build_filters</code>/<code>build_specs</code>/
|
||||
<code>emit_batch</code>'s original signature) — resolved via <code>Arc</code>'s deref
|
||||
coercion (<code>&idx.meta()</code> coerces to <code>&IndexMeta</code>) rather than changing
|
||||
every downstream signature to accept <code>Arc<IndexMeta></code>.</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and
|
||||
<code>cargo test --workspace</code> both green (0 failures) after the full
|
||||
propagation, <code>scripts/smoke_test_index.sh</code> green (870 kmers, same as every
|
||||
prior round), plus a manual CLI run of <code>index</code> (×2) → <code>merge</code> → <code>select</code>
|
||||
→ <code>reindex</code> → <code>utils --new-label</code> (rename) → <code>utils --stats</code>, all exit 0,
|
||||
confirming the four most-affected commands (the ones (10)'s verification
|
||||
already flagged as under-covered by the automated test suite) still work
|
||||
end to end against the new <code>Arc<IndexMeta></code>/on-disk-<code>IndexState</code> shape.</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign itself
|
||||
(only the <code>PartitionRouter</code>-<code>&mut</code>-removal piece landed, as a side
|
||||
effect); the <code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly
|
||||
deferred); the future cache-manager crate. (8)'s <code>obikalgorithm::
|
||||
Algorithm</code> trait, resumed and closed in (12) below.</p>
|
||||
<h2 id="12-done-2026-08-21-obikalgorithmalgorithm-the-shared-trait-resumed-and-closed-in-one-session">(12) done (2026-08-21): <code>obikalgorithm::Algorithm</code> — the shared trait, resumed and closed in one session</h2>
|
||||
<p>Resumed (8)'s point 2 through a point-by-point discussion of what's
|
||||
actually common across the four pipeline algorithms, now that (11) made
|
||||
<code>KmerIndex</code> itself immutable everywhere. Four sub-points, each closed
|
||||
before moving to the next:</p>
|
||||
<p><strong>1. Receiver (<code>&self</code> vs <code>&mut self</code>)</strong> — investigated whether (11)'s
|
||||
removal of <code>&mut KmerIndex</code> also removed the need for <code>PartitionRouter::
|
||||
run</code> to take <code>&mut self</code>. It didn't: <code>PartitionRouter</code> holds real
|
||||
per-run state of its own (<code>writers: Vec<Option<SKFileWriter>></code>, open file
|
||||
handles, purely in-process RAM — confirmed by checking where <code>writers</code> is
|
||||
stored, nothing to do with <code>KmerIndex</code>/disk truth), unrelated to the
|
||||
index. First proposal (wrap <code>writers</code> in <code>RefCell</code> so all four could
|
||||
share a uniform <code>&self</code>) was retracted on pushback: manufacturing
|
||||
interior mutability with runtime borrow checks to satisfy a cosmetic
|
||||
uniformity that Rust doesn't even require is over-engineering — a trait
|
||||
method's receiver must match exactly across implementors, but nothing
|
||||
stops that shared receiver from being <code>&mut self</code> with three of the four
|
||||
implementations simply not using the mutability. Settled: trait declares
|
||||
<code>&mut self</code>; <code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> (previously <code>&self</code>)
|
||||
now also take <code>&mut self</code>, unused.</p>
|
||||
<p><strong>2. <code>path_source</code> as a <code>PartitionRouter</code> setter, not a <code>run()</code> param</strong> —
|
||||
added a <code>files: Option<Box<dyn Iterator<Item = PathBuf> + Send>></code> field +
|
||||
<code>.files(impl Iterator<Item = PathBuf> + Send + 'static) -> Self</code> setter
|
||||
(boxed rather than a generic type parameter on <code>PartitionRouter<'a></code>:
|
||||
negligible cost — one <code>PathBuf</code> per input <em>file</em>, not per k-mer — for a
|
||||
much more usable type when passing the builder around). <code>run</code> now does
|
||||
<code>self.files.take().ok_or_else(...)</code>, erroring if <code>.files(...)</code> was never
|
||||
called, instead of taking <code>path_source</code> as a parameter.</p>
|
||||
<p><strong>3. Unifying the three progress-callback bound shapes</strong> — reopened, then
|
||||
resolved differently than (8) originally framed it. First proposal
|
||||
(force everything to <code>FnMut(Progress) + Send</code>) was rejected on the same
|
||||
principle as point 1: <code>Dereplicator</code>/<code>Counter</code>'s <code>Fn(Progress) + Sync</code>
|
||||
isn't arbitrary — their callback is invoked concurrently from multiple
|
||||
rayon worker threads, and <code>FnMut</code> requires exclusive access, so forcing
|
||||
it would mean wrapping the callback in a <code>Mutex</code> for zero benefit at the
|
||||
one real call site (<code>pb.inc(1)</code>, already thread-safe). The actual
|
||||
resolution: move <code>on_progress</code> off <code>run()</code>'s signature entirely, onto a
|
||||
per-algorithm <code>.on_progress(...)</code> setter — same treatment as point 2's
|
||||
<code>path_source</code> — so each algorithm keeps its own bound (<code>PartitionRouter</code>:
|
||||
<code>FnMut(Progress) + 'a</code>, sequential, no <code>Send</code> needed; <code>LayerBuilder</code>:
|
||||
<code>FnMut(Progress) + Send + 'a</code>, crosses into <code>PartitionRunner</code>'s
|
||||
<code>thread::scope</code>-spawned controller thread once; <code>Dereplicator</code>/<code>Counter</code>:
|
||||
<code>Fn(Progress) + Sync + 'a</code>, invoked concurrently from rayon workers).
|
||||
This <em>also</em> dissolves the original problem <code>run()</code> had: once the
|
||||
callback isn't part of <code>run</code>'s signature at all, there's nothing left to
|
||||
unify there, and point 4 (below) becomes trivial.</p>
|
||||
<p><strong>4. <code>Output</code> as an associated type, <code>Error</code> fixed</strong> — trivial once (3)
|
||||
moved the callback out: <code>Error</code> was already uniform (all four return
|
||||
<code>obiskio::SKResult<T></code> = <code>Result<T, SKError></code>), only <code>Output</code> varied
|
||||
(<code>()</code>/<code>()</code>/<code>KmerSpectrum</code>/<code>usize</code>). First cut reused <code>obiskio::SKResult</code>
|
||||
directly as the trait's return type — <strong>caught and corrected the same
|
||||
session</strong>: <code>SKError</code> enumerates I/O-specific cases (<code>BadMagic</code>/
|
||||
<code>Truncated</code>/<code>Compression</code>/...), meaningless at the level of a generic
|
||||
"algorithm" abstraction, and borrowing it made <code>obikalgorithm</code> — meant to
|
||||
be minimal and neutral — depend on a low-level I/O crate purely to reuse
|
||||
its error type. Textbook instance of the "petits pois" failure mode
|
||||
(patch around a convenient existing type instead of asking what this
|
||||
crate should actually own). Fixed to a genuinely generic, boxed error
|
||||
type owned by <code>obikalgorithm</code> itself:</p>
|
||||
<div class="highlight"><pre><span></span><code><span class="c1">// obikalgorithm — no dependency on obiskio or any other crate</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Error</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nb">Box</span><span class="o"><</span><span class="k">dyn</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">error</span><span class="p">::</span><span class="n">Error</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Send</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">Sync</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="p">::</span><span class="n">result</span><span class="p">::</span><span class="nb">Result</span><span class="o"><</span><span class="n">T</span><span class="p">,</span><span class="w"> </span><span class="n">Error</span><span class="o">></span><span class="p">;</span>
|
||||
|
||||
<span class="k">pub</span><span class="w"> </span><span class="k">trait</span><span class="w"> </span><span class="n">Algorithm</span><span class="w"> </span><span class="p">{</span>
|
||||
<span class="w"> </span><span class="k">type</span><span class="w"> </span><span class="nc">Output</span><span class="p">;</span>
|
||||
<span class="w"> </span><span class="k">fn</span><span class="w"> </span><span class="nf">run</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="bp">self</span><span class="p">)</span><span class="w"> </span><span class="p">-></span><span class="w"> </span><span class="nb">Result</span><span class="o"><</span><span class="bp">Self</span><span class="p">::</span><span class="n">Output</span><span class="o">></span><span class="p">;</span>
|
||||
<span class="p">}</span>
|
||||
</code></pre></div>
|
||||
<p>Any concrete error (<code>SKError</code>, <code>std::io::Error</code>, ...) converts
|
||||
automatically via <code>?</code>, through <code>std</code>'s own blanket <code>impl<E: Error + Send
|
||||
+ Sync> From<E> for Box<dyn Error + Send + Sync></code> — no custom <code>From</code> impl
|
||||
needed, no dependency on the crate that defines the concrete error type.
|
||||
The four algorithms' <code>run</code> bodies needed no change beyond the signature's
|
||||
return type (every existing <code>?</code> on an <code>SKError</code>-returning subcall keeps
|
||||
compiling, converting through the same blanket impl at the boundary).</p>
|
||||
<p><code>PartitionRouter</code>/<code>Dereplicator</code>/<code>Counter</code>/<code>LayerBuilder</code> each <code>impl
|
||||
Algorithm for X<'_> { type Output = ...; fn run(&mut self) -> obikalgorithm::Result<...> { ... } }</code>
|
||||
— the old inherent <code>run</code> methods were removed outright (not kept as
|
||||
duplicates), so callers now <code>use obikalgorithm::Algorithm;</code> to call
|
||||
<code>.run()</code>. Every field-lifetime-bound boxed callback (<code>Box<dyn
|
||||
FnMut(Progress) + 'a></code> etc.) is tied to the algorithm's own <code>'a</code> (the
|
||||
<code>&'a KmerIndex</code> lifetime already on the struct), not <code>'static</code> — avoids
|
||||
forcing callers' progress closures to <code>move</code>-capture (and therefore clone
|
||||
or <code>Arc</code>-wrap) local state like <code>TracedBar</code>/EMA-rate accumulators that
|
||||
they'd otherwise want to keep using by reference after <code>run()</code> returns.</p>
|
||||
<p><strong>Why a new crate, not a submodule of <code>obikindexer</code></strong>: <code>obikmer::cmd::
|
||||
index::mod</code> and <code>obikphylo</code>'s own test helpers both need to call <code>.run()</code>
|
||||
on these algorithms, so the trait has to be reachable from outside
|
||||
<code>obikindexer</code> — putting it in <code>obikindexer</code> itself would work file-wise
|
||||
but conflates "the trait every algorithm implements" with "one crate's
|
||||
particular four implementations of it", the same reasoning that already
|
||||
separated <code>obikindex</code> (data model) from <code>obikindexer</code> (algorithms
|
||||
operating on it). <code>obikalgorithm</code> has <strong>no dependencies at all</strong> (see
|
||||
above); <code>obikindexer</code>, <code>obikmer</code>, and <code>obikphylo</code> (dev-dependency, for
|
||||
its test helper) all depend on it.</p>
|
||||
<p><strong>Blast radius</strong>: <code>obikindexer</code>'s four algorithm modules (struct field +
|
||||
setter + trait impl each); <code>obikmer::cmd::index::mod</code> (three call sites:
|
||||
<code>.on_progress(cb)</code> before <code>.run()</code>, unqualified now that the trait is in
|
||||
scope); <code>obikindexer::algorithms::partitionner::tests</code> and <code>obikphylo::
|
||||
siblings::tests</code> (both had direct <code>.run(None::<fn(Progress)>)</code>-shaped
|
||||
calls needing the same treatment). New <code>obikalgorithm</code> crate registered
|
||||
in the workspace <code>Cargo.toml</code>, depended on by <code>obikindexer</code>/<code>obikmer</code>
|
||||
(regular) and <code>obikphylo</code> (dev).</p>
|
||||
<p><strong>Verification</strong>: <code>cargo check --workspace --all-targets</code> and <code>cargo test
|
||||
--workspace</code> both green (0 failures), <code>scripts/smoke_test_index.sh</code> green
|
||||
(870 kmers, same as every prior round) — this round didn't repeat the
|
||||
manual <code>merge</code>/<code>select</code>/<code>reindex</code> CLI exercise from (11), since nothing
|
||||
in this pass touched those commands' code paths (only the four pipeline
|
||||
algorithms and <code>cmd::index</code>, already covered by the smoke test). Reverified
|
||||
after the <code>obiskio</code>-dependency fix above (same three checks, still green,
|
||||
<code>obikalgorithm/Cargo.toml</code> now has zero <code>[dependencies]</code>).</p>
|
||||
<p>Still not done: (5)'s <code>KmerPartition</code>/<code>Layer</code> self-naming redesign; the
|
||||
<code>distance.rs</code> → <code>obikphylo</code> relocation ((9), explicitly deferred); the
|
||||
future cache-manager crate (mentioned in (8) as a later, mirrored
|
||||
extension-trait exercise, not started).</p>
|
||||
<h2 id="the-problem">The problem</h2>
|
||||
<p>Reading a layer's data (MPHF + matrix) is not free: <code>MphfLayer::open</code> mmaps
|
||||
<code>mphf.bin</code> plus (<code>evidence.bin</code>/<code>fingerprint.bin</code> + <code>unitigs.bin</code>), and the
|
||||
|
||||
Reference in New Issue
Block a user