Implement SNP distance models with gamma correction and PHYLIP output
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
This commit is contained in:
Eric Coissac
@@ -909,6 +909,56 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -2195,6 +2245,56 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<nav class="md-nav" aria-label="--distance unification: SNP corrections as first-class metrics (implemented, 2026-08-28)">
|
||||
<ul class="md-nav__list">
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#snp-distance-catalog" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
snp-* distance catalog
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#exact-formulas-implemented-2026-08-28" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#output-format-phylip-relaxed-by-default-for-the-distance-matrix" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
@@ -4261,6 +4361,249 @@ among the survivors) — a single extra pass is sufficient.</p>
|
||||
<code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
|
||||
from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
|
||||
workspace <code>cargo test</code> green.</p>
|
||||
<h2 id="-distance-unification-snp-corrections-as-first-class-metrics-implemented-2026-08-28"><code>--distance</code> unification: SNP corrections as first-class metrics (implemented, 2026-08-28)</h2>
|
||||
<p><strong>Implemented.</strong> <code>--metric</code> (renamed <code>--distance</code> — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
<code>--metric</code> was a misnomer) gains a family of <code>snp-*</code> values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
machinery (<code>PairwiseTally</code>, <code>obikphylo::siblings::algorithms::pairwise</code>)
|
||||
instead of <code>cache.distance(...)</code>'s existing per-layer traversal — a different
|
||||
code path behind the same CLI surface, not just another branch of one
|
||||
formula function.</p>
|
||||
<p><strong>Why unify at the CLI level despite the implementation split</strong>: phylogenetically
|
||||
a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
|
||||
to how the matrix was produced, so exposing it as a special-cased subcommand
|
||||
instead of a <code>--distance</code> value would misrepresent its role. The
|
||||
implementation divergence (sibling-annex-based vs. plain index scan) is real
|
||||
but belongs at the routing layer, invisible to the CLI's own vocabulary.</p>
|
||||
<p><strong><code>--subsample</code> becomes optional for <code>snp-*</code> distances</strong> (it stays mandatory
|
||||
for <code>--sankoff</code>/<code>--pseudo-alignment</code>, unrelated commands): absent means
|
||||
exhaustive, achieved for free by reusing <code>sample_index</code>'s existing
|
||||
proportional-per-layer-quota mechanism with <code>n</code> set to the index-wide total
|
||||
non-monomorphic-minorant count (already available from the sibling-annex
|
||||
stats) — every layer's quota then equals its own full count, giving Bernoulli
|
||||
<code>p = 1</code> everywhere, i.e. every eligible family is drawn. No second,
|
||||
exhaustive-only driver needed. Present means sampled, exactly as <code>--sankoff</code>
|
||||
already behaves.</p>
|
||||
<p><strong>One shared tally, many derived formulas.</strong> <code>PairwiseTally</code>'s <code>subst[4][4]</code>
|
||||
per-pair substitution counts (plus marginal base frequencies derived from it)
|
||||
are the sufficient statistic for every closed-form correction below — each
|
||||
is a small pure function <code>PairwiseTally -> Array2<f64></code>, at the same level as
|
||||
the already-implemented <code>raw_snp_distance</code>/<code>base_pair_tally</code>/
|
||||
<code>cardinality_tally</code>. No new full scan per formula, whether the tally itself
|
||||
was built exhaustively or from a subsample.</p>
|
||||
<p><strong><code>--raw-snp-counts</code> stays a separate, unrelated flag</strong> — same underlying
|
||||
tally, but a diagnostic (<code>n_snp</code>/<code>n_shared</code>/<code>n_eligible</code> per genome pair, one
|
||||
row per pair) rather than a distance value, and its long-table shape doesn't
|
||||
fold into a single N×N matrix the way a distance does. No change to its
|
||||
existing CSV format.</p>
|
||||
<h3 id="snp-distance-catalog"><code>snp-*</code> distance catalog</h3>
|
||||
<p>All closed-form (method-of-moments / direct formula), none requiring
|
||||
per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
|
||||
<em>tree</em>-ML usage but not its <em>pairwise</em> estimator, which is closed-form like
|
||||
F84/TN93 and is included below. <code>snp-</code> prefix on every CLI value.</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>value</th>
|
||||
<th>corrects for</th>
|
||||
<th>inputs beyond raw counts</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><code>snp-raw</code></td>
|
||||
<td>nothing (uncorrected p-distance)</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-jc</code> (Jukes-Cantor, JC69)</td>
|
||||
<td>multiple substitutions per site</td>
|
||||
<td>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k2p</code> (Kimura 2-parameter, K80)</td>
|
||||
<td>+ transition/transversion rate bias</td>
|
||||
<td>ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-k81</code> (Kimura 3-parameter, K3ST)</td>
|
||||
<td>+ splits transversions into 2 categories</td>
|
||||
<td>ts/tv split, by category</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f81</code> (Felsenstein 81)</td>
|
||||
<td>+ unequal base frequencies (no ts/tv split)</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tajima-nei</code> (Tajima-Nei 1984)</td>
|
||||
<td>same goal as F81 (equal-input model), different formula, better small-sample behavior</td>
|
||||
<td>empirical base freqs</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-t92</code> (Tamura 3-parameter)</td>
|
||||
<td>K2P + GC-content bias</td>
|
||||
<td>ts/tv split, GC content</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-f84</code> (Felsenstein 84)</td>
|
||||
<td>full empirical base freqs + single ts/tv rate</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-hky85</code> (Hasegawa-Kishino-Yano, pairwise estimator)</td>
|
||||
<td>same inputs as F84, different formula</td>
|
||||
<td>empirical base freqs, ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tn93</code> (Tamura-Nei)</td>
|
||||
<td>full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate</td>
|
||||
<td>empirical base freqs, purine-ts/pyrimidine-ts/tv split</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-logdet</code> (LogDet / paralinear)</td>
|
||||
<td>no shared-model or stationarity assumption at all — general divergence-matrix determinant</td>
|
||||
<td>full empirical 4×4 divergence matrix (already <code>subst[4][4]</code>)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>snp-tv</code> (transversions-only p-distance)</td>
|
||||
<td>diagnostic/deep-divergence variant — drops transitions entirely (they saturate first)</td>
|
||||
<td>tv-only counts</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p><strong><code>+Γ</code> rate-heterogeneity modifier, applicable to <code>snp-jc</code>, <code>snp-k2p</code>,
|
||||
<code>snp-k81</code>, <code>snp-t92</code>, <code>snp-f84</code>, <code>snp-hky85</code>, <code>snp-tn93</code></strong> (not <code>snp-raw</code>,
|
||||
nothing to correct; not <code>snp-logdet</code>, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter <code>α</code> supplied
|
||||
by the user (<code>--gamma-shape <alpha></code>), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.</p>
|
||||
<p><strong>Implemented now: <code>snp-raw</code>, <code>snp-jc</code>, <code>snp-k2p</code>, <code>snp-k81</code>, <code>snp-f81</code>,
|
||||
<code>snp-t92</code>, <code>snp-tn93</code>, <code>snp-tv</code>, all with <code>+Γ</code> except <code>raw</code>/<code>tv</code></strong> — see
|
||||
"Exact formulas" below. <code>snp-tajima-nei</code>, <code>snp-f84</code>, <code>snp-hky85</code>,
|
||||
<code>snp-logdet</code> are catalogued above but <strong>not implemented</strong>: <code>snp-logdet</code>
|
||||
needs the true <em>directional</em> per-pair base co-occurrence matrix
|
||||
(<code>PairwiseTally</code> only keeps the symmetrised substitution counts
|
||||
<code>BasePairTally</code> itself wants — see <code>snp_distance.rs</code>'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), <code>snp-tajima-nei</code> needs each genome's <em>own</em> base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
<code>snp-f84</code>/<code>snp-hky85</code> had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against <a href="https://github.com/emmanuelparadis/ape">ape</a>'s own
|
||||
<code>src/dist_dna.c</code>, not re-derived from memory). Adding any of these later is
|
||||
a new function in <code>obikphylo::siblings::algorithms::snp_distance</code>, plus for
|
||||
<code>snp-logdet</code>/<code>snp-tajima-nei</code> a new field on <code>PairStats</code>/a per-genome
|
||||
accumulator — not an architecture change.</p>
|
||||
<h3 id="exact-formulas-implemented-2026-08-28">Exact formulas (implemented, 2026-08-28)</h3>
|
||||
<p>Sufficient statistic, per genome pair <code>(i, j)</code>, from
|
||||
<code>PairwiseTally::categories</code>/<code>PairwiseTally::base_freq</code> (base order always
|
||||
<code>0=A, 1=C, 2=G, 3=T</code>, matching <code>FamilyMask</code>/<code>STATE_SYMBOL</code>):</p>
|
||||
<ul>
|
||||
<li><span class="arithmatex">\(n_{ts1}\)</span>: A↔G substitutions (purine transitions), <span class="arithmatex">\(n_{ts2}\)</span>: C↔T
|
||||
(pyrimidine transitions)</li>
|
||||
<li><span class="arithmatex">\(n_{tv1}\)</span>: A↔C and G↔T substitutions, <span class="arithmatex">\(n_{tv2}\)</span>: A↔T and C↔G
|
||||
(Kimura's two transversion categories)</li>
|
||||
<li><span class="arithmatex">\(n_{shared}\)</span>: loci where both genomes agree</li>
|
||||
<li><span class="arithmatex">\(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\)</span> (total eligible
|
||||
loci for the pair)</li>
|
||||
<li><span class="arithmatex">\(\pi_A, \pi_C, \pi_G, \pi_T\)</span>: pair-pooled base frequencies,
|
||||
<span class="arithmatex">\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)</span>
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)</li>
|
||||
</ul>
|
||||
<p>Derived proportions used below:</p>
|
||||
<div class="arithmatex">\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]</div>
|
||||
<p>Every formula below was checked term-by-term against <code>ape</code>'s own
|
||||
<code>src/dist_dna.c</code> (not re-derived from memory) before being ported to
|
||||
<code>obikphylo::siblings::algorithms::snp_distance</code>.</p>
|
||||
<p><strong><code>snp-raw</code></strong> — uncorrected p-distance:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{raw} = p
|
||||
\]</div>
|
||||
<p><strong><code>snp-tv</code></strong> — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, <em>is</em> the correction):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{tv} = Q
|
||||
\]</div>
|
||||
<p><strong><code>snp-jc</code></strong> (Jukes-Cantor, JC69):</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-k2p</code></strong> (Kimura 2-parameter, K80), with <span class="arithmatex">\(a_1 = 1-2P-Q\)</span>, <span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-k81</code></strong> (Kimura 3-parameter, K3ST), with <span class="arithmatex">\(a_1 = 1-2P-2Q_1\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2P-2Q_2\)</span>, <span class="arithmatex">\(a_3 = 1-2Q_1-2Q_2\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-f81</code></strong> (Felsenstein 81), with <span class="arithmatex">\(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]</div>
|
||||
<p><strong><code>snp-t92</code></strong> (Tamura 3-parameter), with GC content
|
||||
<span class="arithmatex">\(g = \pi_C+\pi_G\)</span>, <span class="arithmatex">\(w = 2g(1-g)\)</span>, <span class="arithmatex">\(a_1 = 1 - \dfrac{P}{w} - Q\)</span>,
|
||||
<span class="arithmatex">\(a_2 = 1-2Q\)</span>:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]</div>
|
||||
<p><strong><code>snp-tn93</code></strong> (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
<span class="arithmatex">\(g_R = \pi_A+\pi_G\)</span>, <span class="arithmatex">\(g_Y = \pi_C+\pi_T\)</span>, and</p>
|
||||
<div class="arithmatex">\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]</div>
|
||||
<div class="arithmatex">\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]</div>
|
||||
<p><strong><code>+Γ</code> gamma correction</strong> (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of <span class="arithmatex">\(-\ln(x)\)</span> terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
<span class="arithmatex">\(\alpha\left(x^{-1/\alpha} - 1\right)\)</span> instead — the standard mechanical
|
||||
substitution (as <span class="arithmatex">\(\alpha \to \infty\)</span>, this expression → <span class="arithmatex">\(-\ln(x)\)</span>,
|
||||
recovering the uncorrected formula exactly). E.g. for JC:</p>
|
||||
<div class="arithmatex">\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]</div>
|
||||
<p>Verified term-by-term against <code>ape</code>'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to <code>snp-k2p</code>'s own <span class="arithmatex">\(a_1\)</span>/<span class="arithmatex">\(a_2\)</span> terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one <code>corrected_log</code> helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
<code>ape</code>-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.</p>
|
||||
<h3 id="output-format-phylip-relaxed-by-default-for-the-distance-matrix">Output format: PHYLIP-relaxed by default for the distance matrix</h3>
|
||||
<p><strong>Implemented.</strong> The primary distance-matrix output
|
||||
(<code>_dist.csv</code> today) gains multiple formats: <strong>PHYLIP-relaxed becomes the
|
||||
default</strong> (widely read by external NJ tools — PHYLIP <code>neighbor</code>, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
label truncation, since genome labels here routinely exceed it), a <code>--csv</code>
|
||||
flag opts back into the current CSV format, PHYLIP-strict is a possible
|
||||
future addition (not now). This changes the <em>default</em> output of every
|
||||
existing <code>--distance</code> value (jaccard, hamming, bray-curtis, ...), not just
|
||||
the new <code>snp-*</code> ones — accepted explicitly (pre-release, single developer
|
||||
user, no external consumers to break). Scoped to the distance matrix only:
|
||||
<code>--shared-kmers</code> and <code>--raw-snp-counts</code> are counts, not distances, and keep
|
||||
their existing CSV-only format.</p>
|
||||
<h2 id="references">References</h2>
|
||||
<p>The Mash mutation-rate model this discussion contrasts with:
|
||||
(Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>
|
||||
|
||||
Reference in New Issue
Block a user