Implement SNP distance models with gamma correction and PHYLIP output
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
This commit is contained in:
Eric Coissac
@@ -2182,9 +2182,9 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
|
||||
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
|
||||
workspace `cargo test` green.
|
||||
|
||||
## `--distance` unification: SNP corrections as first-class metrics (discussion, 2026-08-28)
|
||||
## `--distance` unification: SNP corrections as first-class metrics (implemented, 2026-08-28)
|
||||
|
||||
**Decided, not yet implemented.** `--metric` (renamed `--distance` — several of
|
||||
**Implemented.** `--metric` (renamed `--distance` — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
`--metric` was a misnomer) gains a family of `snp-*` values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
@@ -2253,9 +2253,147 @@ formula as the base correction, weighted by a shape parameter `α` supplied
|
||||
by the user (`--gamma-shape <alpha>`), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.
|
||||
|
||||
**Implemented now: `snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`,
|
||||
`snp-t92`, `snp-tn93`, `snp-tv`, all with `+Γ` except `raw`/`tv`** — see
|
||||
"Exact formulas" below. `snp-tajima-nei`, `snp-f84`, `snp-hky85`,
|
||||
`snp-logdet` are catalogued above but **not implemented**: `snp-logdet`
|
||||
needs the true *directional* per-pair base co-occurrence matrix
|
||||
(`PairwiseTally` only keeps the symmetrised substitution counts
|
||||
`BasePairTally` itself wants — see `snp_distance.rs`'s own module docs for
|
||||
why that loses exactly the compositional-asymmetry information LogDet
|
||||
exists to detect), `snp-tajima-nei` needs each genome's *own* base
|
||||
composition (not the pair-pooled estimate the formulas below use), and
|
||||
`snp-f84`/`snp-hky85` had no formula independently verified against a
|
||||
primary source at implementation time (unlike every formula below, checked
|
||||
line-by-line against [ape](https://github.com/emmanuelparadis/ape)'s own
|
||||
`src/dist_dna.c`, not re-derived from memory). Adding any of these later is
|
||||
a new function in `obikphylo::siblings::algorithms::snp_distance`, plus for
|
||||
`snp-logdet`/`snp-tajima-nei` a new field on `PairStats`/a per-genome
|
||||
accumulator — not an architecture change.
|
||||
|
||||
### Exact formulas (implemented, 2026-08-28)
|
||||
|
||||
Sufficient statistic, per genome pair `(i, j)`, from
|
||||
`PairwiseTally::categories`/`PairwiseTally::base_freq` (base order always
|
||||
`0=A, 1=C, 2=G, 3=T`, matching `FamilyMask`/`STATE_SYMBOL`):
|
||||
|
||||
- \(n_{ts1}\): A↔G substitutions (purine transitions), \(n_{ts2}\): C↔T
|
||||
(pyrimidine transitions)
|
||||
- \(n_{tv1}\): A↔C and G↔T substitutions, \(n_{tv2}\): A↔T and C↔G
|
||||
(Kimura's two transversion categories)
|
||||
- \(n_{shared}\): loci where both genomes agree
|
||||
- \(L = n_{ts1} + n_{ts2} + n_{tv1} + n_{tv2} + n_{shared}\) (total eligible
|
||||
loci for the pair)
|
||||
- \(\pi_A, \pi_C, \pi_G, \pi_T\): pair-pooled base frequencies,
|
||||
\(\pi_a = \dfrac{2 \cdot (\text{agreements on } a) + \sum_b n_{a \leftrightarrow b}}{2L}\)
|
||||
(both genomes' calls at this pair's eligible loci, pooled — Nei & Kumar's
|
||||
standard pairwise estimator, not a whole-index average)
|
||||
|
||||
Derived proportions used below:
|
||||
|
||||
\[
|
||||
p = \frac{n_{ts1}+n_{ts2}+n_{tv1}+n_{tv2}}{L}, \quad
|
||||
P = \frac{n_{ts1}+n_{ts2}}{L}, \quad
|
||||
Q = \frac{n_{tv1}+n_{tv2}}{L}, \quad
|
||||
Q_1 = \frac{n_{tv1}}{L}, \quad
|
||||
Q_2 = \frac{n_{tv2}}{L}, \quad
|
||||
P_1 = \frac{n_{ts1}}{L}, \quad
|
||||
P_2 = \frac{n_{ts2}}{L}
|
||||
\]
|
||||
|
||||
Every formula below was checked term-by-term against `ape`'s own
|
||||
`src/dist_dna.c` (not re-derived from memory) before being ported to
|
||||
`obikphylo::siblings::algorithms::snp_distance`.
|
||||
|
||||
**`snp-raw`** — uncorrected p-distance:
|
||||
|
||||
\[
|
||||
d_{raw} = p
|
||||
\]
|
||||
|
||||
**`snp-tv`** — transversions-only p-distance (deliberately uncorrected —
|
||||
dropping transitions, which saturate first, *is* the correction):
|
||||
|
||||
\[
|
||||
d_{tv} = Q
|
||||
\]
|
||||
|
||||
**`snp-jc`** (Jukes-Cantor, JC69):
|
||||
|
||||
\[
|
||||
d_{JC} = -\frac{3}{4} \ln\!\left(1 - \frac{4p}{3}\right)
|
||||
\]
|
||||
|
||||
**`snp-k2p`** (Kimura 2-parameter, K80), with \(a_1 = 1-2P-Q\), \(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{K2P} = -\frac{1}{2}\ln a_1 - \frac{1}{4}\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-k81`** (Kimura 3-parameter, K3ST), with \(a_1 = 1-2P-2Q_1\),
|
||||
\(a_2 = 1-2P-2Q_2\), \(a_3 = 1-2Q_1-2Q_2\):
|
||||
|
||||
\[
|
||||
d_{K81} = -\frac{1}{4}\left(\ln a_1 + \ln a_2 + \ln a_3\right)
|
||||
\]
|
||||
|
||||
**`snp-f81`** (Felsenstein 81), with \(E = 1 - \left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right)\):
|
||||
|
||||
\[
|
||||
d_{F81} = -E \ln\!\left(1 - \frac{p}{E}\right)
|
||||
\]
|
||||
|
||||
**`snp-t92`** (Tamura 3-parameter), with GC content
|
||||
\(g = \pi_C+\pi_G\), \(w = 2g(1-g)\), \(a_1 = 1 - \dfrac{P}{w} - Q\),
|
||||
\(a_2 = 1-2Q\):
|
||||
|
||||
\[
|
||||
d_{T92} = -w \ln a_1 - \frac{1}{2}(1-w)\ln a_2
|
||||
\]
|
||||
|
||||
**`snp-tn93`** (Tamura-Nei), with purine/pyrimidine pooled frequencies
|
||||
\(g_R = \pi_A+\pi_G\), \(g_Y = \pi_C+\pi_T\), and
|
||||
|
||||
\[
|
||||
k_1 = \frac{2\pi_A\pi_G}{g_R}, \quad
|
||||
k_2 = \frac{2\pi_C\pi_T}{g_Y}, \quad
|
||||
k_3 = 2\left(g_R g_Y - \frac{\pi_A\pi_G\, g_Y}{g_R} - \frac{\pi_C\pi_T\, g_R}{g_Y}\right)
|
||||
\]
|
||||
|
||||
\[
|
||||
w_1 = 1 - \frac{P_1}{k_1} - \frac{Q}{2g_R}, \quad
|
||||
w_2 = 1 - \frac{P_2}{k_2} - \frac{Q}{2g_Y}, \quad
|
||||
w_3 = 1 - \frac{Q}{2g_R g_Y}
|
||||
\]
|
||||
|
||||
\[
|
||||
d_{TN93} = -k_1 \ln w_1 - k_2 \ln w_2 - k_3 \ln w_3
|
||||
\]
|
||||
|
||||
**`+Γ` gamma correction** (Jin & Nei 1990): every formula above is a
|
||||
weighted sum of \(-\ln(x)\) terms; the gamma-corrected version replaces
|
||||
each such term with the same weight applied to
|
||||
\(\alpha\left(x^{-1/\alpha} - 1\right)\) instead — the standard mechanical
|
||||
substitution (as \(\alpha \to \infty\), this expression → \(-\ln(x)\),
|
||||
recovering the uncorrected formula exactly). E.g. for JC:
|
||||
|
||||
\[
|
||||
d_{JC,\Gamma} = \frac{3}{4}\,\alpha\left[\left(1-\frac{4p}{3}\right)^{-1/\alpha} - 1\right]
|
||||
\]
|
||||
|
||||
Verified term-by-term against `ape`'s own gamma branches for JC69/K80/F81
|
||||
(including K80's two-term form — algebraically identical to the generic
|
||||
substitution applied to `snp-k2p`'s own \(a_1\)/\(a_2\) terms above, checked
|
||||
both symbolically and numerically before simplifying the implementation to
|
||||
share one `corrected_log` helper across every model rather than
|
||||
special-casing K80). K81/T92/TN93's gamma branches follow the same
|
||||
mechanical substitution but weren't independently checked against an
|
||||
`ape`-equivalent reference for those three specifically — flagged here, not
|
||||
silently assumed correct.
|
||||
|
||||
### Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
**Decided, not yet implemented.** The primary distance-matrix output
|
||||
**Implemented.** The primary distance-matrix output
|
||||
(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
|
||||
default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
|
||||
Reference in New Issue
Block a user