Add phylogenetic export support for TNT, PhyG, and IQ-TREE
Extends the phylogenetic pipeline with exporters for TNT, PhyG, and IQ-TREE that generate executable scripts, cost matrices, and recoded alignments. Adds internal helpers for state indexing, floating-point matrix scaling, and Floyd-Warshall metric closure to satisfy external tool constraints. Adjusts visibility modifiers for sibling iterators and entropy annex structs, and updates design documentation for evolutionary distance metrics.
This commit is contained in:
Eric Coissac
@@ -2182,6 +2182,92 @@ Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
|
||||
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
|
||||
workspace `cargo test` green.
|
||||
|
||||
## `--distance` unification: SNP corrections as first-class metrics (discussion, 2026-08-28)
|
||||
|
||||
**Decided, not yet implemented.** `--metric` (renamed `--distance` — several of
|
||||
its existing values, e.g. Bray-Curtis, aren't metrics in the strict sense,
|
||||
`--metric` was a misnomer) gains a family of `snp-*` values computed from the
|
||||
central-position SNP pipeline, routed internally to the sibling-annex
|
||||
machinery (`PairwiseTally`, `obikphylo::siblings::algorithms::pairwise`)
|
||||
instead of `cache.distance(...)`'s existing per-layer traversal — a different
|
||||
code path behind the same CLI surface, not just another branch of one
|
||||
formula function.
|
||||
|
||||
**Why unify at the CLI level despite the implementation split**: phylogenetically
|
||||
a SNP-corrected distance is a distance like any other — NJ/UPGMA are agnostic
|
||||
to how the matrix was produced, so exposing it as a special-cased subcommand
|
||||
instead of a `--distance` value would misrepresent its role. The
|
||||
implementation divergence (sibling-annex-based vs. plain index scan) is real
|
||||
but belongs at the routing layer, invisible to the CLI's own vocabulary.
|
||||
|
||||
**`--subsample` becomes optional for `snp-*` distances** (it stays mandatory
|
||||
for `--sankoff`/`--pseudo-alignment`, unrelated commands): absent means
|
||||
exhaustive, achieved for free by reusing `sample_index`'s existing
|
||||
proportional-per-layer-quota mechanism with `n` set to the index-wide total
|
||||
non-monomorphic-minorant count (already available from the sibling-annex
|
||||
stats) — every layer's quota then equals its own full count, giving Bernoulli
|
||||
`p = 1` everywhere, i.e. every eligible family is drawn. No second,
|
||||
exhaustive-only driver needed. Present means sampled, exactly as `--sankoff`
|
||||
already behaves.
|
||||
|
||||
**One shared tally, many derived formulas.** `PairwiseTally`'s `subst[4][4]`
|
||||
per-pair substitution counts (plus marginal base frequencies derived from it)
|
||||
are the sufficient statistic for every closed-form correction below — each
|
||||
is a small pure function `PairwiseTally -> Array2<f64>`, at the same level as
|
||||
the already-implemented `raw_snp_distance`/`base_pair_tally`/
|
||||
`cardinality_tally`. No new full scan per formula, whether the tally itself
|
||||
was built exhaustively or from a subsample.
|
||||
|
||||
**`--raw-snp-counts` stays a separate, unrelated flag** — same underlying
|
||||
tally, but a diagnostic (`n_snp`/`n_shared`/`n_eligible` per genome pair, one
|
||||
row per pair) rather than a distance value, and its long-table shape doesn't
|
||||
fold into a single N×N matrix the way a distance does. No change to its
|
||||
existing CSV format.
|
||||
|
||||
### `snp-*` distance catalog
|
||||
|
||||
All closed-form (method-of-moments / direct formula), none requiring
|
||||
per-pair or per-tree maximum-likelihood fitting — that excludes HKY85's
|
||||
*tree*-ML usage but not its *pairwise* estimator, which is closed-form like
|
||||
F84/TN93 and is included below. `snp-` prefix on every CLI value.
|
||||
|
||||
| value | corrects for | inputs beyond raw counts |
|
||||
|---|---|---|
|
||||
| `snp-raw` | nothing (uncorrected p-distance) | — |
|
||||
| `snp-jc` (Jukes-Cantor, JC69) | multiple substitutions per site | — |
|
||||
| `snp-k2p` (Kimura 2-parameter, K80) | + transition/transversion rate bias | ts/tv split |
|
||||
| `snp-k81` (Kimura 3-parameter, K3ST) | + splits transversions into 2 categories | ts/tv split, by category |
|
||||
| `snp-f81` (Felsenstein 81) | + unequal base frequencies (no ts/tv split) | empirical base freqs |
|
||||
| `snp-tajima-nei` (Tajima-Nei 1984) | same goal as F81 (equal-input model), different formula, better small-sample behavior | empirical base freqs |
|
||||
| `snp-t92` (Tamura 3-parameter) | K2P + GC-content bias | ts/tv split, GC content |
|
||||
| `snp-f84` (Felsenstein 84) | full empirical base freqs + single ts/tv rate | empirical base freqs, ts/tv split |
|
||||
| `snp-hky85` (Hasegawa-Kishino-Yano, pairwise estimator) | same inputs as F84, different formula | empirical base freqs, ts/tv split |
|
||||
| `snp-tn93` (Tamura-Nei) | full empirical base freqs + separate purine/pyrimidine transition rates + transversion rate | empirical base freqs, purine-ts/pyrimidine-ts/tv split |
|
||||
| `snp-logdet` (LogDet / paralinear) | no shared-model or stationarity assumption at all — general divergence-matrix determinant | full empirical 4×4 divergence matrix (already `subst[4][4]`) |
|
||||
| `snp-tv` (transversions-only p-distance) | diagnostic/deep-divergence variant — drops transitions entirely (they saturate first) | tv-only counts |
|
||||
|
||||
**`+Γ` rate-heterogeneity modifier, applicable to `snp-jc`, `snp-k2p`,
|
||||
`snp-k81`, `snp-t92`, `snp-f84`, `snp-hky85`, `snp-tn93`** (not `snp-raw`,
|
||||
nothing to correct; not `snp-logdet`, no standard gamma formulation) — same
|
||||
formula as the base correction, weighted by a shape parameter `α` supplied
|
||||
by the user (`--gamma-shape <alpha>`), not estimated by ML. A modifier on
|
||||
existing values, not a separate enum arm per distance.
|
||||
|
||||
### Output format: PHYLIP-relaxed by default for the distance matrix
|
||||
|
||||
**Decided, not yet implemented.** The primary distance-matrix output
|
||||
(`_dist.csv` today) gains multiple formats: **PHYLIP-relaxed becomes the
|
||||
default** (widely read by external NJ tools — PHYLIP `neighbor`, FastME,
|
||||
T-REX, SplitsTree — relaxed rather than strict to avoid the 10-character
|
||||
label truncation, since genome labels here routinely exceed it), a `--csv`
|
||||
flag opts back into the current CSV format, PHYLIP-strict is a possible
|
||||
future addition (not now). This changes the *default* output of every
|
||||
existing `--distance` value (jaccard, hamming, bray-curtis, ...), not just
|
||||
the new `snp-*` ones — accepted explicitly (pre-release, single developer
|
||||
user, no external consumers to break). Scoped to the distance matrix only:
|
||||
`--shared-kmers` and `--raw-snp-counts` are counts, not distances, and keep
|
||||
their existing CSV-only format.
|
||||
|
||||
## References
|
||||
|
||||
The Mash mutation-rate model this discussion contrasts with:
|
||||
|
||||
Reference in New Issue
Block a user