feat(distance): implement native Sankoff calibration and backends
Replaces external Python glue with native Rust modules for Sankoff model calibration, exporting calibrated cost matrices, FASTA alignments, and YAML parameters. Adds dedicated writers for TNT and PhyG that apply integer scaling and Floyd-Warshall metric closure to enforce triangle inequality. Integrates these exporters into the distance command pipeline to streamline downstream tree inference workflows, while updating theory documentation to reflect IQ-TREE integration and state renumbering improvements.
This commit is contained in:
Eric Coissac
@@ -620,6 +620,285 @@ simply does not survive domain-level divergence. Cross-domain placement
|
||||
would need conserved-marker characters (rRNA, ribosomal proteins), not this
|
||||
estimator.
|
||||
|
||||
### Native `--sankoff --tnt`/`--phyg` export (2026-08-12), superseding the external scripts above
|
||||
|
||||
The ad hoc Python glue from the previous section is superseded: `obikmer
|
||||
distance --sankoff` now calibrates the matrix natively (`p_hat`, 6-category
|
||||
substitution costs, `c_ctx` weighted by `mean_sub_cost` — see the worked
|
||||
example above) and `--tnt`/`--phyg` each write a ready-to-run script from
|
||||
it, no external script needed. `∅` is an ordinary 16th state throughout
|
||||
(never `-`), specifically to avoid gap-semantics confusion in downstream
|
||||
tools — see "A concrete Sankoff cost matrix" above for why.
|
||||
|
||||
**TNT (`--tnt`).** `write_sankoff_tnt` (`obikmer/src/cmd/distance.rs`)
|
||||
recodes to TNT's own `0-9A-F` xread alphabet (its default reader rejects
|
||||
the wider IUPAC set otherwise), scales and rounds costs to integers
|
||||
(`smatrix`/`cost` reject decimals), then re-runs integer Floyd-Warshall on
|
||||
the rounded matrix (`scaled_metric_matrix`) — independently rounding each
|
||||
cell of an already-metric real-valued matrix can break the triangle
|
||||
inequality (e.g. two real costs of `1.734` round to `173` each, summing to
|
||||
`346`, while their real sum `3.468` rounds to `347`), which TNT otherwise
|
||||
silently "fixes" itself with an unreproducible correction. Verified against
|
||||
the real 20-genome benchmark index: zero triangle-inequality violations
|
||||
after the fix, TNT loads the file without its "triangle inequality
|
||||
violated... Fixed" warning.
|
||||
|
||||
Two syntax facts worth recording because they're wrong in intuitive
|
||||
guesses and contradicted actual TNT behavior when tested: TNT's plain
|
||||
command stream has **no comment syntax** of its own — `/* */` and `[ ]`
|
||||
only work inside the (separately-enabled, off by default) macro scripting
|
||||
language, and error with "No command!" otherwise. The working substitute
|
||||
is `quote TEXT ;` (prints the text, doesn't affect parsing) — but the text
|
||||
itself can't contain a literal `;` (TNT's universal terminator); the
|
||||
manual's own escape (`.,`) exists but the script here just avoids
|
||||
semicolons in the text instead.
|
||||
|
||||
The default search command embedded in the script is `mult` (traditional:
|
||||
random addition sequences + TBR), not `xmult` (New Technology search:
|
||||
ratchet/drift/tree-fusion). `xmult` with TNT's default `mxram` (16 MB,
|
||||
must be set *before* `xread` if changed) ran out of RAM on the real
|
||||
908k-character dataset ("`xmult - out of ram`"); `mult` does not, matching
|
||||
what had already been validated by hand outside this session.
|
||||
|
||||
**PhyG (`--phyg`).** `write_sankoff_phyg` writes a `tcm:` custom-alphabet
|
||||
matrix (same scale+round+metric-closure treatment as TNT) and reuses
|
||||
`--sankoff`'s own `_sankoff.fasta` as-is via `prefasta:` — PhyG's `tcm:`
|
||||
alphabet is read from the matrix file's own first line, so (unlike TNT) no
|
||||
recoding is needed. PhyG auto-adds its own indel/gap state as an
|
||||
`(n+1)`-th row/column of the tcm; inert here since the alignment encodes
|
||||
absence as `0`, never `-`.
|
||||
|
||||
`report("file", newick, overwrite)` — exactly as shown in PhyG's own
|
||||
manual — triggers `Unrecognized/missing report option ... defaulting to
|
||||
'graphs'` on the locally installed binary (1.3, commit `3c1a1fa`); the
|
||||
working form adds `graphs` explicitly: `report("file", graphs, newick,
|
||||
overwrite)`. Manual/binary mismatches like this (also true of `criterion:`
|
||||
— the binary accepts `parsimony`/`ml`/`pmdl`, the manual instead documents
|
||||
`mapa`/`ncm`/`parsimony`/`pmdl`/`si`) mean command syntax against this PhyG
|
||||
build should be verified empirically, not trusted from the PDF alone.
|
||||
`instances:N` (not a separate CPU flag) is what actually parallelises the
|
||||
search across cores — PhyG uses all physical cores by default but only
|
||||
across as many instances as are running, so raise it to the physical core
|
||||
count to use them all (the CLI-level `+RTS -NX -RTS` flag also exists but
|
||||
controls something else: capping/limiting cores, not requesting more).
|
||||
|
||||
Both scripts share one `--sankoff-cost-scale` (default `100`), not two
|
||||
separate flags — they scale the same calibrated matrix for the same
|
||||
reason (integer-only cost commands) and no PhyG-specific
|
||||
accumulator-width constraint was ever found to justify a different
|
||||
default from TNT's (TNT: hinted 32-bit accumulators in its own manual;
|
||||
PhyG: no such hint found — Haskell's native `Int` is typically 64-bit).
|
||||
|
||||
**Open problem: PhyG reports all branch lengths as `0.0`.** The graph-level
|
||||
parsimony cost is correct (`3.3286×10⁸` on the real dataset, consistent
|
||||
with TNT's `328574911` on the same calibrated matrix), but every individual
|
||||
edge in the exported Newick shows `:0.0`, with the total cost only ever
|
||||
shown as a whole-tree annotation (`[3.32860377e8]`). Not fixed, not fully
|
||||
diagnosed — PhyG's manual describes per-edge branch length as computed by
|
||||
ancestral-state (HTU) backtracking, well documented for sequence/standard
|
||||
character types, but nothing found (the term "Sankoff" doesn't even appear
|
||||
in the manual) confirming this backtracking is wired up for a custom
|
||||
`tcm:` matrix character. Switching `criterion:` to a likelihood-family
|
||||
option (`ml`, or the manual's `mapa`/`ncm`/`si`) was considered as a
|
||||
possible fix but is very unlikely to be one: those criteria are
|
||||
information-theoretic reparametrisations of the *same* step-counting
|
||||
machinery as parsimony (`ncm` in particular is known in the literature to
|
||||
be numerically equivalent to weighted parsimony), not classical
|
||||
continuous-time-Markov ML with a real rate matrix — so they wouldn't
|
||||
change how branch length is attributed per edge either.
|
||||
|
||||
**Export format note (not a bug in the generator).** Neither script's
|
||||
`.tre` output opens in PearTree (FigTree's successor) via File > Open —
|
||||
association/Launch-Services quirks were ruled out (the file was opened
|
||||
directly through the app, not by double-click). Likely cause, not yet
|
||||
confirmed: TNT's export is a minimal NEXUS `begin trees;` block with no
|
||||
preceding `Taxa` block and bare numeric (untranslated) leaf labels; PhyG's
|
||||
is multiple raw Newick trees concatenated with no NEXUS wrapper at all
|
||||
plus a trailing `[cost]` bracket tag after the root label. Both differ
|
||||
from a "normal" single-tree, fully-declared NEXUS file; this is PhyG/TNT's
|
||||
own export format, not something `write_sankoff_tnt`/`write_sankoff_phyg`
|
||||
could fix without post-processing the *other* program's output after the
|
||||
fact.
|
||||
|
||||
### Next direction: genuine ML branch lengths, not parsimony (open, 2026-08-12)
|
||||
|
||||
Decided: parsimony (the whole `--sankoff`/`--tnt`/`--phyg` pipeline above)
|
||||
is a stopgap, not the destination. The goal is maximum likelihood with
|
||||
real, calibrated branch lengths (expected substitutions/site), which
|
||||
parsimony step-counts were never going to give directly (see the open
|
||||
"branch lengths are `0.0`" problem above — even if fixed, TNT/PhyG-style
|
||||
parsimony branch length is a step count, not a continuous ML estimate).
|
||||
|
||||
**Model choices, settled:**
|
||||
- **The exchangeability `R` is symmetric; the rate matrix `Q` is not.**
|
||||
(Superseded an earlier, wrong framing here that treated "symmetric
|
||||
model" as one thing — see the resolution below on `R` vs `Q` vs `π` for
|
||||
the full reasoning.) `R(a,b) = R(b,a)` because `BasePairTally` never
|
||||
captured direction — a fact about the data, not a modelling choice.
|
||||
`Q(a,b) = R(a,b)·π_b` is asymmetric whenever the real state frequencies
|
||||
`π` are (which they are, empirically) — biology drives this via `π`,
|
||||
not via `R`.
|
||||
- **`∅` stays an ordinary 16th state**, as already established for
|
||||
TNT/PhyG — same reasoning applies to any ML tool: encode as a real
|
||||
alphabet symbol, never as `-`/gap, or the RAxML-era failure (empty set
|
||||
silently treated as missing data) repeats.
|
||||
|
||||
**Stationary frequencies for the 16 states — resolved (2026-08-12).**
|
||||
A CTMC needs a rate matrix `Q`, generally asymmetric. `Q(i,j) = R(i,j) ·
|
||||
π_j`, where `R` (exchangeability) is symmetric and `π` (stationary
|
||||
frequencies) need not be — this factoring is what makes `Q` reversible
|
||||
(satisfies detailed balance, `π_i·Q(i,j) = π_j·Q(j,i)`) for *any* `π`, not
|
||||
just uniform, as long as `R` is symmetric. Two separate, both-easy
|
||||
quantities, not one hard inverse problem:
|
||||
- **`R` is already calibrated**: `sub_cost` (`-ln(observed rate)` per pair,
|
||||
from `BasePairTally`) *is* `R` up to a log transform — recover it as
|
||||
`R(a,b) = exp(-sub_cost(a,b))`. Symmetric by construction, because the
|
||||
tally itself never distinguished direction (unordered-pair counts only)
|
||||
— not a modelling choice, a fact about what the data can say.
|
||||
- **`π` is a direct count**: empirical marginal frequency of each of the
|
||||
16 states across the whole alignment (same kind of scan already used to
|
||||
confirm `N` occurs 1383 times in the real 20-genome benchmark). With
|
||||
~908k sites × 20 genomes, the counts are large enough that this is
|
||||
precise on its own — no need to spend ML degrees of freedom
|
||||
re-estimating it via IQ-TREE's `+FO`.
|
||||
|
||||
Checked and ruled out along the way: IQ-TREE's `+F` (empirical, "compute
|
||||
from the alignment") does **not** work as a shortcut for this — for a
|
||||
custom-file morphology model, `readParameters` always requires the file's
|
||||
own frequency line unconditionally; omitting it and passing `+F` instead
|
||||
just fails (`ERROR: State frequencies could not be read`). `π` has to be
|
||||
computed by `obikmer` and written into the file, not left to IQ-TREE.
|
||||
|
||||
Net effect: no free-rate ML estimation needed for this piece at all (the
|
||||
mistaken assumption that motivated most of this discussion — that
|
||||
building an asymmetric `Q` a priori would require solving a linear system
|
||||
from `Q` itself — doesn't apply, because `R`, the only piece that's
|
||||
genuinely hard to get directionally, is symmetric and already in hand).
|
||||
|
||||
**Candidate tool: IQ-TREE**, because it supports user-defined multistate
|
||||
models (unlike RAxML's `MULTI` data type, which is limited to the
|
||||
equal-rate Mk model and can't take a custom rate matrix at all — a genuine
|
||||
tool limitation, not a gap-symbol encoding problem this time). IQ-TREE 3
|
||||
(3.0.1) is now installed locally (Homebrew, `iqtree3`).
|
||||
|
||||
**IQ-TREE custom-model format — verified empirically against the local
|
||||
binary (2026-08-12).** The web docs' `-mdef` NEXUS `begin models;
|
||||
frequency NAME = ...; model NAME = ...; end;` mechanism (initially assumed
|
||||
to apply directly, see history below) turned out to be for **named
|
||||
components used inside `MIX{...}`/`FMIX{...}` mixture models only** — it
|
||||
does **not** apply to a single, non-mixture custom morphology matrix, and
|
||||
using it that way fails (`ERROR: File not found <name>` — traced in
|
||||
IQ-TREE 3's own source, `model/modelmorphology.cpp`: any `-m` string that
|
||||
isn't `MK`/`ORDERED`/`GTR`/`GTRX` is passed straight to
|
||||
`ModelMarkov::readParameters()`, which opens it **as a literal file path**,
|
||||
never consulting the `-mdef` models block at all for this data type).
|
||||
|
||||
**The confirmed working recipe** (built a tiny 5-taxon/3-state toy dataset
|
||||
and rate file, ran it end to end with `iqtree3`, got a real ML tree with
|
||||
non-zero branch lengths and an optimized log-likelihood — ground truth,
|
||||
not documentation):
|
||||
- No `-mdef` needed. Write one plain file (any name) containing, as
|
||||
whitespace/newline-separated numbers, in order: the **lower-triangular
|
||||
rate matrix** (`N(N-1)/2` values, PAML row-major order — for 16 states,
|
||||
120 values, the same count and layout already produced for TNT's
|
||||
`smatrix`), immediately followed by the **N state frequencies** on the
|
||||
same stream (no header, no blank line required — confirmed by reading
|
||||
`ModelMorphology::readRates`/`ModelMarkov::readStateFreq` directly, which
|
||||
just pull tokens off the stream in sequence).
|
||||
- Invoke with `-m <path-to-that-file>+ASC` (`+ASC` for the no-constant-site
|
||||
correction, as before). An explicit `+F{f1,...,fN}` on the command line
|
||||
overrides the file's own frequency line if given (confirmed in
|
||||
`ModelMorphology::init`) — useful once real calibrated stationary
|
||||
frequencies exist, a placeholder equal-frequency line works meanwhile
|
||||
(the still-open gap noted above).
|
||||
- `--seqtype MORPH` (alphabet `0`-`9`,`A`-`V`, ≤31 states) — reuse the same
|
||||
`0-9A-F` recoding already built for TNT (`TNT_STATE_SYMBOL`).
|
||||
|
||||
**Risk, confirmed, precisely characterised, and resolved by design
|
||||
(2026-08-12).**
|
||||
`--seqtype MORPH{16}` **does not force the state count** for real ML
|
||||
analysis — tested directly (`--seqtype MORPH{4}` on the 3-symbol toy
|
||||
alignment gave the byte-for-byte identical 3-state result as no `{4}` at
|
||||
all) and confirmed in source: the value it sets
|
||||
(`params.alisim_num_states_morph`, `utils/tools.cpp`) is consumed only by
|
||||
the `--alisim` simulator; the main analysis path always calls
|
||||
`getDataBlockMorphStates`/an equivalent scan (`alignment/alignment.cpp`),
|
||||
for both FASTA/PHYLIP and NEXUS input (a NEXUS `symbols=` declaration
|
||||
doesn't change this either — checked, same code path). No CLI flag or
|
||||
NEXUS declaration overrides it.
|
||||
|
||||
The precise rule (from `getDataBlockMorphStates`, `alignment.cpp:1058`):
|
||||
`N` = **one plus the highest state ordinal actually observed anywhere in
|
||||
the alignment**, ordinal being the symbol's position in IQ-TREE's own
|
||||
fixed table `"0123456789ABCDEFGHIJKLMNOPQRSTUV"` — not a count of distinct
|
||||
symbols seen. So the risk is narrower than "any missing symbol breaks it":
|
||||
concretely, it's whether the symbol mapped to state index 15 (`F` in the
|
||||
`0-9A-F` recoding already used for TNT, i.e. our `N` = "all four bases
|
||||
ambiguous") occurs **at least once anywhere** in the real alignment — if
|
||||
it does, `N` correctly comes out to 16 regardless of which lower-index
|
||||
symbols (including `0`/`∅`) are rare or absent; if it doesn't, `N` silently
|
||||
undercounts and misaligns every value in a 16-entry rate/frequency file,
|
||||
with no error to catch it. **Checked against the real biological alignment**
|
||||
(`/tmp/msg_test/eub_sankoff.fasta`, the 20-genome benchmark index): the
|
||||
symbol `N` (IUPAC "all four bases ambiguous," state index 15) occurs 1383
|
||||
times across 13 of the 20 sequences — present, so this specific real
|
||||
dataset is not at risk. Still worth a real, general presence check inside
|
||||
`obikmer` before this is wired in, rather than assuming every future
|
||||
dataset will have `N` too (nothing in IQ-TREE would catch it if not).
|
||||
|
||||
**Resolution: subset + compact-renumber, not rely on all 16 appearing.**
|
||||
Since IQ-TREE always infers `N` from the alignment's own content and
|
||||
nothing overrides that, the fix is to make the file `obikmer` writes match
|
||||
that inference *by construction*, for every run, rather than hope the
|
||||
16th (or any particular) state happens to occur:
|
||||
1. Scan the real alignment for which of the 16 canonical states actually
|
||||
occur anywhere (not per-column — anywhere in the whole alignment).
|
||||
2. Renumber the occurring states to a **compact, consecutive** `0..k-1`
|
||||
range, preserving their relative order (the original bitmask/`STATE_
|
||||
SYMBOL` ordering) — not just filtering, since a *gap* in the ordinal
|
||||
sequence (e.g. keeping states `{0,1,2,4}` numbered as-is instead of
|
||||
`{0,1,2,3}`) reproduces the exact same "highest observed ordinal"
|
||||
miscount this was meant to fix.
|
||||
3. Recode the alignment itself with this new compact `k`-symbol alphabet
|
||||
(same recoding mechanism already used for TNT's `0-9A-F`, just over a
|
||||
possibly-smaller symbol set).
|
||||
4. Extract the matching `k×k` submatrix (rows/columns for the kept
|
||||
states only) from the full calibrated 16×16 cost matrix, in the same
|
||||
lower-triangular order the rate-matrix file needs — and, later, the
|
||||
matching `k`-length subset of stationary frequencies once those are
|
||||
calibrated (still the open gap noted earlier in this section).
|
||||
|
||||
Consequence, and why nothing is lost: a state that never occurs in a given
|
||||
alignment can, by definition, never contribute a transition to score in
|
||||
that same alignment — dropping it from that run's matrix costs nothing.
|
||||
The subset (and therefore `k`) can differ from one dataset/run to the
|
||||
next; this has to be done freshly per alignment, not computed once and
|
||||
reused.
|
||||
|
||||
*(Superseded reasoning, kept for the record: the `-mdef` NEXUS route
|
||||
below was the original plan, based on IQ-TREE's own web documentation for
|
||||
protein mixture models, before the local install allowed testing it —
|
||||
`GTRX` combined with a `-mdef`-referenced custom model, `+Fname` frequency
|
||||
reference. Both pieces exist and parse without error individually, but
|
||||
`GTRX`/`GTR` are IQ-TREE's own fixed built-in equal-structure multistate
|
||||
model, not a hook for an arbitrary custom matrix; a custom matrix is a
|
||||
file path in `-m` directly, no `-mdef` or `GTRX` involved.)*
|
||||
|
||||
Source for the empirical findings above: `model/modelmorphology.cpp` and
|
||||
`model/modelmarkov.cpp` in the local `iqtree/iqtree3` source (cloned to
|
||||
inspect the exact parsing logic after documentation didn't resolve the
|
||||
`+Fname` reference error) — more reliable here than the PDF/web manual,
|
||||
which (like TNT/PhyG) doesn't always match this specific binary. Original
|
||||
(partially superseded) sources: [Substitution
|
||||
Models](https://iqtree.github.io/doc/Substitution-Models), [Complex
|
||||
Models](https://iqtree.github.io/doc/Complex-Models).
|
||||
|
||||
**Relation to the existing calibration.** `sub_cost[a][b] = -ln(rate)`
|
||||
(see "A concrete Sankoff cost matrix" above) is already a log-rate — a
|
||||
genuine CTMC rate matrix `Q` could plausibly be recovered as
|
||||
`rate(a,b) = exp(-cost(a,b))`, renormalised so each row sums to zero, once
|
||||
the stationary-frequency gap above is closed. Not yet attempted.
|
||||
|
||||
## Heterozygosity, ploidy, and consensus-assembly inputs
|
||||
|
||||
A within-genome multiplicity signal (more than one of the 4 central forms
|
||||
|
||||
Reference in New Issue
Block a user