feat(phylo): add --iqtree-min-freq to filter rare nucleotide states

Introduces --iqtree-min-freq (default 0.001) to treat low-frequency nucleotide states as missing data during IQ-TREE alignment generation when --free-loss is active. This triggers a recoding pass that folds rare states into the missing symbol, followed by non-informative site removal and alphabet recomputation to maintain output consistency. The change also adds Sankoff model configuration files and updates related tests and documentation.
This commit is contained in:
Eric Coissac
2026-08-17 11:06:03 +02:00
parent c8f2b16b4c
commit 128db64564
10 changed files with 756 additions and 7 deletions
+50
View File
@@ -2132,6 +2132,56 @@ where the old `exp()`-first code would have underflowed to `0.0`, cost
asserted finite). Every pre-existing `cardcomp` test still passes
unchanged (numerically identical results when no underflow occurs).
## `--iqtree-min-freq`: rare states still perturb IQ-TREE after the underflow fix (implemented 2026-08-15)
After the `pairwise_cost_matrix` fix above, the user re-checked
`_iqtree_states.csv` on the real `phyloskims_sal_vac` run: no more exact
zeros, but `N` (all 4 bases), `V`/`H`/`D` (3 bases each) sit at
`8×10⁻⁵``4×10⁻⁴`, and `B` (3 bases) just above `10⁻³` — all four
3-simultaneous-base states plus `N`, exactly the states the underflow fix
was papering over the symptom of. `iqtree3` kept emitting "Numerical
underflow for lh-derivative" on this same run — a model with several
near-zero-frequency states is still numerically marginal for ML
optimisation even once no cost/exchangeability entry is *exactly*
infinite/zero.
**Decided and implemented**: extend `--free-loss`'s existing "sampling
failure, not true signal" treatment of non-detection to rare
multi-base states too — biologically, a family showing 34 simultaneously
observed central bases across genomes is far more likely to be
assembly/detection noise than a genuinely ancestral, widely-preserved
polymorphism, the same reasoning already used for `-`.
**Mechanism** (`obikmer/src/cmd/phylo/iqtree.rs`, `write_iqtree`): a
*second* filtering pass, active only under `--free-loss`. After the first
`compact_alphabet` call (post ascertainment-filtering), any state with
`freq < --iqtree-min-freq` (default `0.001`) has every occurrence recoded
to `-` (`recode_symbols_as_absent`, a new small helper) — reusing the
*existing* `-`→`?` machinery in `compact_alphabet`/`write_iqtree_alignment`
rather than duplicating it, since a state recoded to `-` flows through
exactly the same downstream path a genuinely-absent call already does.
`drop_ascertainment_noninformative` then re-runs on the recoded alignment
(a site can become non-informative once its only distinguishing call was
the now-recoded rare state), and `compact_alphabet` is recomputed a third
time on the result — this final alphabet is what `_iqtree_states.csv`/
`_iqtree.model`/`_iqtree.fasta` all get built from, so the three files
stay consistent with each other by construction, same as before this
change.
Not a fixed-point loop (recoding one rare state can't, by construction,
lower another *already-above-threshold* state's frequency enough to also
cross the threshold, since removing calls only redistributes mass upward
among the survivors) — a single extra pass is sufficient.
`--iqtree-min-freq 0` disables the filter (no state can have frequency
`< 0`), recovering the pre-this-change behavior exactly.
Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
(20 common A/C sites + one site introducing a single, otherwise-informative
`M` call at ~1/62 frequency, `--iqtree-min-freq 0.05`; asserts `M` absent
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
workspace `cargo test` green.
## References
The Mash mutation-rate model this discussion contrasts with: