feat(phylo): add --iqtree-min-freq to filter rare nucleotide states
Introduces --iqtree-min-freq (default 0.001) to treat low-frequency nucleotide states as missing data during IQ-TREE alignment generation when --free-loss is active. This triggers a recoding pass that folds rare states into the missing symbol, followed by non-informative site removal and alphabet recomputation to maintain output consistency. The change also adds Sankoff model configuration files and updates related tests and documentation.
This commit is contained in:
@@ -2132,6 +2132,56 @@ where the old `exp()`-first code would have underflowed to `0.0`, cost
|
||||
asserted finite). Every pre-existing `cardcomp` test still passes
|
||||
unchanged (numerically identical results when no underflow occurs).
|
||||
|
||||
## `--iqtree-min-freq`: rare states still perturb IQ-TREE after the underflow fix (implemented 2026-08-15)
|
||||
|
||||
After the `pairwise_cost_matrix` fix above, the user re-checked
|
||||
`_iqtree_states.csv` on the real `phyloskims_sal_vac` run: no more exact
|
||||
zeros, but `N` (all 4 bases), `V`/`H`/`D` (3 bases each) sit at
|
||||
`8×10⁻⁵`–`4×10⁻⁴`, and `B` (3 bases) just above `10⁻³` — all four
|
||||
3-simultaneous-base states plus `N`, exactly the states the underflow fix
|
||||
was papering over the symptom of. `iqtree3` kept emitting "Numerical
|
||||
underflow for lh-derivative" on this same run — a model with several
|
||||
near-zero-frequency states is still numerically marginal for ML
|
||||
optimisation even once no cost/exchangeability entry is *exactly*
|
||||
infinite/zero.
|
||||
|
||||
**Decided and implemented**: extend `--free-loss`'s existing "sampling
|
||||
failure, not true signal" treatment of non-detection to rare
|
||||
multi-base states too — biologically, a family showing 3–4 simultaneously
|
||||
observed central bases across genomes is far more likely to be
|
||||
assembly/detection noise than a genuinely ancestral, widely-preserved
|
||||
polymorphism, the same reasoning already used for `-`.
|
||||
|
||||
**Mechanism** (`obikmer/src/cmd/phylo/iqtree.rs`, `write_iqtree`): a
|
||||
*second* filtering pass, active only under `--free-loss`. After the first
|
||||
`compact_alphabet` call (post ascertainment-filtering), any state with
|
||||
`freq < --iqtree-min-freq` (default `0.001`) has every occurrence recoded
|
||||
to `-` (`recode_symbols_as_absent`, a new small helper) — reusing the
|
||||
*existing* `-`→`?` machinery in `compact_alphabet`/`write_iqtree_alignment`
|
||||
rather than duplicating it, since a state recoded to `-` flows through
|
||||
exactly the same downstream path a genuinely-absent call already does.
|
||||
`drop_ascertainment_noninformative` then re-runs on the recoded alignment
|
||||
(a site can become non-informative once its only distinguishing call was
|
||||
the now-recoded rare state), and `compact_alphabet` is recomputed a third
|
||||
time on the result — this final alphabet is what `_iqtree_states.csv`/
|
||||
`_iqtree.model`/`_iqtree.fasta` all get built from, so the three files
|
||||
stay consistent with each other by construction, same as before this
|
||||
change.
|
||||
|
||||
Not a fixed-point loop (recoding one rare state can't, by construction,
|
||||
lower another *already-above-threshold* state's frequency enough to also
|
||||
cross the threshold, since removing calls only redistributes mass upward
|
||||
among the survivors) — a single extra pass is sufficient.
|
||||
|
||||
`--iqtree-min-freq 0` disables the filter (no state can have frequency
|
||||
`< 0`), recovering the pre-this-change behavior exactly.
|
||||
|
||||
Covered by `iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing`
|
||||
(20 common A/C sites + one site introducing a single, otherwise-informative
|
||||
`M` call at ~1/62 frequency, `--iqtree-min-freq 0.05`; asserts `M` absent
|
||||
from the written `_iqtree_states.csv` and `A`/`C` still present). Full
|
||||
workspace `cargo test` green.
|
||||
|
||||
## References
|
||||
|
||||
The Mash mutation-rate model this discussion contrasts with:
|
||||
|
||||
Reference in New Issue
Block a user