OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

feat(phylo): add --iqtree-min-freq to filter rare nucleotide states

Browse Source 
Introduces --iqtree-min-freq (default 0.001) to treat low-frequency nucleotide states as missing data during IQ-TREE alignment generation when --free-loss is active. This triggers a recoding pass that folds rare states into the missing symbol, followed by non-informative site removal and alphabet recomputation to maintain output consistency. The change also adds Sankoff model configuration files and updates related tests and documentation.
This commit is contained in:
Eric Coissac
2026-08-17 11:06:03 +02:00
parent c8f2b16b4c
commit 128db64564
10 changed files with 756 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DevDoc/theory/evolutionary_distances/index.html
+65
View File
@@ -898,6 +898,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#-iqtree-min-freq-rare-states-still-perturb-iq-tree-after-the-underflow-fix-implemented-2026-08-15" class="md-nav__link">
<span class="md-ellipsis">
--iqtree-min-freq: rare states still perturb IQ-TREE after the underflow fix (implemented 2026-08-15)
</span>
</a>
</li>
<li class="md-nav__item">
@@ -2117,6 +2128,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#-iqtree-min-freq-rare-states-still-perturb-iq-tree-after-the-underflow-fix-implemented-2026-08-15" class="md-nav__link">
<span class="md-ellipsis">
--iqtree-min-freq: rare states still perturb IQ-TREE after the underflow fix (implemented 2026-08-15)
</span>
</a>
</li>
<li class="md-nav__item">
@@ -4140,6 +4162,49 @@ Covered by <code>cardcomp::tests::underflow_prone_transition_gets_finite_cost_no
where the old <code>exp()</code>-first code would have underflowed to <code>0.0</code>, cost
asserted finite). Every pre-existing <code>cardcomp</code> test still passes
unchanged (numerically identical results when no underflow occurs).</p>
<h2 id="-iqtree-min-freq-rare-states-still-perturb-iq-tree-after-the-underflow-fix-implemented-2026-08-15"><code>--iqtree-min-freq</code>: rare states still perturb IQ-TREE after the underflow fix (implemented 2026-08-15)</h2>
<p>After the <code>pairwise_cost_matrix</code> fix above, the user re-checked
<code>_iqtree_states.csv</code> on the real <code>phyloskims_sal_vac</code> run: no more exact
zeros, but <code>N</code> (all 4 bases), <code>V</code>/<code>H</code>/<code>D</code> (3 bases each) sit at
<code>8×10⁻⁵</code>–<code>4×10⁻⁴</code>, and <code>B</code> (3 bases) just above <code>10⁻³</code> — all four
3-simultaneous-base states plus <code>N</code>, exactly the states the underflow fix
was papering over the symptom of. <code>iqtree3</code> kept emitting "Numerical
underflow for lh-derivative" on this same run — a model with several
near-zero-frequency states is still numerically marginal for ML
optimisation even once no cost/exchangeability entry is <em>exactly</em>
infinite/zero.</p>
<p><strong>Decided and implemented</strong>: extend <code>--free-loss</code>'s existing "sampling
failure, not true signal" treatment of non-detection to rare
multi-base states too — biologically, a family showing 3–4 simultaneously
observed central bases across genomes is far more likely to be
assembly/detection noise than a genuinely ancestral, widely-preserved
polymorphism, the same reasoning already used for <code>-</code>.</p>
<p><strong>Mechanism</strong> (<code>obikmer/src/cmd/phylo/iqtree.rs</code>, <code>write_iqtree</code>): a
<em>second</em> filtering pass, active only under <code>--free-loss</code>. After the first
<code>compact_alphabet</code> call (post ascertainment-filtering), any state with
<code>freq &lt; --iqtree-min-freq</code> (default <code>0.001</code>) has every occurrence recoded
to <code>-</code> (<code>recode_symbols_as_absent</code>, a new small helper) — reusing the
<em>existing</em> <code>-</code>→<code>?</code> machinery in <code>compact_alphabet</code>/<code>write_iqtree_alignment</code>
rather than duplicating it, since a state recoded to <code>-</code> flows through
exactly the same downstream path a genuinely-absent call already does.
<code>drop_ascertainment_noninformative</code> then re-runs on the recoded alignment
(a site can become non-informative once its only distinguishing call was
the now-recoded rare state), and <code>compact_alphabet</code> is recomputed a third
time on the result — this final alphabet is what <code>_iqtree_states.csv</code>/
<code>_iqtree.model</code>/<code>_iqtree.fasta</code> all get built from, so the three files
stay consistent with each other by construction, same as before this
change.</p>
<p>Not a fixed-point loop (recoding one rare state can't, by construction,
lower another <em>already-above-threshold</em> state's frequency enough to also
cross the threshold, since removing calls only redistributes mass upward
among the survivors) — a single extra pass is sufficient.</p>
<p><code>--iqtree-min-freq 0</code> disables the filter (no state can have frequency
<code>&lt; 0</code>), recovering the pre-this-change behavior exactly.</p>
<p>Covered by <code>iqtree::tests::iqtree_min_freq_folds_rare_states_into_missing</code>
(20 common A/C sites + one site introducing a single, otherwise-informative
<code>M</code> call at ~1/62 frequency, <code>--iqtree-min-freq 0.05</code>; asserts <code>M</code> absent
from the written <code>_iqtree_states.csv</code> and <code>A</code>/<code>C</code> still present). Full
workspace <code>cargo test</code> green.</p>
<h2 id="references">References</h2>
<p>The Mash mutation-rate model this discussion contrasts with:
(Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>