feat(phylo): add --iqtree-min-freq to filter rare nucleotide states

Introduces --iqtree-min-freq (default 0.001) to treat low-frequency nucleotide states as missing data during IQ-TREE alignment generation when --free-loss is active. This triggers a recoding pass that folds rare states into the missing symbol, followed by non-informative site removal and alphabet recomputation to maintain output consistency. The change also adds Sankoff model configuration files and updates related tests and documentation.

doc/usage/phylo/index.html

@@ -946,6 +946,17 @@
     </span>
   </a>
   
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#rare-states-and-iqtree-min-freq" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Rare states and --iqtree-min-freq
+      
+    </span>
+  </a>
+  
 </li>
         
       </ul>
@@ -1510,6 +1521,17 @@
     </span>
   </a>
   
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#rare-states-and-iqtree-min-freq" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        Rare states and --iqtree-min-freq
+      
+    </span>
+  </a>
+  
 </li>
         
       </ul>
@@ -1790,6 +1812,11 @@
 <td>Also write an IQ-TREE custom model and alignment (implies <code>--sankoff</code>)</td>
 </tr>
 <tr>
+<td><code>--iqtree-min-freq</code></td>
+<td><code>0.001</code></td>
+<td>With <code>--iqtree --free-loss</code>: also treat as missing any state rarer than this in the alignment</td>
+</tr>
+<tr>
 <td><code>--sankoff-cost-scale</code></td>
 <td><code>100</code></td>
 <td>Integer scaling factor applied to costs before rounding (required by TNT/PhyG's integer-only cost commands)</td>
@@ -2051,10 +2078,12 @@
 <p><strong><code>_sankoff.tcm</code></strong> (<code>--phyg</code>) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state <code>0</code>, since it's never actually triggered).</p>
 <p><strong><code>_sankoff.pg</code></strong> (<code>--phyg</code>) — script: <code>read(prefasta:..., tcm:...)</code> against <code>_sankoff.fasta</code>/<code>_sankoff.tcm</code>, a default 300s/4-instance <code>search</code>, <code>report(...)</code> writing <code>&lt;prefix&gt;_sankoff.tre</code> (bare numeric labels, as for <code>--tnt</code>). Run with <code>phyg</code> from the output directory (the script uses relative file names). Feed the tree plus <code>_sankoff.fasta</code> into <a href="../name-tree/"><code>name-tree</code></a> for taxon names.</p>
 <p><strong><code>_iqtree.model</code></strong> (<code>--iqtree</code>) — lower-triangular exchangeability matrix <span class="arithmatex">\(R(a,b) = e^{-\text{cost}(a,b)}\)</span> (one row of increasing length per state, whitespace-separated, PAML order), followed by one line of empirical state frequencies. Only states actually occurring in the alignment are kept, compactly renumbered <code>0..k-1</code>.</p>
-<p><strong><code>_iqtree.fasta</code></strong> (<code>--iqtree</code>) — alignment recoded to that same compact <code>0..k-1</code> alphabet (symbols <code>0-9A-F</code>). Under <code>--free-loss</code>, non-detection becomes <code>?</code> and columns left non-informative once missing calls are ignored are dropped first (required for <code>+ASC</code>). Run with:
+<p><strong><code>_iqtree.fasta</code></strong> (<code>--iqtree</code>) — alignment recoded to that same compact <code>0..k-1</code> alphabet (symbols <code>0-9A-F</code>). Under <code>--free-loss</code>, non-detection becomes <code>?</code> and columns left non-informative once missing calls are ignored are dropped first (required for <code>+ASC</code>); with <code>--iqtree-min-freq</code> also set (the default), any state rarer than that threshold is folded into the same <code>?</code> treatment, and non-informative columns are re-checked and dropped again after that. Run with:
 <div class="highlight"><pre><span></span><code>iqtree3 -s &lt;prefix&gt;_iqtree.fasta --seqtype MORPH -m &lt;prefix&gt;_iqtree.model+ASC --prefix &lt;prefix&gt;_iqtree -T AUTO
 </code></pre></div></p>
-<p><strong><code>_iqtree_states.csv</code></strong> (<code>--iqtree</code>) — one row per state actually kept in <code>_iqtree.model</code>/<code>_iqtree.fasta</code> (header <code>iqtree_symbol,canonical_symbol,frequency</code>): <code>iqtree_symbol</code> is the compact <code>0-9A-F</code> symbol as written in those two files, <code>canonical_symbol</code> is the matching <code>_sankoff_matrix.csv</code> state, <code>frequency</code> is that state's empirical frequency at full precision (<code>_iqtree.model</code>'s own frequency line is rounded to 6 decimals). Under <code>--free-loss</code>, absent (<code>0</code>/<code>?</code>) is never a kept state, so it never appears here. Use this file to identify which real state a given row/column of <code>_iqtree.model</code>'s matrix corresponds to — e.g. to check whether a state showing zero exchangeability with everything else is expected (a state combination that never co-occurs with anything else in this data) or worth investigating further.</p>
+<p><strong><code>_iqtree_states.csv</code></strong> (<code>--iqtree</code>) — one row per state actually kept in <code>_iqtree.model</code>/<code>_iqtree.fasta</code> (header <code>iqtree_symbol,canonical_symbol,frequency</code>): <code>iqtree_symbol</code> is the compact <code>0-9A-F</code> symbol as written in those two files, <code>canonical_symbol</code> is the matching <code>_sankoff_matrix.csv</code> state, <code>frequency</code> is that state's empirical frequency at full precision (<code>_iqtree.model</code>'s own frequency line is rounded to 6 decimals). Under <code>--free-loss</code>, absent (<code>0</code>/<code>?</code>) is never a kept state, so it never appears here — nor does any state <code>--iqtree-min-freq</code> folded away for being too rare. Use this file to identify which real state a given row/column of <code>_iqtree.model</code>'s matrix corresponds to — e.g. to check whether a state showing zero exchangeability with everything else is expected (a state combination that never co-occurs with anything else in this data) or worth investigating further.</p>
+<h3 id="rare-states-and-iqtree-min-freq">Rare states and <code>--iqtree-min-freq</code></h3>
+<p>States that combine 3 or 4 central bases at once (IUPAC <code>V</code>/<code>H</code>/<code>K</code>.../<code>N</code>) are inherently rare — and, on real data, rare enough that they can make <code>iqtree3</code> itself numerically unstable ("Numerical underflow for lh-derivative" warnings, near-degenerate likelihood optimization). They're also more likely to be assembly/detection noise than genuine, widely-shared multi-way polymorphism, the same "sampling failure, not true signal" reasoning <code>--free-loss</code> already applies to non-detection. With <code>--free-loss</code> set, <code>--iqtree-min-freq</code> (default <code>0.001</code>, i.e. one in a thousand) extends that same missing-data treatment to any state below this frequency, not just absence. Check <code>_iqtree_states.csv</code> to see exactly which states survived and at what frequency; set <code>--iqtree-min-freq 0</code> to disable this and keep every state that occurs at all (the old behavior). Has no effect without <code>--free-loss</code> — there is no missing-data symbol to fold rare states into otherwise.</p>