feat: introduce _iqtree_states.csv for compact symbol mapping

Generates a new CSV output that maps IQ-TREE's compact state symbols to canonical states alongside full-precision empirical frequencies. Updates documentation to clarify that state frequencies sum to 1.0 by design and documents conditional behavior under `--free-loss`. Includes unit tests verifying absent state exclusion, frequency summation, and CSV structure. Also restricts entropy annex resolution to non-monomorphic minorants to eliminate redundant per-genome checks.

DevDoc/architecture/siblings/index.html

@@ -2051,13 +2051,28 @@ but indexed by <code>family_idx</code> (every minorant of the layer, monomorphic
 included — the same numbering <code>Selection</code>/<code>scan_layer_families</code> already
 use), one <code>f32</code> entropy15 value per entry, <code>-1.0</code> sentinel for monomorphic/
 not-yet-computed. First use of <code>--entropy</code>/<code>--entropy-sd</code> on an index
-pays a one-time cost (<code>ensure_entropy_annexes</code> in <code>entropy.rs</code>: a full,
-unsampled <code>Selection::All</code> scan, resolving every non-monomorphic
-minorant's <code>genome_mask</code> once to compute and persist its entropy) — every
+pays a one-time cost (<code>ensure_entropy_annexes</code> in <code>entropy.rs</code>) — every
 later run (any <code>μ</code>/<code>σ</code>, any command) reads the file positionally, no
 re-scan, restoring the usual <code>Selection::Some</code> "skip resolving excluded
 families" speedup that a naive "weigh during the resolving scan" design
 would have permanently forfeited.</p>
+<p><strong>Bug found and fixed (2026-08-15): <code>ensure_entropy_annexes</code> scanned with
+<code>Selection::All</code> instead of bounding to non-monomorphic minorants.</strong>
+Monomorphism (<code>family_size() &lt; 2</code>) is knowable directly from the annex
+bits alone, no per-genome resolution needed — but the original
+implementation called <code>scan_layer_families</code> with <code>Selection::All</code>
+anyway, so <code>fill_sub_matrix_carries</code> (the expensive per-genome
+resolution) ran for <em>every</em> minorant, ~98% of which are monomorphic
+(measured elsewhere in this doc) and had their <code>genome_mask</code> immediately
+discarded once the callback checked <code>family_size() &lt; 2</code>. Fixed by adding
+[<code>subsample::non_monomorphic_selection_layer</code>] — a cheap, annex-only,
+non-sampling pass (same shape as <code>reservoir_sample_layer</code>, but keeping
+every non-monomorphic minorant's <code>family_idx</code> instead of a bounded
+reservoir) — and passing <code>Selection::Some(&amp;eligible)</code> instead of
+<code>Selection::All</code>, so the expensive resolution now runs only for the ~2%
+of minorants that can actually produce a real entropy value. A
+<code>debug_assert!(mask.family_size() &gt;= 2, ...)</code> inside the
+<code>scan_layer_families</code> callback guards the invariant.</p>
 <p><strong>Resolved</strong>: the existing hard "non-monomorphic minorant" eligibility
 filter stays a hard gate upstream of the Gaussian weighting — only
 qualifying families ever get a stored entropy value or a weighted draw.</p>

DevDoc/theory/evolutionary_distances/index.html

@@ -887,6 +887,17 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#_iqtree_statescsv-compact-symbol-traceability-2026-08-15" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        _iqtree_states.csv: compact-symbol traceability (2026-08-15)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -2095,6 +2106,17 @@
       </ul>
     </nav>
   
+</li>
+      
+        <li class="md-nav__item">
+  <a href="#_iqtree_statescsv-compact-symbol-traceability-2026-08-15" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        _iqtree_states.csv: compact-symbol traceability (2026-08-15)
+      
+    </span>
+  </a>
+  
 </li>
       
         <li class="md-nav__item">
@@ -4048,6 +4070,42 @@ with a different taxon count crashes with <code>Assertion 'i &gt;= 0 &amp;&amp;
 the old leaf count onto the new, smaller alignment. Not an <code>obikmer</code> bug;
 avoid by using a fresh <code>--prefix</code> (or <code>-redo</code>) whenever the underlying
 alignment's taxon set changes, never <code>--undo</code> across them.</p>
+<h2 id="_iqtree_statescsv-compact-symbol-traceability-2026-08-15"><code>_iqtree_states.csv</code>: compact-symbol traceability (2026-08-15)</h2>
+<p>User-reported suspicion: <code>--iqtree --free-loss</code> state frequencies "don't
+sum to 1". Verified both by code trace and unit test
+(<code>obikmer/src/cmd/phylo/iqtree.rs</code>, <code>free_loss_excludes_absent_state_and_freq_sums_to_one</code>)
+— <strong>not a bug</strong>: <code>compact_alphabet</code>'s counting loop <code>continue</code>s on
+<code>free_loss &amp;&amp; b == b'-'</code> <em>before</em> recoding/counting, so the absent state
+never enters <code>occurs</code>/<code>counts</code>, and <code>freq[i] = counts[old]/total</code> sums to
+1.0 by construction (<code>total</code> is defined as the sum over exactly the same
+states). Confirmed against real data the user provided: a pasted
+<code>_iqtree.model</code> frequency line summed to <code>1.000001</code> — a 6-decimal
+print-rounding artifact (<code>format!("{p:.6}")</code> × 15 values), not a
+computation error; IQ-TREE's own 3-decimal-rounded report of the same
+values summed to exactly <code>1.000</code>.</p>
+<p>Investigating that report surfaced a real, separate gap: nothing mapped
+<code>_iqtree.model</code>/<code>_iqtree.fasta</code>'s compact state symbols (<code>0-9A-F</code>, IQ-TREE
+renumbers away unused states from the full 16, see this module's own
+"<code>--iqtree</code>... state count" discussion) back to the canonical 16-state
+alphabet (<code>STATE_SYMBOL</code>, <code>sankoff.rs</code>) — so a pattern like "compact state
+0 has zero exchangeability with every other state" (<code>R(a,b) =
+exp(-cost(a,b)) = 0</code> for an entire row/column) couldn't be traced back to
+which real state that was, or whether it was expected (a state combination
+that was simply never observed alongside anything else in the calibration
+data, giving <code>cost = -ln(0) = +∞</code>) or a genuine problem.</p>
+<p><strong>Fix</strong>: <code>write_iqtree_states_csv</code> (<code>iqtree.rs</code>) writes
+<code>&lt;prefix&gt;_iqtree_states.csv</code> — one row per surviving state,
+<code>iqtree_symbol,canonical_symbol,frequency</code>, frequency at full <code>f64</code>
+precision (not the model file's truncated 6 decimals). Written alongside
+<code>_iqtree.model</code>/<code>_iqtree.fasta</code> from the same <code>CompactAlphabet</code> both
+already use, so there is no risk of the three files disagreeing. Covered
+by <code>states_csv_maps_compact_symbols_back_to_canonical_ones</code>.</p>
+<p>The zero-exchangeability pattern itself (state 0 in the user's report,
+frequency 3.26%, <code>R=0</code> with every other state) is not yet explained —
+plausibly a genuinely unobserved transition in the calibration
+(<code>cardinality_transitions</code>/<code>composition_transitions</code> count <code>0</code> for every
+pair involving it), which is a legitimate, if numerically extreme, result
+of <code>-ln(0)</code>, not necessarily a bug — not investigated further.</p>
 <h2 id="references">References</h2>
 <p>The Mash mutation-rate model this discussion contrasts with:
 (Fan <em>et al.</em> 2015; Marbl Lab 2026)<sup id="fnref:Mash-distances-doc"><a class="footnote-ref" href="#fn:Mash-distances-doc">1</a></sup> <sup id="fnref:Fan2015-mash-formula"><a class="footnote-ref" href="#fn:Fan2015-mash-formula">2</a></sup>.</p>