feat: introduce _iqtree_states.csv for compact symbol mapping

Generates a new CSV output that maps IQ-TREE's compact state symbols to canonical states alongside full-precision empirical frequencies. Updates documentation to clarify that state frequencies sum to 1.0 by design and documents conditional behavior under `--free-loss`. Includes unit tests verifying absent state exclusion, frequency summation, and CSV structure. Also restricts entropy annex resolution to non-monomorphic minorants to eliminate redundant per-genome checks.
This commit is contained in:
Eric Coissac
2026-08-17 09:38:26 +02:00
parent c6cfdac043
commit 9654201885
6 changed files with 230 additions and 4 deletions
+41
View File
@@ -2056,6 +2056,47 @@ the old leaf count onto the new, smaller alignment. Not an `obikmer` bug;
avoid by using a fresh `--prefix` (or `-redo`) whenever the underlying
alignment's taxon set changes, never `--undo` across them.
## `_iqtree_states.csv`: compact-symbol traceability (2026-08-15)
User-reported suspicion: `--iqtree --free-loss` state frequencies "don't
sum to 1". Verified both by code trace and unit test
(`obikmer/src/cmd/phylo/iqtree.rs`, `free_loss_excludes_absent_state_and_freq_sums_to_one`)
— **not a bug**: `compact_alphabet`'s counting loop `continue`s on
`free_loss && b == b'-'` *before* recoding/counting, so the absent state
never enters `occurs`/`counts`, and `freq[i] = counts[old]/total` sums to
1.0 by construction (`total` is defined as the sum over exactly the same
states). Confirmed against real data the user provided: a pasted
`_iqtree.model` frequency line summed to `1.000001` — a 6-decimal
print-rounding artifact (`format!("{p:.6}")` × 15 values), not a
computation error; IQ-TREE's own 3-decimal-rounded report of the same
values summed to exactly `1.000`.
Investigating that report surfaced a real, separate gap: nothing mapped
`_iqtree.model`/`_iqtree.fasta`'s compact state symbols (`0-9A-F`, IQ-TREE
renumbers away unused states from the full 16, see this module's own
"`--iqtree`... state count" discussion) back to the canonical 16-state
alphabet (`STATE_SYMBOL`, `sankoff.rs`) — so a pattern like "compact state
0 has zero exchangeability with every other state" (`R(a,b) =
exp(-cost(a,b)) = 0` for an entire row/column) couldn't be traced back to
which real state that was, or whether it was expected (a state combination
that was simply never observed alongside anything else in the calibration
data, giving `cost = -ln(0) = +∞`) or a genuine problem.
**Fix**: `write_iqtree_states_csv` (`iqtree.rs`) writes
`<prefix>_iqtree_states.csv` — one row per surviving state,
`iqtree_symbol,canonical_symbol,frequency`, frequency at full `f64`
precision (not the model file's truncated 6 decimals). Written alongside
`_iqtree.model`/`_iqtree.fasta` from the same `CompactAlphabet` both
already use, so there is no risk of the three files disagreeing. Covered
by `states_csv_maps_compact_symbols_back_to_canonical_ones`.
The zero-exchangeability pattern itself (state 0 in the user's report,
frequency 3.26%, `R=0` with every other state) is not yet explained —
plausibly a genuinely unobserved transition in the calibration
(`cardinality_transitions`/`composition_transitions` count `0` for every
pair involving it), which is a legitimate, if numerically extreme, result
of `-ln(0)`, not necessarily a bug — not investigated further.
## References
The Mash mutation-rate model this discussion contrasts with: