fix: prevent probability underflow in pairwise cost matrix

Replaces premature exponentiation-based row normalization with log-sum-exp arithmetic to prevent tiny probabilities from collapsing to exactly zero. This eliminates spurious infinite costs for valid but rare transitions while preserving correct IEEE 754 semantics for genuinely unobserved pairs. Adds explicit guards against NaN in degenerate rows and includes a regression test verifying finite costs for probabilities as low as 1e-200.

DevDocMD/theory/evolutionary_distances.md

@@ -2090,12 +2090,47 @@ precision (not the model file's truncated 6 decimals). Written alongside
 already use, so there is no risk of the three files disagreeing. Covered
 by `states_csv_maps_compact_symbols_back_to_canonical_ones`.
 
-The zero-exchangeability pattern itself (state 0 in the user's report,
-frequency 3.26%, `R=0` with every other state) is not yet explained —
-plausibly a genuinely unobserved transition in the calibration
-(`cardinality_transitions`/`composition_transitions` count `0` for every
-pair involving it), which is a legitimate, if numerically extreme, result
-of `-ln(0)`, not necessarily a bug — not investigated further.
+**Root cause of the zero-exchangeability pattern found and fixed
+(2026-08-15): premature `exp()` in `pairwise_cost_matrix` underflowed
+merely-tiny probabilities to exactly `0.0`.** The user also reported
+`iqtree3` emitting "Numerical underflow for lh-derivative" warnings on
+the same run — a real signal, traced to `obikphylo/src/cardcomp.rs`'s
+`pairwise_cost_matrix`, not to the frequency computation (which is a
+plain, safe `f64` division, never close to underflow at any realistic
+scale). The function already accumulated `log_p` in log-space (correct),
+but then row-normalised by exponentiating each cell *first*
+(`raw[a][b] = log_p.exp()`) and summing the results — `f64::exp` hard
+underflows to exactly `0.0` for any input below roughly `-709`, which a
+sum of several individually-small-but-nonzero probability factors
+(composition/cardinality terms, `best_pairing_cost`'s pairing terms) can
+reach easily on real, skewed calibration data. Once `raw[a][b]` was
+exactly `0.0`, normalisation and `-ln` turned a merely tiny probability
+into a `+∞` cost indistinguishable from a *literally* unobserved
+transition (`p == 0.0` exactly, e.g. `p_comp[i][j]` never once tallied) —
+conflating two different things: "never observed" (should be `+∞`, a
+correct MLE result) and "observed, but the joint probability of this
+multi-step transition is extremely small" (should be a large *finite*
+cost).
+
+**Fix**: row-normalise via log-sum-exp instead of exponentiating first —
+`row_max = max_b(log_p[a][b])`, `log_sum = row_max + ln(Σ_b
+exp(log_p[a][b] - row_max))` (every shifted term is in `(0,1]`, so this
+never underflows for a finite `log_p[a][b]`), then
+`cost[a][b] = log_sum - log_p[a][b]` directly — no intermediate
+probability is ever materialised. This falls out of IEEE 754 arithmetic
+without a special case: a genuinely-unobserved factor (`log_p[a][b] ==
+-∞`, from the existing `if p > 0.0 {...} else { NEG_INFINITY }` guards
+already in the log-accumulation loop) still yields `cost = +∞` exactly
+(`finite − (−∞) = +∞`), preserving the correct semantics for that case,
+while every merely-tiny-but-nonzero transition now gets a large but
+*finite* cost. A degenerate all-`-∞` row (a state with literally zero
+probability of transitioning to anything, `row_max == -∞`) is guarded
+explicitly to avoid a `-∞ − (-∞) = NaN` in the log-sum-exp itself.
+Covered by `cardcomp::tests::underflow_prone_transition_gets_finite_cost_not_infinite`
+(all off-diagonal composition probabilities set to `1e-200`, well past
+where the old `exp()`-first code would have underflowed to `0.0`, cost
+asserted finite). Every pre-existing `cardcomp` test still passes
+unchanged (numerically identical results when no underflow occurs).
 
 ## References