Add phylogenetic CLI options for family overlap and missing data

Introduces CLI flags for computing pairwise family overlap matrices and filtering genomes below a shared family threshold. Adds a free-loss mode that recodes locus non-detection states to missing data symbols in Sankoff-calibrated alignments, resolving ascertainment bias handling for IQ-TREE. Updates empirical transition parameters, removes the legacy model asset, and extends output writers for CSV diagnostics, FASTA pseudo-alignments, and Newick trees.

docmd/theory/evolutionary_distances.md

@@ -1836,6 +1836,126 @@ lookups proportionally; `p_hat` is unbiased. Off by default (exact).
    matrix).
 5. Subsampling (Step 5).
 
+## Locus dropout under incomplete coverage: absence as missing data, not a penalized state (2026-08-13, discussion)
+
+Distinct from the SNP/Sankoff calibration above (which conditions on
+*conserved-flank* comparability), this addresses the coarser presence/absence
+signal of a family itself, for genome-skim/reduced-representation inputs
+where per-genome coverage is often `<1x`, sometimes `<0.5x`. At that depth,
+non-detection of a locus is dominated by Poisson sampling failure, not by
+true loss — even for a single-copy locus, `P(non-detection | present)` is
+non-negligible. Treating "absent" as a real 0-state therefore risks
+clustering genomes by shared undersampling rather than shared ancestry.
+
+**Parsimony.** Loss and gain are the two directions of the same transition,
+so they cannot be freed asymmetrically without a biological justification for
+trusting one direction and not the other — none exists here. Freeing both
+directions to zero cost makes the character contribute nothing to any tree's
+score regardless of topology, which is operationally equivalent to just
+coding the state as ambiguous (`?`) rather than 0/1. `?` is the more honest
+encoding when downstream ancestral-state reconstruction is wanted: cost-zero
+Fitch/Sankoff still assigns internal states arbitrarily among equal-cost
+options, whereas `?` makes the "we don't know" explicit.
+
+**ML.** The correct analogue is not a tuned asymmetric loss/gain rate but
+literal missing-data coding at the tip: an ambiguous state (all-states tip
+likelihood vector, e.g. `(1,1)` for a 2-state character) so Felsenstein
+pruning marginalises over it and the character contributes likelihood 1
+(neutral) for that taxon — the same mechanism already used for the `N`
+IUPAC-ambiguous case in the 16-state Sankoff encoding (see "IQ-TREE
+custom-model format" above). No rate parameter to justify or estimate.
+
+**Open item, not yet decided.** No per-locus/per-genome confidence
+distinction is implemented: the current plan is to code *every* non-detected
+locus as missing/ambiguous, uniformly, without a depth-based threshold for
+"confidently absent." Deferred because at typical skim coverage the
+confidently-absent case is rare enough that a uniform treatment is
+defensible as a first pass. Known consequence to watch for: a genome with
+very low overall coverage ends up with a mostly-`?` row and constrains the
+tree weakly — will likely need a minimum-detected-loci-per-genome filter
+upstream (exclude or flag under-covered genomes) before this becomes a
+correctness problem rather than a power problem. Not designed yet.
+
+### `--free-loss` implemented (2026-08-13)
+
+`obikmer phylo --sankoff --free-loss` (with `--tnt`/`--phyg`/`--iqtree`)
+recodes `∅` (a family's non-detection in a genome) to `?` instead of the
+default ordinary, costed 16th state — the resolution of the discussion
+above. `?`, not `-`: `-` still carries gap/indel semantics in TNT/PhyG/
+IQ-TREE (see "`∅` stays an ordinary 16th state" above, and the reasoning
+that motivated it in the first place — the RAxML-era failure where `-`
+was silently swallowed as missing data was undesired back when `∅` was
+meant to be fully informative; `--free-loss` is the first case where that
+same tool behaviour is exactly what's wanted). `?` is the standard
+missing/ambiguous symbol these tools already recognise without needing to
+be declared in the custom alphabet, so parsimony treats it as
+"compatible with any state" (Sankoff/Fitch's own missing-data extension —
+no per-transition cost edit needed) and Felsenstein pruning marginalises
+it for IQ-TREE's ML.
+
+Implementation is a recoding at each export's existing `-`-normalisation
+boundary, not a change to matrix calibration: `write_sankoff_alignment_fasta`
+(`sankoff.rs`, feeds `--phyg` too via its `prefasta:`), `write_sankoff_tnt`
+(`tnt.rs`, `?` written directly, bypassing `TNT_STATE_SYMBOL` — there is no
+state index for it), and `compact_alphabet`/`write_iqtree_alignment`
+(`iqtree.rs`, `∅` excluded from the compact alphabet entirely rather than
+assigned a symbol). The calibrated cost matrix itself is unchanged — its
+`∅`-row/column simply becomes unused once no sequence contains that state.
+
+**Unverified, flagged rather than assumed** (this project's own standard for
+tool-specific claims — see the TNT/PhyG/IQ-TREE syntax facts above, all
+checked against the real binaries, not the manuals alone):
+- PhyG auto-adds its own `(n+1)`-th gap/indel row to the `tcm:` matrix
+  (`phyg.rs`); with `--free-loss` the reused `_sankoff.fasta` now contains
+  real `?` characters PhyG did not have to interpret before. Not confirmed
+  against the local PhyG binary that `?` is read as "any state" rather than
+  routed through that gap row, or otherwise mishandled.
+- IQ-TREE's exact `?`/missing-data semantics for `--seqtype MORPH` with a
+  custom-file model plus `+ASC` — assumed to marginalise via Felsenstein
+  pruning by the general convention, not yet re-verified against the local
+  `iqtree3` build the way the rest of the custom-model mechanism was.
+
+Both should be checked against a real run (small index) before `--free-loss`
+results are trusted for TNT/PhyG/IQ-TREE, the same way every other
+tool-specific behaviour in this file was established.
+
+### TNT confirmed (2026-08-13); IQ-TREE `+ASC` constant-column fix (2026-08-13)
+
+**TNT.** Real run, 20-genome benchmark index, `--free-loss --tnt`: `?`
+present for all 20 taxa in both `sankoff.fasta` and `sankoff.tnt`, loaded
+with no syntax error and no triangle-inequality warning, `mult` search
+completed, `sankoff.tre` holds 20 equally-parsimonious trees. Topology is
+exactly what the theory predicts: strain-level clades unaffected (the 4 *E.
+coli*, the *Klebsiella* trio, the *Salmonella* pairs — driven by real
+substitution signal, not gene-content) but resolution among the deeply
+divergent lineages (*Candidozyma*, *Saccharolobus*, *Wolbachia*,
+*Acidobacterium*, *Opitutus*, *Bacillus*/*Shouchella*) is weak and unstable
+across the 20 tied trees — the expected trade-off of discarding the
+gene-content/dropout signal that used to help place exactly those lineages.
+TNT's own handling of `?` — read as "compatible with any state" — no longer
+a flagged assumption.
+
+**IQ-TREE: `+ASC` broke on real data — non-informative columns survive
+`--free-loss`, distinct bug from the `--exclude-genome` one above.**
+`snp_pseudo_alignment`'s "variable family" test (`family_size() >= 2`) is a
+whole-annex property, blind to any single column's actual calls. Once
+non-detection is hidden as `?` (ignored for constancy) rather than counted
+as its own real state, a family that is genuinely variable *somewhere in the
+dataset* can still have only one distinct call among the genomes that
+actually carry it in one particular column, with the rest `?` — constant
+once missing data is excluded, which is exactly what `+ASC` forbids. Same
+underlying failure mode as `--exclude-genome`'s (documented above, "Two
+consistency bugs found and fixed post-implementation"), same remedy: rescan
+columns and drop the ones that are constant once `-` is ignored — but a
+different trigger (hiding cells vs. dropping whole rows), so a separate fix
+was needed rather than reusing `drop_excluded` directly. Implemented as
+`iqtree::drop_ascertainment_noninformative`, run only under `--free-loss`,
+only on IQ-TREE's own copy of the alignment (`--tnt`/`--phyg` have no
+no-invariant-site requirement and must not lose those columns). Not yet
+re-run against the local `iqtree3` binary to confirm `+ASC` now accepts the
+filtered alignment — next verification step, alongside the still-open PhyG
+`?`-vs-gap-row question above.
+
 ## References
 
 The Mash mutation-rate model this discussion contrasts with: