add name-tree command and fix --free-loss cost matrix

Introduce the obikmer name-tree subcommand to map numeric leaf labels in phylogenetic tree exports back to taxon names using a reference FASTA file. Correct the --free-loss flag behavior by removing cardinality transition costs from pairwise cost calculations, ensuring sibling gains and losses are priced identically to whole-family events. Update documentation, configuration parameters, and add reference phylogenetic data files.
This commit is contained in:
Eric Coissac
2026-08-16 13:11:06 +02:00
parent 8615da59a8
commit eee71430a4
14 changed files with 1444 additions and 82 deletions
+73
View File
@@ -1956,6 +1956,79 @@ re-run against the local `iqtree3` binary to confirm `+ASC` now accepts the
filtered alignment — next verification step, alongside the still-open PhyG
`?`-vs-gap-row question above.
### `--family-overlap`/`--min-shared-family`, and a second `free_loss` bug: cardinality-transition cost not zeroed (2026-08-13)
**Diagnostic added: `--family-overlap`.** Real per-genome-pair count of
shared variable families (columns where neither genome is ``), diagonal
kept (degenerates to "genome's own total family count", not a separate
computation — see `obikmer/src/cmd/phylo/family_overlap.rs`). On the
20-genome benchmark this predicted the `--tnt`/`--iqtree` rogue-placement
pattern exactly: Candidozyma shares **0** families with all 19 others (hence
its ~0-length branch, placed anywhere); the mean off-diagonal shared-family
count per genome splits cleanly below/above ~1000 (`Candidozyma` 0,
`Saccharolobus` 0.9, `Opitutus` 20.5, `Wolbachia` 37.1, `Acidobacterium`
41.2, `Shouchella` 49.4, `Bacillus` 54.5, `Proteus` 629, `Yersinia` 907.5 —
all below 1000 — vs. `Klebsiella` ~5000-6000, `Salmonella` ~8000-17000,
`E. coli` ~19000-24000, all comfortably above). `--min-shared-family N`
automates exclusion below that threshold, reusing the `--exclude-genome`
machinery. Restricting to the 3 genera above threshold (11 genomes: 4
*E. coli*, 3 *Klebsiella*, 4 *Salmonella*) gives 3 cleanly separated
monophyletic clades in both TNT and IQ-TREE (without `+ASC`, see below) —
confirms the family-overlap statistic, not just the tree topology, as the
right lens for this failure mode. `N=1000` is benchmark-specific (huge
cross-domain divergence), not a universal constant — recompute
`--family-overlap` and read its own gap on any other dataset.
**Second `free_loss` bug, found by re-deriving the design rather than by a
failing run: the cardinality-transition cost wasn't zeroed.**
`--free-loss` neutralises `` (whole-family non-detection) by recoding it to
`?`, so transitions to/from the empty state are never scored. But the
calibrated 16-state cost matrix (`obikindex::pairwise_cost_matrix`) prices
*every* cardinality change — not just `X → ∅` — via a shared
`P_cardinality(|A|→|B|)` factor (the same constant this project has called
`c_ctx` throughout: "used everywhere a member is gained or lost", see "A
concrete Sankoff cost matrix" above). So a transition between two
*non-empty* states that differ in how many family members are detected
(e.g. `{A} -> {A,C}`, gaining a second sibling) was still carrying the full
calibrated gain/loss penalty under `--free-loss`, even though the same
low/incomplete-coverage argument for not trusting whole-family absence
applies identically one level down: whether a genome shows 1 vs 2 detected
members of a family it does carry is exactly as vulnerable to sampling
failure as whether the family was detected at all. Fixed by adding a
`free_loss: bool` parameter to `pairwise_cost_matrix` that drops the
`P_cardinality` factor from the log-probability sum entirely when set,
leaving cost driven only by composition matching (shared-base retention,
paired substitutions) — never by cardinality difference alone. Unit-tested
(`free_loss_ignores_cardinality_transition_probs`): two wildly different
cardinality models produce an identical cost matrix once `free_loss` is
true. Not yet re-run against a real index to see how much this changes the
resulting tree beyond the unit-level guarantee.
**`+ASC` misbehaves on closely-related taxa, separately from either bug
above.** Restricting to the 3-genus, 11-genome subset and running `--iqtree`
with `+ASC` converges suspiciously fast compared to the same run without
`+ASC` (which takes the expected ~100 NNI iterations, ~1m38s, converges to a
stable score, total tree length 7.147, and gives the clean 3-clade
topology). Not yet root-caused with a real `+ASC` log in hand (the one run
overwrote its own log under the shared `--prefix`, see the operational note
below) — working hypothesis, not confirmed: `+ASC`'s correction divides by
`P(site variable | current model, current branch lengths)`, recomputed
every iteration; on closely-related strains (short branches), a plausible
model predicts most sites *should* be constant, so `P(variable)` can be
small enough that the correction becomes numerically unstable. Needs a
dedicated run (own `--prefix`, log kept) before this is more than a
hypothesis.
**Operational note, not a code bug: IQ-TREE's `--undo`/checkpoint reuse
across different alignments crashes.** Reusing `iqtree.ckp.gz` (and the tree
it stores) from a run on one taxon set against a regenerated `iqtree.fasta`
with a different taxon count crashes with `Assertion 'i >= 0 && i <
(int)seq_names.size()' failed` in `Alignment::getSeqName` /
`PhyloTree::assignLeafNames` — `--undo` tries to graft a stored tree with
the old leaf count onto the new, smaller alignment. Not an `obikmer` bug;
avoid by using a fresh `--prefix` (or `-redo`) whenever the underlying
alignment's taxon set changes, never `--undo` across them.
## References
The Mash mutation-rate model this discussion contrasts with: