add name-tree command and fix --free-loss cost matrix
Introduce the obikmer name-tree subcommand to map numeric leaf labels in phylogenetic tree exports back to taxon names using a reference FASTA file. Correct the --free-loss flag behavior by removing cardinality transition costs from pairwise cost calculations, ensuring sibling gains and losses are priced identically to whole-family events. Update documentation, configuration parameters, and add reference phylogenetic data files.
This commit is contained in:
@@ -1956,6 +1956,79 @@ re-run against the local `iqtree3` binary to confirm `+ASC` now accepts the
|
||||
filtered alignment — next verification step, alongside the still-open PhyG
|
||||
`?`-vs-gap-row question above.
|
||||
|
||||
### `--family-overlap`/`--min-shared-family`, and a second `free_loss` bug: cardinality-transition cost not zeroed (2026-08-13)
|
||||
|
||||
**Diagnostic added: `--family-overlap`.** Real per-genome-pair count of
|
||||
shared variable families (columns where neither genome is `∅`), diagonal
|
||||
kept (degenerates to "genome's own total family count", not a separate
|
||||
computation — see `obikmer/src/cmd/phylo/family_overlap.rs`). On the
|
||||
20-genome benchmark this predicted the `--tnt`/`--iqtree` rogue-placement
|
||||
pattern exactly: Candidozyma shares **0** families with all 19 others (hence
|
||||
its ~0-length branch, placed anywhere); the mean off-diagonal shared-family
|
||||
count per genome splits cleanly below/above ~1000 (`Candidozyma` 0,
|
||||
`Saccharolobus` 0.9, `Opitutus` 20.5, `Wolbachia` 37.1, `Acidobacterium`
|
||||
41.2, `Shouchella` 49.4, `Bacillus` 54.5, `Proteus` 629, `Yersinia` 907.5 —
|
||||
all below 1000 — vs. `Klebsiella` ~5000-6000, `Salmonella` ~8000-17000,
|
||||
`E. coli` ~19000-24000, all comfortably above). `--min-shared-family N`
|
||||
automates exclusion below that threshold, reusing the `--exclude-genome`
|
||||
machinery. Restricting to the 3 genera above threshold (11 genomes: 4
|
||||
*E. coli*, 3 *Klebsiella*, 4 *Salmonella*) gives 3 cleanly separated
|
||||
monophyletic clades in both TNT and IQ-TREE (without `+ASC`, see below) —
|
||||
confirms the family-overlap statistic, not just the tree topology, as the
|
||||
right lens for this failure mode. `N=1000` is benchmark-specific (huge
|
||||
cross-domain divergence), not a universal constant — recompute
|
||||
`--family-overlap` and read its own gap on any other dataset.
|
||||
|
||||
**Second `free_loss` bug, found by re-deriving the design rather than by a
|
||||
failing run: the cardinality-transition cost wasn't zeroed.**
|
||||
`--free-loss` neutralises `∅` (whole-family non-detection) by recoding it to
|
||||
`?`, so transitions to/from the empty state are never scored. But the
|
||||
calibrated 16-state cost matrix (`obikindex::pairwise_cost_matrix`) prices
|
||||
*every* cardinality change — not just `X → ∅` — via a shared
|
||||
`P_cardinality(|A|→|B|)` factor (the same constant this project has called
|
||||
`c_ctx` throughout: "used everywhere a member is gained or lost", see "A
|
||||
concrete Sankoff cost matrix" above). So a transition between two
|
||||
*non-empty* states that differ in how many family members are detected
|
||||
(e.g. `{A} -> {A,C}`, gaining a second sibling) was still carrying the full
|
||||
calibrated gain/loss penalty under `--free-loss`, even though the same
|
||||
low/incomplete-coverage argument for not trusting whole-family absence
|
||||
applies identically one level down: whether a genome shows 1 vs 2 detected
|
||||
members of a family it does carry is exactly as vulnerable to sampling
|
||||
failure as whether the family was detected at all. Fixed by adding a
|
||||
`free_loss: bool` parameter to `pairwise_cost_matrix` that drops the
|
||||
`P_cardinality` factor from the log-probability sum entirely when set,
|
||||
leaving cost driven only by composition matching (shared-base retention,
|
||||
paired substitutions) — never by cardinality difference alone. Unit-tested
|
||||
(`free_loss_ignores_cardinality_transition_probs`): two wildly different
|
||||
cardinality models produce an identical cost matrix once `free_loss` is
|
||||
true. Not yet re-run against a real index to see how much this changes the
|
||||
resulting tree beyond the unit-level guarantee.
|
||||
|
||||
**`+ASC` misbehaves on closely-related taxa, separately from either bug
|
||||
above.** Restricting to the 3-genus, 11-genome subset and running `--iqtree`
|
||||
with `+ASC` converges suspiciously fast compared to the same run without
|
||||
`+ASC` (which takes the expected ~100 NNI iterations, ~1m38s, converges to a
|
||||
stable score, total tree length 7.147, and gives the clean 3-clade
|
||||
topology). Not yet root-caused with a real `+ASC` log in hand (the one run
|
||||
overwrote its own log under the shared `--prefix`, see the operational note
|
||||
below) — working hypothesis, not confirmed: `+ASC`'s correction divides by
|
||||
`P(site variable | current model, current branch lengths)`, recomputed
|
||||
every iteration; on closely-related strains (short branches), a plausible
|
||||
model predicts most sites *should* be constant, so `P(variable)` can be
|
||||
small enough that the correction becomes numerically unstable. Needs a
|
||||
dedicated run (own `--prefix`, log kept) before this is more than a
|
||||
hypothesis.
|
||||
|
||||
**Operational note, not a code bug: IQ-TREE's `--undo`/checkpoint reuse
|
||||
across different alignments crashes.** Reusing `iqtree.ckp.gz` (and the tree
|
||||
it stores) from a run on one taxon set against a regenerated `iqtree.fasta`
|
||||
with a different taxon count crashes with `Assertion 'i >= 0 && i <
|
||||
(int)seq_names.size()' failed` in `Alignment::getSeqName` /
|
||||
`PhyloTree::assignLeafNames` — `--undo` tries to graft a stored tree with
|
||||
the old leaf count onto the new, smaller alignment. Not an `obikmer` bug;
|
||||
avoid by using a fresh `--prefix` (or `-redo`) whenever the underlying
|
||||
alignment's taxon set changes, never `--undo` across them.
|
||||
|
||||
## References
|
||||
|
||||
The Mash mutation-rate model this discussion contrasts with:
|
||||
|
||||
Reference in New Issue
Block a user