OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

Add phylogenetic CLI options for family overlap and missing data

Browse Source 
Introduces CLI flags for computing pairwise family overlap matrices and filtering genomes below a shared family threshold. Adds a free-loss mode that recodes locus non-detection states to missing data symbols in Sankoff-calibrated alignments, resolving ascertainment bias handling for IQ-TREE. Updates empirical transition parameters, removes the legacy model asset, and extends output writers for CSV diagnostics, FASTA pseudo-alignments, and Newick trees.
This commit is contained in:
Eric Coissac
2026-08-16 11:54:43 +02:00
parent e2b9374426
commit 8615da59a8
12 changed files with 833 additions and 465 deletions
Download Patch File Download Diff File Expand all files Collapse all files
UserDocMD/usage/phylo.md
+12 -1
View File
@@ -47,10 +47,12 @@ A **family** is the set of up to 4 kmers that share identical flanking sequence
|---|---|---|
| `--sibling-annex` | off | Build the sibling presence-mask annex (prerequisite for every option below) |
| `--exclude-genome LABEL` | none | Exclude a genome (repeatable) from every SNP/Sankoff/export computation below |
| `--min-shared-family N` | none | Auto-exclude any genome whose mean shared-family count against every other genome (see `--family-overlap`) falls below `N` — same exclusion as `--exclude-genome`, applied on top of it |
| `--sibling-stats` | off | Write the family-size (sibling count) distribution, per genome and globally |
| `--raw-snp-distance` | off | Write the single-copy central-SNP p-distance matrix |
| `--raw-snp-counts` | off | Write per-pair diagnostic counts (n_snp, n_shared, n_eligible) instead of a matrix |
| `--snp` | off | Write a SNP-only pseudo-alignment in FASTA, IUPAC-coded |
| `--family-overlap` | off | Write an NxN matrix of, for each genome pair, how many variable families both genomes actually carry a call for; the diagonal holds each genome's own total family count |
### Locus eligibility
@@ -62,12 +64,19 @@ A family is eligible for a genome pair $(i, j)$ only if genome $i$ carries exact
`--exclude-genome` removes a genome from these computations, re-checking column variability among the remaining genomes so that a column made monomorphic by the exclusion is dropped rather than kept artificially. It does not affect the `--metric` distance-matrix path.
### Family overlap and low-coverage genomes
`--family-overlap` writes, for every genome pair, how many variable families both genomes actually carry a call for (neither is absent) — a direct measure of how much informative content two genomes actually share. On genome-skim or otherwise incomplete-coverage collections, a genome with very little overlap with everything else has almost nothing left to constrain its position in a tree, and tends to end up placed unstably (near-zero branch length, grafted inside an unrelated clade) by `--tnt`/`--iqtree`.
`--min-shared-family N` automates the fix: it excludes, before any computation, every genome whose mean shared-family count against all other genomes (the same statistic, averaged per row of the `--family-overlap` matrix) falls below `N`. There is no universal value for `N` — it depends on how divergent and how completely covered the genome collection is; inspect `--family-overlap`'s own output to find where the real gap sits before choosing a threshold.
## Sankoff calibration and phylogenetic exports
| Option | Default | Description |
|---|---|---|
| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment |
| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the calibration |
| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, in `--sankoff`'s pseudo-alignment and every export built from it |
| `--tnt` | off | Also write a TNT script (implies `--sankoff`) |
| `--phyg` | off | Also write PhyG input files (implies `--sankoff`) |
| `--iqtree` | off | Also write an IQ-TREE custom model and alignment (implies `--sankoff`) |
@@ -84,6 +93,8 @@ which are combined into a row-normalized $16 \times 16$ transition probability m
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character, to avoid ambiguity with external tools' own gap semantics). It does not run any external tool.
With `--free-loss`, the empty state is recoded to `?` instead — TNT/PhyG/IQ-TREE's own missing-data symbol — rather than an ordinary, costed 16th state. This matters for genome-skim or otherwise incomplete-coverage collections, where non-detection of a family is dominated by sampling failure rather than true evolutionary loss: scoring it as a real state risks grouping genomes by shared undersampling instead of shared ancestry. `?` rather than `-` because `-` still carries gap/indel semantics in these tools, and a non-detected family is not an observed deletion. Combine with `--min-shared-family`/`--family-overlap` above: `--free-loss` removes the false signal from non-detection, but a genome left with too little real overlap with everything else will still be placed unstably — excluding it is the other half of the fix.
### Exports
All three exports reuse the `--sankoff` calibrated matrix and pseudo-alignment, recoded for the target tool:
@@ -94,4 +105,4 @@ All three exports reuse the `--sankoff` calibrated matrix and pseudo-alignment,
## Output files
With `-o/--output PREFIX`, the relevant subset of the following files is written: `<prefix>_dist.csv`, `<prefix>_shared.csv`, `<prefix>_nj.nwk`, `<prefix>_upgma.nwk`, `<prefix>_siblings.csv`, `<prefix>_rawsnp.csv`, `<prefix>_rawsnp_counts.csv`, `<prefix>_snp.fasta`, `<prefix>_sankoff_matrix.csv`, `<prefix>_sankoff_params.yaml`, `<prefix>_sankoff.fasta`, `<prefix>_sankoff.tnt`, `<prefix>_sankoff.tcm`, `<prefix>_sankoff.pg`, `<prefix>_iqtree.model`, `<prefix>_iqtree.fasta`. Without `-o`, only the plain `--metric` distance matrix is produced, on stdout.
With `-o/--output PREFIX`, the relevant subset of the following files is written: `<prefix>_dist.csv`, `<prefix>_shared.csv`, `<prefix>_nj.nwk`, `<prefix>_upgma.nwk`, `<prefix>_siblings.csv`, `<prefix>_rawsnp.csv`, `<prefix>_rawsnp_counts.csv`, `<prefix>_snp.fasta`, `<prefix>_family_overlap.csv`, `<prefix>_sankoff_matrix.csv`, `<prefix>_sankoff_params.yaml`, `<prefix>_sankoff.fasta`, `<prefix>_sankoff.tnt`, `<prefix>_sankoff.tcm`, `<prefix>_sankoff.pg`, `<prefix>_iqtree.model`, `<prefix>_iqtree.fasta`. Without `-o`, only the plain `--metric` distance matrix is produced, on stdout.