OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

large refactoring

Browse Source
This commit is contained in:
Eric Coissac
2026-08-17 09:28:53 +02:00
parent 49e66f16a2
commit 7bac0f3850
236 changed files with 77584 additions and 3733 deletions
Download Patch File Download Diff File Expand all files Collapse all files
UserDocMD/usage/phylo.md
+45
View File
@@ -70,6 +70,50 @@ A family is eligible for a genome pair $(i, j)$ only if genome $i$ carries exact
`--min-shared-family N` automates the fix: it excludes, before any computation, every genome whose mean shared-family count against all other genomes (the same statistic, averaged per row of the `--family-overlap` matrix) falls below `N`. There is no universal value for `N` — it depends on how divergent and how completely covered the genome collection is; inspect `--family-overlap`'s own output to find where the real gap sits before choosing a threshold.
## Sampling at scale: `--subsample`, `--shannon`, `--entropy`
On a large index (billions of families), building a full pseudo-alignment or fully calibrating the Sankoff model is not just slow — it may not fit in the time you have. `--subsample` bounds the work to a fixed number of families; `--shannon` reports how informative each family is; `--entropy`/`--entropy-sd` bias which families get kept toward the informative ones instead of choosing uniformly at random.
| Option | Default | Description |
|---|---|---|
| `--subsample N` | none (keep everything) | Cap the number of variable families (family size ≥ 2) retained, to approximately `N` |
| `--shannon` | off | Write `<prefix>_shannon.csv`: per-family Shannon entropy, one row per family |
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
`--subsample`/`--entropy`/`--entropy-sd` affect every option that scans variable families: `--snp`, `--family-overlap`, `--shannon`, and the whole Sankoff pipeline (`--sankoff`/`--tnt`/`--phyg`/`--iqtree`, next section) — all of them draw from the *same* selection of families in one invocation, so the Sankoff calibration and the pseudo-alignment it calibrates always describe the same sites, and `--family-overlap`'s counts stay consistent with `--snp`'s columns. `--raw-snp-distance`/`--raw-snp-counts` are not affected — they always scan every family, since their p-distance estimate is a whole-index statistic, not something that benefits from being restricted to a sample.
### `--subsample N`
Without `--subsample`, every variable family (family size ≥ 2, i.e. every family where at least one genome differs from the rest) is used. With `--subsample N`, roughly `N` families are kept instead, chosen at random but in proportion to how many candidate families each part of the index actually holds — so the sample stays representative of the whole index, not skewed toward whichever part happens to be scanned first. If the index has fewer than `N` candidate families in the first place, `--subsample` has no effect: everything is kept.
`--subsample` trades completeness for speed: `--snp`'s alignment gets fewer columns, `--sankoff`'s calibration is based on fewer observations, but the resolution work (the expensive part of a `phylo` run on a large index) scales with `N` instead of with the index's true size. Pick `N` as large as your time budget allows — a few hundred thousand to a few million families is usually enough for the transition-probability estimates in `--sankoff`'s calibration to stabilize; a smaller `N` speeds up exploratory runs.
### `--shannon`: measuring how informative a family is
Not every variable family is equally useful for a tree: a family that differs in only one genome out of a thousand carries very little signal, and one where the pattern looks essentially random across genomes may be too saturated (multiple substitutions have overwritten the original signal) to carry real information either. `--shannon` quantifies this with the Shannon entropy (in bits) of each family's states across the genomes that carry it — low entropy means "almost everyone agrees" (an invariant or near-invariant family, phylogenetically shallow), while entropy near the ceiling for a 4-state character means "close to a random draw between the possible bases" (saturated).
`<prefix>_shannon.csv` (or `shannon.csv` without `-o`) has one row per family visited:
| Column | Meaning |
|---|---|
| `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
| `family_idx` | the family's position within that layer |
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}` — the same alphabet `--sankoff`'s 16-state model uses), genomes absent from the family excluded from the count |
| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison — a genome carrying more than one base at once counts once per base, so this can differ from `entropy15` |
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2–4, since monomorphic families aren't visited) |
| `n_genomes_present` | how many genomes the entropy was computed over |
Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful to inspect the entropy distribution and decide reasonable `--entropy`/`--entropy-sd` values (see below) before committing to a full run.
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
By default, `--subsample` draws families uniformly — every candidate family has the same chance of being kept, regardless of how informative it actually is. `--entropy`/`--entropy-sd` change that: instead of a uniform draw, each family's chance of being kept is weighted by how close its own entropy (the `entropy15` value `--shannon` reports) is to `MU`, using a bell-shaped (Gaussian) curve of width `SIGMA` — a family with entropy exactly `MU` is the most likely to be kept, and the chance falls off smoothly the further its entropy is from `MU`, with no hard cutoff (a few families outside the target band can still get in, just less often).
The filter activates as soon as *either* `--entropy` or `--entropy-sd` is given; whichever one you don't set defaults to `1.0`/`0.5`. It can be combined with `--subsample N` (the target count is still approximately `N`, now biased toward the entropy band instead of uniform — expect somewhat *fewer* than `N` families in practice, since low-weight families are dropped rather than replaced) or used alone (`--entropy` without `--subsample`: a soft entropy filter over the whole index, no size target).
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy has to be computed once, up front, and is then saved alongside the index). Every following run — even with different `MU`/`SIGMA` values — reuses that saved data and stays fast.
## Sankoff calibration and phylogenetic exports
| Option | Default | Description |
@@ -129,6 +173,7 @@ Matrix layout (`_dist.csv`, `_shared.csv`, and every other "CSV matrix" below):
| `<prefix>_rawsnp_counts.csv` | `--raw-snp-counts` | CSV table | per-pair diagnostic counts behind `_rawsnp.csv` |
| `<prefix>_snp.fasta` | `--snp` | FASTA | SNP-only pseudo-alignment, IUPAC-coded |
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
| `<prefix>_shannon.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
**`_siblings.csv`** — family size = number of distinct central bases observed at a family (1–4), not "sibling count" (0–3).