Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
## `--exclude-genome`, `--min-shared-family`
| Option | Description |
|---|---|
| `--exclude-genome LABEL` | Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by `--pseudo-alignment`/`--sankoff`/a `snp-*``--distance` value. Does not change the value computed for any remaining pair |
| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*``--distance` — never to the whole-index distances or their matrix/NJ/UPGMA output |
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index:
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*``--distance` value |
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
`<prefix>_siblings.csv` — family size = number of distinct central bases observed at a family (1-4).
| Column | Meaning |
|---|---|
| `genome` | genome label, or the literal `global` for the last row |
| `1`, `2`, `3`, `4` | for a genome row: number of families of that size where the genome carries ≥ 1 member. For the `global` row: the actual deduplicated family-size histogram — not the sum of the rows above |
### Family Overlap
`--family-overlap` writes `<prefix>_family_overlap.csv`: header `genome,<label1>,<label2>,...`, one row per genome, cell `[i][j]` = number of variable families (family size ≥ 2) where both genome `i` and genome `j` carry a call. The diagonal is always `0`. Every genome is written, unfiltered by `--exclude-genome`/`--min-shared-family`.
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
### `--pseudo-alignment`
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
`--subsample`, `--free-loss`, `--no-ambiguity`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*``--distance` value — one draw feeds all of them in a single invocation. `--subsample` is mandatory for `--pseudo-alignment`/`--sankoff`; for a `snp-*``--distance` value it is optional (omitted means every non-monomorphic family in the index, not an approximation).
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*``--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
Every invocation, with or without `--session`, draws its own fresh random sample by default — running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately. `--session` does not change this: it makes a *specific* sample reusable on request, it does not make sampling itself reproducible from one independent run to the next.
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--subsample`/`--free-loss`/`--no-ambiguity`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
A `--sankoff`-family run and a `snp-*``--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If the index has fewer than `N` candidate families, `--subsample` has no effect.
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison |
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful for choosing `--entropy`/`--entropy-sd` values before a full run.
By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment. Requires `--subsample N` |
| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the base-composition part of the calibration |
| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, throughout `--sankoff` and every export built from it |
Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). See [The 16-state Sankoff model](../theory/phylogeny/snp_based/sankoff_model.md) for how the calibrated cost matrix is built from cardinality and base-composition transitions.
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character). It does not run any external tool.
With `--free-loss`, the empty state is recoded to `?` (TNT/PhyG/IQ-TREE's own missing-data symbol) instead of an ordinary, costed 16th state — `?` rather than `-`, since `-` still carries gap/indel semantics in these tools. `--free-loss` also zeroes the cardinality-transition cost between any two states, not just to/from the empty one: gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family.
All three exports reuse the `--sankoff` calibrated matrix and pseudo-alignment, recoded for the target tool:
- **`--tnt`**: a self-contained TNT script (alignment recoded to TNT's fixed 16-symbol alphabet, integer-scaled cost matrix re-closed to a metric, a default search block).
- **`--phyg`**: a custom cost-matrix file plus a PhyG script reusing the `--sankoff` alignment directly.
- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths. Only states actually occurring in the alignment are kept and compactly renumbered.
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
**`_sankoff_matrix.csv`** — header `state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N`: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), `0` standing for the empty/absent state. One row per source state, one value per destination state, cost $-\ln P(a,b)$, 4 decimals.
**`_sankoff.fasta`** — recoded to match `_sankoff_matrix.csv`'s alphabet: absent state is `0` (or `?` under `--free-loss`). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.
**`_sankoff.tcm`** (`--phyg`) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state `0`).
**`_sankoff.pg`** (`--phyg`) — script: `read(prefasta:..., tcm:...)` against `_sankoff.fasta`/`_sankoff.tcm`, a default 300s/4-instance `search`, `report(...)` writing `<prefix>_sankoff.tre`. Run with `phyg` from the output directory (the script uses relative file names).
**`_iqtree.model`** (`--iqtree`) — lower-triangular exchangeability matrix $R(a,b) = e^{-\text{cost}(a,b)}$ (one row of increasing length per state, whitespace-separated, PAML order), followed by one line of empirical state frequencies. Only states actually occurring in the alignment are kept, compactly renumbered `0..k-1`.
**`_iqtree.fasta`** (`--iqtree`) — alignment recoded to that same compact `0..k-1` alphabet (symbols `0-9A-F`). Under `--free-loss`, non-detection becomes `?` and columns left non-informative once missing calls are ignored are dropped first (required for `+ASC`); with `--iqtree-min-freq` also set (the default), any state rarer than that threshold is folded into the same `?` treatment, and non-informative columns are re-checked and dropped again. Run with:
**`_iqtree_states.csv`** (`--iqtree`) — one row per state actually kept in `_iqtree.model`/`_iqtree.fasta` (header `iqtree_symbol,canonical_symbol,frequency`): `iqtree_symbol` is the compact `0-9A-F` symbol as written in those two files, `canonical_symbol` is the matching `_sankoff_matrix.csv` state, `frequency` is that state's empirical frequency at full precision. Under `--free-loss`, absent (`0`/`?`) is never a kept state, so it never appears here — nor does any state `--iqtree-min-freq` folded away for being too rare.
States that combine 3 or 4 central bases at once (IUPAC `V`/`H`/`K`.../`N`) are inherently rare, and can make `iqtree3` itself numerically unstable ("Numerical underflow for lh-derivative" warnings). With `--free-loss` set, `--iqtree-min-freq` (default `0.001`, one in a thousand) extends the missing-data treatment to any state below this frequency, not just absence. Check `_iqtree_states.csv` to see exactly which states survived and at what frequency; set `--iqtree-min-freq 0` to keep every state that occurs at all. Has no effect without `--free-loss`.