obikmer/UserDocMD/usage/phylo.md

phylo

Compute pairwise evolutionary-distance proxies between the genomes stored in an index — a plain distance matrix, optionally trees (NJ/UPGMA), and optionally a central-position SNP model with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).

obikmer phylo INDEX [OPTIONS]

Arguments

Argument	Description
INDEX	Index directory

Distance matrix

Option	Default	Description
--metric	jaccard	One of jaccard, mash, hamming, bray-curtis, relfreq-bray-curtis, euclidean, relfreq-euclidean, hellinger, hellinger-euclidean
--presence-threshold	1	Minimum count for a kmer to be considered present, for Jaccard/Mash on a count index
--shared-kmers	off	Also write the shared-kmer count matrix
--nj	off	Compute and write a Neighbor-Joining tree (Newick)
--upgma	off	Compute and write a UPGMA tree (Newick)
-o, --output	none (stdout)	Output file prefix; without it, the distance matrix is printed to stdout as CSV

hamming requires a presence/absence index. All other metrics work on either index type; on a presence index, jaccard/mash/hamming are the only ones available.

Metric definitions

• jaccard: D = 1 - \dfrac{|A \cap B|}{|A \cup B|} over the sets of kmers present in each genome.
• mash: derived from the Jaccard distance via D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right) where J = 1 - D_{\text{jaccard}} and k is the index's kmer size; clamped to 1.0 when J \le 0.
• hamming: number of kmer positions where presence differs between the two genomes (presence index only, not normalized): D = \sum_i \mathbb{1}[a_i \ne b_i].
• bray-curtis: D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B} on raw per-kmer counts.
• relfreq-bray-curtis: the same formula computed on per-genome relative frequencies p_i = c_i / \sum_j c_j instead of raw counts.
• euclidean: D = \sqrt{\sum_i (c_i^A - c_i^B)^2} on raw counts.
• relfreq-euclidean: the same formula on relative frequencies.
• hellinger: D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2} on relative frequencies, bounded in [0, 1].
• hellinger-euclidean: the unnormalized variant, D = \sqrt{2} \times D_{\text{hellinger}}.

Central-position SNP model

This is a separate operation from the distance-matrix computation above: if any option below is used, no --metric matrix is computed in the same invocation.

A family is the set of up to 4 kmers that share identical flanking sequence and differ only at the exact central base. Because k is odd, the central position is well defined and maps to itself under reverse complementation. All computations below first require building the sibling annex, an index-wide record of which of the 4 possible central bases are observed at each family, across every genome.

Option	Default	Description
--sibling-annex	off	Build the sibling presence-mask annex (prerequisite for every option below)
--exclude-genome LABEL	none	Exclude a genome (repeatable) from every SNP/Sankoff/export computation below
--min-shared-family N	none	Auto-exclude any genome whose mean shared-family count against every other genome (see --family-overlap) falls below N — same exclusion as --exclude-genome, applied on top of it
--sibling-stats	off	Write the family-size (sibling count) distribution, per genome and globally
--raw-snp-distance	off	Write the single-copy central-SNP p-distance matrix
--raw-snp-counts	off	Write per-pair diagnostic counts (n_snp, n_shared, n_eligible) instead of a matrix
--snp	off	Write a SNP-only pseudo-alignment in FASTA, IUPAC-coded
--family-overlap	off	Write an NxN matrix of, for each genome pair, how many variable families both genomes actually carry a call for; the diagonal holds each genome's own total family count

Locus eligibility

A family is eligible for a genome pair (i, j) only if genome i carries exactly one of the family's observed forms (single-copy, unambiguous) and genome j also carries exactly one. A genome carrying more than one form at a locus makes that locus ineligible for any pair involving it.

--raw-snp-distance tallies, over every eligible locus of every genome pair, n_{\text{snp}} (the two genomes' single forms differ) versus n_{\text{shared}} (they agree — this includes invariant families). The output ratio is \hat{p} = \dfrac{n_{\text{snp}}}{n_{\text{snp}} + n_{\text{shared}}}.

--snp restricts itself to variable families (family size \ge 2) and writes one FASTA record per genome, one column per family, IUPAC-coded from each genome's presence mask at that family (a single form → the plain base; several forms → the matching IUPAC ambiguity code; no form → -).

--exclude-genome removes a genome from these computations, re-checking column variability among the remaining genomes so that a column made monomorphic by the exclusion is dropped rather than kept artificially. It does not affect the --metric distance-matrix path.

Family overlap and low-coverage genomes

--family-overlap writes, for every genome pair, how many variable families both genomes actually carry a call for (neither is absent) — a direct measure of how much informative content two genomes actually share. On genome-skim or otherwise incomplete-coverage collections, a genome with very little overlap with everything else has almost nothing left to constrain its position in a tree, and tends to end up placed unstably (near-zero branch length, grafted inside an unrelated clade) by --tnt/--iqtree.

--min-shared-family N automates the fix: it excludes, before any computation, every genome whose mean shared-family count against all other genomes (the same statistic, averaged per row of the --family-overlap matrix) falls below N. There is no universal value for N — it depends on how divergent and how completely covered the genome collection is; inspect --family-overlap's own output to find where the real gap sits before choosing a threshold.

Sankoff calibration and phylogenetic exports

Option	Default	Description
--sankoff	off	Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment
--sankoff-ratio-ceiling	0.5	Exclude genome pairs whose raw SNP ratio exceeds this value from the calibration
--free-loss	off	Recode a family's non-detection as the ? missing-data symbol instead of an ordinary, costed state, in --sankoff's pseudo-alignment and every export built from it
--tnt	off	Also write a TNT script (implies --sankoff)
--phyg	off	Also write PhyG input files (implies --sankoff)
--iqtree	off	Also write an IQ-TREE custom model and alignment (implies --sankoff)
--sankoff-cost-scale	100	Integer scaling factor applied to costs before rounding (required by TNT/PhyG's integer-only cost commands)

The 16-state model

Each family is treated as a character with 16 possible states: one per subset of the 4 possible central bases actually observed (including the empty subset). Calibration combines two tallies, both restricted to genome pairs at or below --sankoff-ratio-ceiling:

• a 5 \times 5 transition matrix over family cardinality (0–4 observed forms) between paired genomes, and
• a 4 \times 4 base-substitution transition matrix from unambiguous single-copy loci,

which are combined into a row-normalized 16 \times 16 transition probability matrix P, converted to a symmetric cost matrix via \text{cost}(a,b) = -\ln P(a,b).

--sankoff alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol 0 (never a gap character, to avoid ambiguity with external tools' own gap semantics). It does not run any external tool.

With --free-loss, the empty state is recoded to ? instead — TNT/PhyG/IQ-TREE's own missing-data symbol — rather than an ordinary, costed 16th state. This matters for genome-skim or otherwise incomplete-coverage collections, where non-detection of a family is dominated by sampling failure rather than true evolutionary loss: scoring it as a real state risks grouping genomes by shared undersampling instead of shared ancestry. ? rather than - because - still carries gap/indel semantics in these tools, and a non-detected family is not an observed deletion. Combine with --min-shared-family/--family-overlap above: --free-loss removes the false signal from non-detection, but a genome left with too little real overlap with everything else will still be placed unstably — excluding it is the other half of the fix.

Exports

All three exports reuse the --sankoff calibrated matrix and pseudo-alignment, recoded for the target tool:

• --tnt: a self-contained TNT script (alignment recoded to TNT's fixed 16-symbol alphabet, integer-scaled cost matrix re-closed to a metric, a default search block).
• --phyg: a custom cost-matrix file plus a PhyG script reusing the --sankoff alignment directly.
• --iqtree: a custom substitution-model file (exchangeability matrix recovered as R(a,b) = e^{-\text{cost}(a,b)}, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths (unlike the parsimony step-counts from TNT/PhyG). Only states actually occurring in the alignment are kept and compactly renumbered.

Output files

With -o/--output PREFIX, the relevant subset of the following files is written: <prefix>_dist.csv, <prefix>_shared.csv, <prefix>_nj.nwk, <prefix>_upgma.nwk, <prefix>_siblings.csv, <prefix>_rawsnp.csv, <prefix>_rawsnp_counts.csv, <prefix>_snp.fasta, <prefix>_family_overlap.csv, <prefix>_sankoff_matrix.csv, <prefix>_sankoff_params.yaml, <prefix>_sankoff.fasta, <prefix>_sankoff.tnt, <prefix>_sankoff.tcm, <prefix>_sankoff.pg, <prefix>_iqtree.model, <prefix>_iqtree.fasta. Without -o, only the plain --metric distance matrix is produced, on stdout.