Skip to content

phylo

Compare the genomes stored in an index and build phylogenetic trees from them. Two complementary approaches are available:

  • Whole-genome distances compare the k-mer content of the genomes, without any alignment.
  • SNP-based distances and analyses compare the nucleotide variants found at the central position of related k-mers.

Distance matrices can be used to build Neighbor-Joining or UPGMA trees. SNP-based analyses can also produce a pseudo-alignment and calibrated substitution costs, with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).

Usage

obikmer phylo INDEX [OPTIONS]

Arguments

Argument Description
INDEX Index directory

General options

Option Default Description
--cpu-max N all available cores Maximum number of CPU cores to use. It caps every thread pool of the process and can only lower the number of cores, never raise it above those available. Common to all subcommands
-o, --output PREFIX none Prefix of the output files. Without it, the distance matrix goes to standard output and every other file is written in the current directory under its name without the prefix. See "Output files" below

SNP families

SNP-based analyses use groups of k-mers that differ only at their central nucleotide. These groups are called families (see the central-position SNP model for the exact definitions and eligibility rules). A variable family is one character of the analysis, and the variants a genome carries give its state for that character.

Distance estimation and tree reconstruction

Option Default Description
--distance none Compute and write a distance matrix. The accepted values are listed just below the table. No matrix is computed unless this is given, or --nj or --upgma is (they need a matrix and then use jaccard)
--gamma-shape ALPHA\|auto none Rate-heterogeneity correction, for snp-* values that support it (see Gamma rate heterogeneity). Either a fixed \(\alpha\) or auto/estimate to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it
--presence-threshold 1 How counts are turned into presence/absence on an index that stores counts: a kmer is present in a genome only when its count reaches this value. Used by jaccard, mash and hamming, and by the shared-kmer counts of --shared-kmers. No effect on an index without counts, nor on the distances that use the counts themselves
--session DIR, --session-force none Save and reuse the sample of a snp-* distance. Currently unavailable, see --session below
--csv off Write the matrix as plain CSV instead of the default relaxed-PHYLIP format
--shared-kmers off Write the shared-kmer count matrix (<prefix>_shared.csv). It can be used alone, without any distance, or together with a whole-index distance; it is refused with a snp-* value
--nj off Compute and write a Neighbor-Joining tree (Newick)
--upgma off Compute and write a UPGMA tree (Newick)

Every value routes to one of two independent computations: whole-index distances (jaccard, mash, hamming, bray-curtis, relfreq-bray-curtis, euclidean, relfreq-euclidean, hellinger, hellinger-euclidean — see Whole k-mer set distances for the definitions; jaccard, mash and hamming compare which kmers are present and work on either index type; the six others (bray-curtis, relfreq-bray-curtis, euclidean, relfreq-euclidean, hellinger, hellinger-euclidean) use the counts and need an index that stores them), or snp-* corrections (snp-raw, snp-jc, snp-k2p, snp-k81, snp-f81, snp-t92, snp-tn93, snp-tv — see the central-position SNP model for the definitions).

snp-* values read the sibling annex (below), which is built automatically the first time it is needed, and are, by default, computed exhaustively over every non-monomorphic family in the index; add --n-sites N to bound the computation to approximately N families instead (see "SNP character selection and sampling" below — the same option --pseudo-alignment/--sankoff use, but optional here).

Matrix format

When a matrix is computed and there is no -o, it goes to stdout in relaxed-PHYLIP format (n on the first line, then one label<TAB>value... row per genome). With --csv, the format is instead a header row genome,<label1>,<label2>,... followed by one <label>,<value1>,<value2>,... row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.

--session: currently unavailable

--session DIR and --session-force are meant to save the sample of a snp-* --distance run in a directory, so that a later run with identical selection parameters reuses it instead of drawing again. This feature is being rebuilt and is not available for now: giving either option stops the command with an error. Until it returns, keep the output files of a run (the alignment, or the distance matrix) rather than relying on re-drawing the same sample.

Genome selection and filtering

Option Description
--exclude-genome LABEL Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by --pseudo-alignment/--sankoff/a snp-* --distance value. Does not change the value computed for any remaining pair
--min-shared-family N Auto-exclude, on top of --exclude-genome, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below N. Applies only to --pseudo-alignment/--sankoff/snp-* --distance — never to the whole-index distances or their matrix/NJ/UPGMA output

Neighbor-Joining and UPGMA trees (--nj/--upgma) are always built from every genome in the index, regardless of --exclude-genome/--min-shared-family.

SNP character selection and sampling

SNP-based analyses can use every eligible variable family, or a sample drawn from the index. The options below control which characters are selected, which genomes and states are kept, and how the sample is spread over the index.

--n-sites, --free-loss, --no-ambiguity, --quartet, --pair-balance, --entropy/--entropy-sd are shared by --pseudo-alignment, --sankoff (and everything it implies: --tnt/--phyg/--iqtree), and a snp-* --distance value — one draw feeds all of them in a single invocation. --n-sites is mandatory for --pseudo-alignment/--sankoff, where it takes either a number of sites or all; for a snp-* --distance value it is optional, and omitting it is the same as all (every non-monomorphic family in the index, not an approximation).

Combining --sankoff (or --tnt/--phyg/--iqtree) with a snp-* --distance value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the identical set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate obikmer phylo invocations draws two independent samples even with the same flags.

Every invocation draws its own fresh random sample: running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately.

With --n-sites all (or, for a snp-* --distance value, without --n-sites), every variable family (family size ≥ 2) that passes the other filters is used. With --n-sites N, roughly N families are kept instead (see Sampling design for how the draw stays representative of the whole index). If fewer than N candidate families remain, --n-sites has no effect.

Option Default Description
--n-sites N\|all none (mandatory for --pseudo-alignment and --sankoff) Number of families to draw, or all to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee.
--free-loss off Treat a genome carrying none of a family's observed members as missing data (?) instead of -
--no-ambiguity off Treat a genome carrying more than one member of a family as missing data (?) instead of an IUPAC ambiguity code
--only-diploid off Milder than --no-ambiguity (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as R) and treat those carrying three or four as missing data (?)
--quartet off Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless --free-loss is given. Which genomes count as carriers follows --no-ambiguity
--pair-balance off Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows --no-ambiguity
--sample-stats off Write three CSV files describing what the draw contains (see "Describing a draw" below)
--sample-plan off Write <prefix>_sample_plan.csv: how the draw was spread over the layers of the index
--entropy MU off (1.0 if only --entropy-sd is given) Center of the entropy band to favor when sampling
--entropy-sd SIGMA off (0.5 if only --entropy is given) Width of that band

--quartet: keeping only families that can resolve a tree

A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (AB|CD) it takes at least two distinct states, each carried by at least two genomes. --quartet restricts the draw to the families that show this pattern. The states are those of the alignment. Without --free-loss, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With --free-loss, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless --no-ambiguity is given, in which case it is left out, as in the alignment.

The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time --quartet is used with a given --no-ambiguity / --only-diploid convention and with or without --free-loss, in one pass shared with the other annexes, or ahead of time by --sibling-annex --quartet), then checked exactly on each family drawn, taking --exclude-genome into account. The quota given to --n-sites is therefore spent only on usable families.

--pair-balance: representing every pair of genomes

Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. --pair-balance compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix --family-overlap writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.

Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time --pair-balance is used with a given --no-ambiguity / --only-diploid convention, after the family-overlap matrix it depends on, or ahead of time by --sibling-annex --pair-balance). They combine with --entropy and --quartet: the weight multiplies the entropy kernel, and --quartet restricts the families first.

--entropy MU / --entropy-sd SIGMA: biasing the sample toward informative families

By default, --n-sites N draws families uniformly. With --entropy/--entropy-sd, each family's chance of being kept is instead weighted by how close its own entropy (entropy15) is to MU, using a Gaussian curve of width SIGMA — no hard cutoff (see Sampling design for the weighting formula). The filter activates as soon as either flag is given; the other defaults to 1.0/0.5. Combine with --n-sites N (expect somewhat fewer than N families kept in practice) or use alone (a soft filter over the whole index, no size target).

The first phylo run on a given index that uses --entropy/--entropy-sd pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different MU/SIGMA, reuse that saved data.

The entropy that weights the draw follows --no-ambiguity and --only-diploid, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (entropy15). With --no-ambiguity, only the genomes carrying exactly one member count, since the others become missing data (entropy4). With --only-diploid, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and --sibling-annex --entropy builds the one of the convention given.

Describing a draw: --sample-stats, --sample-plan

Both need a draw to describe (--pseudo-alignment, --sankoff or a snp-* --distance) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.

--sample-stats writes three files:

File Content
<prefix>_sample_genomes.csv per genome (genome,excluded,sites_with_call,share_of_sites,informative_sites): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method
<prefix>_sample_carriers.csv per number of carriers (n_carriers,sites,informative_sites): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all
<prefix>_sample_overlap.csv the sample's own genome × genome matrix, same layout as --family-overlap: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents

A call is any state other than the empty one: absence is never a call, with or without --free-loss. A site is informative under the same rule as --quartet: without --free-loss, absence is a state of its own and can form one of the two pairs; with --free-loss it is missing data and cannot. Excluded genomes are listed with zeros in the first file and count nowhere.

--sample-plan writes <prefix>_sample_plan.csv, one line per layer of the index in the order the draw visited them (partition,layer,eligible,mass,quota,resolved,kept): the variable families eligible when the draw starts, their total weight (their number, or with --pair-balance the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low kept against resolved means masking removes most of what is drawn in that layer; a quota summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.

SNP family preparation and diagnostics

SNP-based options read precomputed tables describing the families: the sibling annex, and tables derived from it (entropy, quartet, family overlap, pair weights). They are built automatically the first time they are needed and reused afterwards. The options below prepare them in advance, or describe the families.

Option Description
--sibling-annex Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the --no-ambiguity / --only-diploid convention given: --entropy the entropy annex, --quartet the quartet annex, --pair-balance the pair-weight and family-overlap annexes, --family-overlap or --min-shared-family the family-overlap annex. For example --sibling-annex --quartet --only-diploid prepares exactly the quartet annex of that convention, and --sibling-annex alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after migrate-mphf); running --sibling-annex first does the same work once, which is the safe way to start several phylo runs on the same index at the same time
--sibling-stats Write <prefix>_siblings.csv: the family-size distribution, per genome and globally
--sibling-hist Write <prefix>_sibling_hist.csv (columns size,count): the global family-size histogram (1-4 members) only, without the per-genome breakdown
--family-overlap Write <prefix>_family_overlap.csv: for every genome pair, how many variable families both genomes carry a call for
--shannon Write <prefix>_entropy.csv: per-family Shannon entropy, one row per family, full unsampled scan
--shannon-summary Write <prefix>_shannon_summary.csv: the same scan as --shannon, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index

--sibling-stats

<prefix>_siblings.csv — family size = number of distinct central bases observed at a family (1-4).

Column Meaning
genome genome label, or the literal global for the last row
1, 2, 3, 4 for a genome row: number of families of that size where the genome carries ≥ 1 member. For the global row: the actual deduplicated family-size histogram — not the sum of the rows above

Family Overlap

--family-overlap writes <prefix>_family_overlap.csv: header genome,<label1>,<label2>,..., one row per genome, cell [i][j] = number of variable families (family size ≥ 2) where both genome i and genome j carry a call. The diagonal is always 0. Every genome is written, unfiltered by --exclude-genome/--min-shared-family. With --no-ambiguity, a genome carrying several members of a family is not counted as carrying it, as in the alignment; without it, any genome with a call counts. The matrix is stored once per convention, so switching --no-ambiguity builds the other one without discarding the first (--min-shared-family follows the same convention).

--min-shared-family N uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for N — inspect --family-overlap's own output to find where the real gap sits in a given genome collection before choosing a threshold.

--shannon: measuring how informative a family is

<prefix>_entropy.csv has one row per family visited:

Column Meaning
partition the index partition that holds the family
layer an internal index-layer identifier — stable within one run, not meaningful across indexes
family_idx the family's position within that layer
entropy15 Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of {A,C,G,T}), genomes absent from the family excluded from the count
entropy4 Shannon entropy (bits) over the 4 plain bases, computed only over the genomes that carry exactly one member of the family — the genomes that remain informative under --no-ambiguity, since a genome carrying several members of a family (paralogous copies) is treated as missing data there. 0 when n_genomes_unambiguous is 0
family_size number of distinct central bases observed anywhere in the index for this family (2-4)
n_genomes_present how many genomes the family is present in; entropy15 is computed over these
n_genomes_unambiguous how many of those genomes carry exactly one member of the family; entropy4 is computed over these

--shannon always scans every non-monomorphic family of the whole index; --n-sites, --entropy and --entropy-sd have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing --entropy/--entropy-sd values, since it gives the complete entropy distribution, including how it varies with n_genomes_present, rather than an estimate from a sample.

--shannon-summary

--shannon-summary writes <prefix>_shannon_summary.csv (entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families): the number of variable families per entropy (entropy15 over every carrier, entropy4 over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (bin_start is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers --no-ambiguity loses, from the same scan as --shannon but without one line per family.

SNP pseudo-alignment

A pseudo-alignment has one column per variable family and one state per genome. Unlike a conventional sequence alignment, its columns come directly from groups of related k-mers of the index. The number of sites is set with --n-sites, and the options of "SNP character selection and sampling" above control which families are included and how ambiguous or absent states are written.

Option Description
--pseudo-alignment Write <prefix>_alignment.fasta: a SNP-only pseudo-alignment. Requires --n-sites N\|all
--fasta Choose FASTA as the pseudo-alignment format (the default when neither --fasta nor --vcf is given). Implies --pseudo-alignment
--vcf Also (or instead) write <prefix>_alignment.vcf, the same sites as a VCF. Implies --pseudo-alignment; combinable with --fasta

--pseudo-alignment

<prefix>_alignment.fasta — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → -.

VCF output (--vcf)

The same drawn sites can be written as <prefix>_alignment.vcf, with --fasta and --vcf freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the VCF pseudo-alignment page. The genotype ploidy follows --no-ambiguity (haploid) and --only-diploid (diploid, with the genomes carrying three or four forms of a family removed beforehand).

Sankoff calibration and phylogenetic exports

Option Default Description
--sankoff off Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment. Requires --n-sites N\|all
--sankoff-ratio-ceiling 0.5 Exclude genome pairs whose raw SNP ratio exceeds this value from the base-composition part of the calibration
--free-loss off Recode a family's non-detection as the ? missing-data symbol instead of an ordinary, costed state, throughout --sankoff and every export built from it
--tnt off Also write a TNT script (implies --sankoff)
--phyg off Also write PhyG input files (implies --sankoff)
--iqtree off Also write an IQ-TREE custom model and alignment (implies --sankoff)
--iqtree-min-freq 0.001 With --iqtree --free-loss: also treat as missing any state rarer than this in the alignment
--sankoff-cost-scale 100 Integer scaling factor applied to costs before rounding, for TNT/PhyG's integer-only cost commands

The 16-state model

Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). See The 16-state Sankoff model for how the calibrated cost matrix is built from cardinality and base-composition transitions.

--sankoff alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol 0 (never a gap character). It does not run any external tool.

With --free-loss, the empty state is recoded to ? (TNT/PhyG/IQ-TREE's own missing-data symbol) instead of an ordinary, costed 16th state — ? rather than -, since - still carries gap/indel semantics in these tools. --free-loss also zeroes the cardinality-transition cost between any two states, not just to/from the empty one: gaining or losing a sibling is priced the same way — for free — as gaining or losing the whole family.

Exports

All three exports reuse the --sankoff calibrated matrix and pseudo-alignment, recoded for the target tool:

  • --tnt: a self-contained TNT script (alignment recoded to TNT's fixed 16-symbol alphabet, integer-scaled cost matrix re-closed to a metric, a default search block).
  • --phyg: a custom cost-matrix file plus a PhyG script reusing the --sankoff alignment directly.
  • --iqtree: a custom substitution-model file (exchangeability matrix recovered as \(R(a,b) = e^{-\text{cost}(a,b)}\), plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths. Only states actually occurring in the alignment are kept and compactly renumbered.

TNT and PhyG both write trees with bare numeric leaf labels (1, 2, ..., in the order the genomes appear in <prefix>_sankoff.fasta).

Output files

With -o/--output PREFIX, the relevant subset of the files below is written. Without -o, the distance matrix goes to standard output and every other file is written in the current directory under its name without the <prefix>_ part (for example nj.nwk, entropy.csv, alignment.fasta). All matrices use genome labels as row/column headers, in index order.

Distance matrix

File Written by Format Content
<prefix>_dist.phy a distance matrix is computed, unless --csv relaxed PHYLIP the --distance matrix
<prefix>_dist.csv a distance matrix is computed, with --csv CSV matrix the --distance matrix, 6 decimals
<prefix>_shared.csv --shared-kmers CSV matrix shared-kmer count per genome pair (integers)
<prefix>_nj.nwk --nj Newick Neighbor-Joining tree
<prefix>_upgma.nwk --upgma Newick UPGMA tree

CSV matrix layout (_dist.csv, _shared.csv, _family_overlap.csv): header genome,<label1>,<label2>,..., one data row per genome, <label>,<value1>,<value2>,....

SNP analyses

File Written by Format Content
<prefix>_siblings.csv --sibling-stats CSV table family-size distribution, per genome and global
<prefix>_sibling_hist.csv --sibling-hist CSV table global family-size histogram (size,count)
<prefix>_family_overlap.csv --family-overlap CSV matrix variable families both genomes of a pair carry a call for
<prefix>_entropy.csv --shannon CSV table per-family Shannon entropy, see --shannon above
<prefix>_shannon_summary.csv --shannon-summary CSV table families per entropy bin, number of carriers and number of single-member carriers
<prefix>_sample_genomes.csv, <prefix>_sample_carriers.csv, <prefix>_sample_overlap.csv --sample-stats CSV what the draw contains: per genome, per number of carriers, per pair of genomes
<prefix>_sample_plan.csv --sample-plan CSV table per layer: eligible families, weight, quota, drawn, kept
<prefix>_alignment.fasta --pseudo-alignment or --fasta FASTA SNP-only pseudo-alignment, IUPAC-coded
<prefix>_alignment.vcf --vcf VCF Same sites as the FASTA, with stable virtual coordinates and a k-mer-based ID

Sankoff calibration and exports

File Written by Format Content
<prefix>_sankoff_matrix.csv --sankoff/--tnt/--phyg/--iqtree CSV matrix calibrated 16×16 cost matrix
<prefix>_sankoff_params.yaml same flags YAML calibration report (raw tallies + derived probabilities)
<prefix>_sankoff.fasta same flags FASTA Sankoff-recoded pseudo-alignment, header carries an n_sites annotation
<prefix>_sankoff.tnt --tnt TNT script ready-to-run parsimony search
<prefix>_sankoff.tcm --phyg PhyG TCM cost matrix in PhyG's own format
<prefix>_sankoff.pg --phyg PhyG script ready-to-run parsimony search
<prefix>_iqtree.model --iqtree IQ-TREE model file custom ML substitution model
<prefix>_iqtree.fasta --iqtree FASTA alignment recoded for that model
<prefix>_iqtree_states.csv --iqtree CSV table maps _iqtree.model/_iqtree.fasta's compact state symbols back to _sankoff_matrix.csv's alphabet

_sankoff_matrix.csv — header state,0,A,C,M,G,R,S,V,T,W,Y,H,K,D,B,N: the 16 symbols are IUPAC codes for the 16 subsets of the 4 possible central bases (bit 0=A, 1=C, 2=G, 3=T), 0 standing for the empty/absent state. One row per source state, one value per destination state, cost \(-\ln P(a,b)\), 4 decimals.

_sankoff_params.yaml:

Key Meaning
ratio_ceiling the --sankoff-ratio-ceiling value used
cardinality_transitions 5×5 list of {from, to, count, probability}, family cardinality (0-4 observed forms)
composition_transitions 4×4 list of {from, to, count, probability}, base letters A/C/G/T, single-copy substitutions

_sankoff.fasta — recoded to match _sankoff_matrix.csv's alphabet: absent state is 0 (or ? under --free-loss). Excluded genomes dropped; columns left monomorphic by that exclusion are re-checked and dropped too.

_sankoff.tnt (--tnt) — xread block (alignment recoded to TNT's fixed 0-9A-F alphabet), an integer-scaled (--sankoff-cost-scale) and metric-closed smatrix, a default hold 20; mult; export search. Run with printf 'proc <path>;\nquit;\n' | tnt. Produces <prefix>_sankoff.tre (bare numeric leaf labels, order matching _sankoff.fasta).

_sankoff.tcm (--phyg) — first line: the 16-symbol alphabet plus a trailing gap symbol (17 total). Each following line: one row of the integer-scaled, metric-closed cost matrix (17 values — the extra gap column/row reuses the cost to/from the empty state 0).

_sankoff.pg (--phyg) — script: read(prefasta:..., tcm:...) against _sankoff.fasta/_sankoff.tcm, a default 300s/4-instance search, report(...) writing <prefix>_sankoff.tre. Run with phyg from the output directory (the script uses relative file names).

_iqtree.model (--iqtree) — lower-triangular exchangeability matrix \(R(a,b) = e^{-\text{cost}(a,b)}\) (one row of increasing length per state, whitespace-separated, PAML order), followed by one line of empirical state frequencies. Only states actually occurring in the alignment are kept, compactly renumbered 0..k-1.

_iqtree.fasta (--iqtree) — alignment recoded to that same compact 0..k-1 alphabet (symbols 0-9A-F). Under --free-loss, non-detection becomes ? and columns left non-informative once missing calls are ignored are dropped first (required for +ASC); with --iqtree-min-freq also set (the default), any state rarer than that threshold is folded into the same ? treatment, and non-informative columns are re-checked and dropped again. Run with:

iqtree3 -s <prefix>_iqtree.fasta --seqtype MORPH -m <prefix>_iqtree.model+ASC --prefix <prefix>_iqtree -T AUTO

_iqtree_states.csv (--iqtree) — one row per state actually kept in _iqtree.model/_iqtree.fasta (header iqtree_symbol,canonical_symbol,frequency): iqtree_symbol is the compact 0-9A-F symbol as written in those two files, canonical_symbol is the matching _sankoff_matrix.csv state, frequency is that state's empirical frequency at full precision. Under --free-loss, absent (0/?) is never a kept state, so it never appears here — nor does any state --iqtree-min-freq folded away for being too rare.

Rare states and --iqtree-min-freq

States that combine 3 or 4 central bases at once (IUPAC V/H/K.../N) are inherently rare, and can make iqtree3 itself numerically unstable ("Numerical underflow for lh-derivative" warnings). With --free-loss set, --iqtree-min-freq (default 0.001, one in a thousand) extends the missing-data treatment to any state below this frequency, not just absence. Check _iqtree_states.csv to see exactly which states survived and at what frequency; set --iqtree-min-freq 0 to keep every state that occurs at all. Has no effect without --free-loss.