Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
This commit is contained in:
1 parent
9da0876ee6
commit
e715528d34
19 files changed
+582
-712
No files matched your search
+9
-125
@@ -17,131 +17,17 @@ obikmer phylo INDEX [OPTIONS]
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--distance` | `jaccard` | See the two tables below for the full list of accepted values |
|
||||
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see below). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data (see "Automatic $\alpha$ estimation" below). No effect on the other values; rejected if given together with a value that doesn't support it |
|
||||
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
|
||||
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
|
||||
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
|
||||
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index metric, not a `snp-*` value |
|
||||
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index distance, not a `snp-*` value |
|
||||
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
|
||||
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
|
||||
| `-o, --output` | none (stdout) | Output file prefix |
|
||||
|
||||
Every value routes to one of two independent computations:
|
||||
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
||||
|
||||
### Whole-index metrics
|
||||
|
||||
| Value | Definition |
|
||||
|---|---|
|
||||
| `jaccard` | $D = 1 - \dfrac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}$ over the sets of kmers present in each genome |
|
||||
| `mash` | derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$ |
|
||||
| `hamming` | number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$ |
|
||||
| `bray-curtis` | $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts |
|
||||
| `relfreq-bray-curtis` | the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts |
|
||||
| `euclidean` | $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts |
|
||||
| `relfreq-euclidean` | the same formula on relative frequencies |
|
||||
| `hellinger` | $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$ |
|
||||
| `hellinger-euclidean` | the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$ |
|
||||
|
||||
`hamming` requires a presence/absence index; the others work on either index type.
|
||||
|
||||
### `snp-*` corrections
|
||||
|
||||
Computed from the central-position SNP model (see "Central-position SNP model" below): a family is the set of up to 4 kmers sharing identical flanking sequence and differing only at the central base. These values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||
|
||||
For a genome pair, let $L$ be its total number of eligible loci (both genomes single-copy at that family), $p$ the raw proportion of substitutions among those loci, $P$/$Q$ the transition/transversion proportions, $Q_1$/$Q_2$ Kimura's two transversion categories (A↔C & G↔T vs. A↔T & C↔G), $P_1$/$P_2$ the purine (A↔G) / pyrimidine (C↔T) transition proportions, and $\pi_A,\pi_C,\pi_G,\pi_T$ the pair's pooled base frequencies.
|
||||
|
||||
**`snp-raw`**
|
||||
|
||||
$$d = p$$
|
||||
|
||||
**`snp-jc`**
|
||||
|
||||
$$d = -\frac{3}{4}\ln\!\left(1-\frac{4p}{3}\right)$$
|
||||
|
||||
**`snp-k2p`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
a_1 &= 1-2P-Q \\
|
||||
a_2 &= 1-2Q \\
|
||||
d &= -\frac{1}{2}\ln a_1-\frac{1}{4}\ln a_2
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-k81`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
a_1 &= 1-2P-2Q_1 \\
|
||||
a_2 &= 1-2P-2Q_2 \\
|
||||
a_3 &= 1-2Q_1-2Q_2 \\
|
||||
d &= -\frac{1}{4}\left(\ln a_1+\ln a_2+\ln a_3\right)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-f81`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
E &= 1-\left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right) \\
|
||||
d &= -E\ln\!\left(1-\frac{p}{E}\right)
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-t92`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
g &= \pi_C+\pi_G \\
|
||||
w &= 2g(1-g) \\
|
||||
a_1 &= 1-\frac{P}{w}-Q \\
|
||||
a_2 &= 1-2Q \\
|
||||
d &= -w\ln a_1-\frac{1}{2}(1-w)\ln a_2
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-tn93`**
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
g_R &= \pi_A+\pi_G \\
|
||||
g_Y &= \pi_C+\pi_T \\
|
||||
k_1 &= \frac{2\pi_A\pi_G}{g_R} \\
|
||||
k_2 &= \frac{2\pi_C\pi_T}{g_Y} \\
|
||||
k_3 &= 2\left(g_Rg_Y-\frac{\pi_A\pi_G\,g_Y}{g_R}-\frac{\pi_C\pi_T\,g_R}{g_Y}\right) \\
|
||||
w_1 &= 1-\frac{P_1}{k_1}-\frac{Q}{2g_R} \\
|
||||
w_2 &= 1-\frac{P_2}{k_2}-\frac{Q}{2g_Y} \\
|
||||
w_3 &= 1-\frac{Q}{2g_Rg_Y} \\
|
||||
d &= -k_1\ln w_1-k_2\ln w_2-k_3\ln w_3
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
**`snp-tv`** — transversions only, deliberately uncorrected:
|
||||
|
||||
$$d = Q$$
|
||||
|
||||
`--gamma-shape ALPHA` applies to every value above except `snp-raw` and `snp-tv`: each $-\ln(x)$ term in the formulas above is replaced by $\alpha\left(x^{-1/\alpha}-1\right)$ (the same weight, same $x$).
|
||||
|
||||
### Automatic $\alpha$ estimation (`--gamma-shape auto`)
|
||||
|
||||
`--gamma-shape auto` (or the equivalent `--gamma-shape estimate`) fits $\alpha$ from the index itself instead of requiring a user-supplied value, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.
|
||||
|
||||
The estimator pools substitution counts by **partition** rather than by genome pair: for partition $i$, let $n_i$ be the total number of substitutions observed across every genome pair, and $L_i$ the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:
|
||||
|
||||
$$R_i = \frac{n_i}{L_i}$$
|
||||
|
||||
Under a single shared substitution rate with no among-site heterogeneity, each $R_i$ would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as elsewhere in this correction, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:
|
||||
|
||||
$$\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}$$
|
||||
|
||||
where $\mu$ is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own $L_i$ removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:
|
||||
|
||||
$$\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}}$$
|
||||
|
||||
$$\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}$$
|
||||
|
||||
If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), $\alpha$ is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged. When an estimate is produced, it's logged at the `info` level before the distance matrix is computed.
|
||||
|
||||
Note: this is a method-of-moments estimator derived from the standard Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition — it is not part of Jin & Nei's (1990) original publication, which only defines the `+Γ` distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ (`--gamma-shape 1`) rather than proposing a way to estimate it from data. `alpha < 1` indicates strong among-site rate heterogeneity (many near-invariant loci, a few fast ones); `alpha` growing large makes the correction converge to the uncorrected formula.
|
||||
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||
|
||||
### Output
|
||||
|
||||
@@ -152,13 +38,13 @@ Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the fir
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--exclude-genome LABEL` | Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by `--pseudo-alignment`/`--sankoff`/a `snp-*` `--distance` value. Does not change the value computed for any remaining pair |
|
||||
| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` — never to the whole-index metrics or their matrix/NJ/UPGMA output |
|
||||
| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` — never to the whole-index distances or their matrix/NJ/UPGMA output |
|
||||
|
||||
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
|
||||
|
||||
## Central-position SNP model
|
||||
|
||||
Requires the sibling annex, built once per index:
|
||||
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index:
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
@@ -169,8 +55,6 @@ Requires the sibling annex, built once per index:
|
||||
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
||||
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--subsample N` |
|
||||
|
||||
A family is eligible for a genome pair $(i,j)$ only if both genomes carry exactly one of its observed forms (single-copy, unambiguous).
|
||||
|
||||
### `--sibling-stats`
|
||||
|
||||
`<prefix>_siblings.csv` — family size = number of distinct central bases observed at a family (1-4).
|
||||
@@ -219,7 +103,7 @@ A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sa
|
||||
|
||||
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
||||
|
||||
Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead, drawn in proportion to how many candidate families each part of the index actually holds, so the sample stays representative of the whole index. If the index has fewer than `N` candidate families, `--subsample` has no effect.
|
||||
Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If the index has fewer than `N` candidate families, `--subsample` has no effect.
|
||||
|
||||
### `--shannon`: measuring how informative a family is
|
||||
|
||||
@@ -238,7 +122,7 @@ Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of
|
||||
|
||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||
|
||||
By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff. The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||
By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||
|
||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
||||
|
||||
@@ -257,7 +141,7 @@ The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays
|
||||
|
||||
### The 16-state model
|
||||
|
||||
Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). Calibration combines a $5 \times 5$ transition matrix over family cardinality (0-4 observed forms) and a $4 \times 4$ base-substitution matrix from unambiguous single-copy loci, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`, into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
|
||||
Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). See [The 16-state Sankoff model](../theory/phylogeny/snp_based/sankoff_model.md) for how the calibrated cost matrix is built from cardinality and base-composition transitions.
|
||||
|
||||
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character). It does not run any external tool.
|
||||
|
||||
|
||||
Reference in new issue
Block a user