Refactor documentation for kmer indexing and phylogenetic models

This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
This commit is contained in:
Eric Coissac committed 2026-10-13 08:38:49 +02:00
1 parent 9da0876ee6
commit e715528d34
19 files changed
+582 -712

No files matched your search

@@ -0,0 +1,85 @@
# Central-position SNP model
## Family
A **family** is the set of up to 4 kmers sharing identical flanking sequence and differing only at the central base — the odd kmer length guarantees a single, well-defined central position (see [Kmers](../../kmer_indexing/kmers.md)). A family is **eligible** for a genome pair $(i,j)$ only if both genomes carry exactly one of its observed forms (single-copy, unambiguous) — this excludes multi-copy and absent loci from the comparison rather than folding them into an undifferentiated "not identical" bucket the way a whole-index distance would.
Conditioning on eligibility this way restricts every comparison to loci that are directly, positively confirmed comparable in both genomes: a locus never enters the statistic because of a genuinely absent homologous region, a diverged paralogous copy, or a genome-size asymmetry — only because a real single-copy substitution (or lack of one) was observed at flanks confirmed intact in both genomes.
## Substitution corrections (`snp-*`)
For a genome pair, let $L$ be its total number of eligible loci, $p$ the raw proportion of substitutions among those loci, $P$/$Q$ the transition/transversion proportions, $Q_1$/$Q_2$ Kimura's two transversion categories (A↔C & G↔T vs. A↔T & C↔G), $P_1$/$P_2$ the purine (A↔G) / pyrimidine (C↔T) transition proportions, and $\pi_A,\pi_C,\pi_G,\pi_T$ the pair's pooled base frequencies.
**`snp-raw`**
$$d = p$$
**`snp-jc`**
$$d = -\frac{3}{4}\ln\!\left(1-\frac{4p}{3}\right)$$
**`snp-k2p`**
$$
\begin{aligned}
a_1 &= 1-2P-Q \\
a_2 &= 1-2Q \\
d &= -\frac{1}{2}\ln a_1-\frac{1}{4}\ln a_2
\end{aligned}
$$
**`snp-k81`**
$$
\begin{aligned}
a_1 &= 1-2P-2Q_1 \\
a_2 &= 1-2P-2Q_2 \\
a_3 &= 1-2Q_1-2Q_2 \\
d &= -\frac{1}{4}\left(\ln a_1+\ln a_2+\ln a_3\right)
\end{aligned}
$$
**`snp-f81`**
$$
\begin{aligned}
E &= 1-\left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right) \\
d &= -E\ln\!\left(1-\frac{p}{E}\right)
\end{aligned}
$$
**`snp-t92`**
$$
\begin{aligned}
g &= \pi_C+\pi_G \\
w &= 2g(1-g) \\
a_1 &= 1-\frac{P}{w}-Q \\
a_2 &= 1-2Q \\
d &= -w\ln a_1-\frac{1}{2}(1-w)\ln a_2
\end{aligned}
$$
**`snp-tn93`**
$$
\begin{aligned}
g_R &= \pi_A+\pi_G \\
g_Y &= \pi_C+\pi_T \\
k_1 &= \frac{2\pi_A\pi_G}{g_R} \\
k_2 &= \frac{2\pi_C\pi_T}{g_Y} \\
k_3 &= 2\left(g_Rg_Y-\frac{\pi_A\pi_G\,g_Y}{g_R}-\frac{\pi_C\pi_T\,g_R}{g_Y}\right) \\
w_1 &= 1-\frac{P_1}{k_1}-\frac{Q}{2g_R} \\
w_2 &= 1-\frac{P_2}{k_2}-\frac{Q}{2g_Y} \\
w_3 &= 1-\frac{Q}{2g_Rg_Y} \\
d &= -k_1\ln w_1-k_2\ln w_2-k_3\ln w_3
\end{aligned}
$$
**`snp-tv`** — transversions only, deliberately uncorrected:
$$d = Q$$
A rate-heterogeneity correction applies to every formula above except `snp-raw` and `snp-tv`: each $-\ln(x)$ term is replaced by $\alpha\left(x^{-1/\alpha}-1\right)$ (same weight, same $x$) — see [Gamma rate heterogeneity](gamma_rate_heterogeneity.md) for the correction itself and how $\alpha$ is estimated.
See [`phylo`](../../../usage/phylo.md) for how to select a `snp-*` value and the prerequisite index annex it requires.
@@ -0,0 +1,27 @@
# Gamma rate heterogeneity
Real substitution rates vary across sites rather than being uniform, which a plain substitution correction (see [Central-position SNP model](central_snp_model.md)) ignores. The standard $+\Gamma$ correction models this by replacing each $-\ln(x)$ term in a correction formula with $\alpha\left(x^{-1/\alpha}-1\right)$ — the same $x$, the same weight, under the assumption that the per-site rate is $\mathrm{Gamma}(\alpha,\alpha)$-distributed (mean 1) rather than fixed. A small $\alpha$ indicates strong among-site rate heterogeneity (many near-invariant sites, a few fast ones); as $\alpha$ grows large the correction converges to the uncorrected formula.
## Automatic $\alpha$ estimation
$\alpha$ can be estimated automatically from the index itself, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.
The estimator pools substitution counts by **partition** rather than by genome pair: for partition $i$, let $n_i$ be the total number of substitutions observed across every genome pair, and $L_i$ the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:
$$R_i = \frac{n_i}{L_i}$$
Under a single shared substitution rate with no among-site heterogeneity, each $R_i$ would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as in the correction itself, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:
$$\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}$$
where $\mu$ is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own $L_i$ removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:
$$\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}}$$
$$\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}$$
If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), $\alpha$ is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged.
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation.
@@ -0,0 +1,24 @@
# Sampling design
Exhaustive computation over every non-monomorphic family in an index (see [Central-position SNP model](central_snp_model.md)) is the default, but can be bounded to approximately `N` families instead. Sampling is designed around two properties: staying representative of the whole index, and letting the user favor families that actually carry signal.
## Proportional sampling
Families are drawn in proportion to how many candidate families each part of the index actually holds, rather than uniformly across index partitions — this keeps the sample's composition representative of the whole index regardless of how unevenly candidate families happen to be distributed across partitions.
## Per-family informativeness: Shannon entropy
A family's informativeness is measured directly rather than assumed uniform. For a family, entropy is computed over its observed states across genomes:
| Quantity | Definition |
|---|---|
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
| `entropy4` | the same, reduced to the 4 plain bases |
A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).
## Entropy-biased sampling
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---subsample---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
@@ -0,0 +1,22 @@
# The 16-state Sankoff model
Each family (see [Central-position SNP model](central_snp_model.md)) is treated as a character with 16 possible states: one per subset of the 4 possible central bases, including the empty subset (no observed form). A genome pair's calibration reduces to two separately-estimated first-order Markov models, both restricted to genome pairs below a configurable SNP-ratio ceiling (excluding saturated pairs from skewing the calibration):
- **Cardinality transitions** ($5 \times 5$, row-stochastic): the probability that a family observed with $i$ forms in one genome ($i \in \{0,\dots,4\}$) is observed with $j$ forms in the other.
- **Composition transitions** ($4 \times 4$, row-stochastic): the probability that an unambiguous, single-copy base observed as one of A/C/G/T in one genome is observed as another base in the other genome.
## Combining into a 16×16 cost matrix
The two matrices are *not* combined as a plain tensor product. For a pair of states $(A, B)$ — each a subset of $\{A,C,G,T\}$ — let `shared` $= A \cap B$, `lost` $= A \setminus B$, `gained` $= B \setminus A$. The log-probability of the transition $A \to B$ is:
$$\log P(A,B) = \underbrace{\log P_{\text{card}}(|A|,|B|)}_{\text{cardinality term, omitted when loss events are treated as free}} \;+\; \underbrace{\sum_{i \,\in\, \text{shared}} \log P_{\text{comp}}(i,i)}_{\text{bases conserved on both sides}} \;-\; \underbrace{\text{best\_pairing\_cost}(\text{lost}, \text{gained})}_{\text{substitutions among the differing bases}}$$
`best_pairing_cost` enumerates every injection pairing elements of `lost` with elements of `gained` and keeps the one minimizing $-\sum \log P_{\text{comp}}$ over the pairs — the discrete analogue of "prefer a substitution over an independent loss and gain": if a base is lost from one side and a different base is gained on the other, that is scored as a single substitution between them (cheaper, under any sane calibration, than treating them as two unrelated cardinality-changing events) whenever such a pairing is possible.
The resulting $16 \times 16$ log-probability matrix is row-normalized in log-space (log-sum-exp, not a direct `exp()`/divide, for numerical stability), giving a genuine row-stochastic transition matrix $P$. The cost matrix is then $\text{cost}(A,B) = -\ln P(A,B)$, symmetrized by simple averaging:
$$\text{sym}(A,B) = \frac{\text{cost}(A,B) + \text{cost}(B,A)}{2}$$
This calibrated cost matrix is what every downstream export (for TNT, PhyG, IQ-TREE) reuses, recoded into each tool's own format.
See [`phylo`](../../../usage/phylo.md#sankoff-calibration-and-phylogenetic-exports) for how to run the calibration and produce these exports.