docs: restructure documentation for kmer indexing and phylogeny theory

Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
This commit is contained in:
Eric Coissac committed 2026-11-19 17:47:59 +01:00
1 parent d78a6b4501
commit 33d7a59dd9
43 files changed
+25375 -1117

No files matched your search

+67 -26
View File
@@ -47,7 +47,9 @@ P(n_K > 0) &= 1-e^{-\lambda} \\
\end{aligned}
$$
Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
##### Assuming equal genome lengths
The number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
$$
@@ -56,11 +58,22 @@ $$
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
$$
I_p(r,S-r+1)
=
\frac{1}{B(r,S-r+1)}
\int_0^p t^{r-1}(1-t)^{S-r}\,dt
$$
and
$$
p = 1-e^{-2G/4^k}.
$$
Different genome sizes
the regularized incomplete beta function is available in standard statistical libraries.
##### Different genome sizes
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
@@ -82,39 +95,63 @@ Using the mean genome size therefore provides a conservative estimate of the FDR
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
### Exact tail probability
##### Exact tail probability
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
Let $P_i(j)$ denote the probability that the k-mer occurs in exactly $j$ of the first $i$ genomes. Initially,
$$
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
$$
When genome $i$ is added, the k-mer is either absent, with probability $1-p_i$, or present, with probability $p_i$. The probability of observing exactly $j$ presences among the first $i$ genomes is therefore
$$
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
$$
This dynamic programming algorithm requires $O(S^2)$ operations and $O(S)$ memory. However, to compute $P(X\geq r)$, only the probabilities for fewer than $r$ presences need to be tracked, since
$$
P(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
$$
This reduces the computational cost to $O(Sr)$ operations and $O(r)$ memory.
#### Distinguishing random from non-random k-mer sharing
The probability $P(X\geq r)$ describes the occurrence of a given k-mer in at least $r$ genomes under the random-sequence model. However, the index contains only k-mers observed in at least one genome. Therefore, when considering the k-mers recorded in the index, the relevant probability is the conditional probability:
$$
q_r = P(X\geq r\mid X\geq 1)
= \frac{P(X\geq r)}{P(X\geq 1)}.
$$
Let $k_r$ be the number of distinct k-mers observed in at least $r$ genomes, among the $k_1$ distinct k-mers recorded in the index. The observed proportion of k-mers shared by at least $r$ genomes is:
$$
O_r = \frac{k_r}{k_1}.
$$
Under the random-sequence model, the expected proportion is $q_r$. We can therefore quantify the deviation from random expectations using the difference:
$$
\left \{
\begin{aligned}
d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\
d_0&=1
D_r &= O_r-q_r \\
&= \frac{k_r}{k_1}-q_r.
\end{aligned}
\right .
$$
before processing the first genome.
Positive values of $D_r$ indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.
The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$ is bounded:
$$
P(X\geq r)=1-\sum_{j=0}^{r-1}d_j.
-1\leq D_r\leq 1.
$$
Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$.
When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean
$$
\lambda=\sum_{i=1}^{S}p_i.
$$
The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe].
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By construction, $D_1=0$, since all k-mers recorded in the index occur in at least one genome. Unlike a ratio-based measure, this difference remains defined even when no k-mers are observed in at least $r$ genomes. It thus provides a bounded measure of the excess or deficit of k-mer sharing relative to the random-sequence model.
*******
@@ -150,11 +187,12 @@ Consequently, the appropriate value of $k$ depends on the total amount of sequen
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Mycoplasma genitalium* having a size $G=580\,Mb$ [@Fraser1995-jc], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Nasuia deltocephalinicola* having a size $G=112\,Kb$ [@Bennett2013-cl], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
$$
\begin{aligned}
P(C\geq1) &= 1- e^{-\frac{580.000^2}{2 \cdot 4^{11}}} \\
P(C\geq1) &= 1- e^{-\frac{112\,000^2}{2 \cdot 4^{11}}} \\
&= 1- e^{-1.5\times10^{3}} \\
&\simeq 1
\end{aligned}
$$
@@ -182,7 +220,7 @@ The distinction between the simple $4^k$ model and the Karlin--Ost framework is
These scales have very different implications for different genome sizes.
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995; @Karlin1997]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995-lt; @Karlin1994-on]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.
@@ -232,7 +270,7 @@ $$
The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985].
<!--
$o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ».
Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ :
@@ -265,7 +303,7 @@ Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nomb
- **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant.
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
-->
## Chance sharing across genomes and the FDR of homology
@@ -359,6 +397,7 @@ Chance alone produces millions of pairwise shared 21-mers at this genome size, a
---
<!-- Notes perso (bac à sable), non publiées dans le HTML
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
## 1. FDR d'un ensemble de k-mers sélectionnés
@@ -414,3 +453,5 @@ Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Ell
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
-->
@@ -34,7 +34,7 @@ $$
D = \sum_i \mathbb{1}[a_i \ne b_i],
$$
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded.
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded. On an index that stores counts, a k-mer is taken as present when its count reaches the presence threshold (`--presence-threshold`, 1 by default).
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
@@ -24,4 +24,4 @@ If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation.
See [`phylo`](../../../usage/phylo.md#distance-estimation-and-tree-reconstruction) for how to select a fixed $\alpha$ or request automatic estimation.
@@ -31,4 +31,4 @@ How well a pair of genomes is represented in a sample depends on how many famili
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
See [`phylo`](../../../usage/phylo.md#snp-character-selection-and-sampling) for how to bound and bias the sample.