Files
obikmer/UserDocMD/theory/kmer_indexing/kmers.md
T
Eric Coissac 33d7a59dd9 docs: restructure documentation for kmer indexing and phylogeny theory
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
2026-11-19 17:47:59 +01:00

32 KiB

K-mer size

A k-mer is a DNA sequence of length k. OBIKmer is designed to index large k-mers. Here, large refers to a k-mer length for which the probability of observing the same k-mer in several genomes by chance alone is sufficiently small. The k-mer length must therefore be large enough that, when a k-mer is observed in several genomes of the dataset, the probability that this shared occurrence is due to chance alone is sufficiently low for the match to be considered evidence of homology.

OBIKmer restricts k to odd values in the range


11 \leq k \leq 31.

These limits result from two different considerations. The upper bound is imposed by the representation used by OBIKmer, whereas the lower bound defines a practical range of k-mer lengths for genome-scale sequence comparisons. This range covers most practical applications: the lower limit is well below the k-mer size that can be considered large, even for the smallest bacterial genomes, whereas the upper limit provides sufficient specificity for most applications, including those involving very large genomes.

Representation limit

OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer requires 2k bits. To ensure computational efficiency, OBIKmer limits the representation of k-mers to a single 64-bit word, thereby restricting their length.


2k \leq 64,

This gives the hard representation limit: k \leq 32. Furthermore, OBIKmer only considers odd values of k, meaning the largest supported k-mer size is 31.

K-mer specificity

One simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are 4^k possible k-mers, so a particular k-mer is expected to occur once every 4^k positions on average. In a sequence of length G, the expected number of occurrences of a particular k-mer is approximately G \cdot 4^{-k}. When k is large enough to ensure 4^k >> G, the chance of a second occurrence of the same k-mer becomes close to zero. Consequently, observing the same k-mer twice can only be explained by a homology relationship, except for low-complexity k-mers, which are overrepresented in genomes.

Estimating False discovery rate on homology inference

OBIKmer uses k-mers long enough that their occurrence in multiple indexed genomes provides evidence of homology. The corresponding false discovery rate (FDR) can be expressed in terms of the probability of observing a k-mer in at least R of the S indexed genomes under a purely random model.

The random model

The random model, or null hypothesis, assumes that the S genomes are unrelated and that each genome is an independent random sequence in which each nucleotide occurs with equal probability. Rejecting this hypothesis for a given k-mer indicates that its presence in multiple genomes is unlikely to be explained by chance alone, providing evidence for homology.

For a genome of size G, the number of occurrences of a given canonical k-mer K is approximated by a Poisson distribution with parameter

\lambda = 2\frac{G}{4^k},

which is also the expected number of occurrences under the random model. The factor of two accounts for k-mer canonicalisation, which halves the k-mer space when k is odd.

The probability of observing at least one occurrence of this k-mer is therefore


\begin{aligned}
P(n_K > 0) &= 1-e^{-\lambda} \\
&= 1-e^{-2G/4^k}.
\end{aligned}
Assuming equal genome lengths

The number of genomes containing at least one occurrence of K among the S sequences follows a binomial distribution. The probability of observing K in at least r genomes is therefore


P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right)^{S-j}.

This sum is the regularized incomplete beta function I_p(r,\,S-r+1), where


I_p(r,S-r+1)
=
\frac{1}{B(r,S-r+1)}
\int_0^p t^{r-1}(1-t)^{S-r}\,dt

and


p = 1-e^{-2G/4^k}.

the regularized incomplete beta function is available in standard statistical libraries.

Different genome sizes

When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes G_i by their mean

\bar G = \frac{1}{S}\sum_{i=1}^{S}G_i.

This gives a common probability

\bar p = 1-e^{-2\bar G/4^k},

and the number of genomes containing the k-mer can then be approximated by a binomial distribution:

P(X\geq r)=\sum_{j=r}^{S}\binom{S}{j}\bar p^j(1-\bar p)^{S-j}.

This approximation is conservative for the upper tail. Hoeffding [-@Hoeffding1956-bx, p. 717, Eq. 26] shows that, for a fixed mean, the binomial upper tail is at least as large as the corresponding Poisson-binomial tail for

r\geq S\bar p+1.

Using the mean genome size therefore provides a conservative estimate of the FDR.

When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities p_i.

Exact tail probability

With genome-specific probabilities p_i, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly j presences as a sum over all subsets of j genomes would be impractical, since there can be up to 2^S subsets. The distribution can instead be obtained by adding the genomes one at a time.

Let P_i(j) denote the probability that the k-mer occurs in exactly j of the first i genomes. Initially,


P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).

When genome i is added, the k-mer is either absent, with probability 1-p_i, or present, with probability p_i. The probability of observing exactly j presences among the first i genomes is therefore


P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).

This dynamic programming algorithm requires O(S^2) operations and O(S) memory. However, to compute P(X\geq r), only the probabilities for fewer than r presences need to be tracked, since


P(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).

This reduces the computational cost to O(Sr) operations and O(r) memory.

Distinguishing random from non-random k-mer sharing

The probability P(X\geq r) describes the occurrence of a given k-mer in at least r genomes under the random-sequence model. However, the index contains only k-mers observed in at least one genome. Therefore, when considering the k-mers recorded in the index, the relevant probability is the conditional probability:


q_r = P(X\geq r\mid X\geq 1)
    = \frac{P(X\geq r)}{P(X\geq 1)}.

Let k_r be the number of distinct k-mers observed in at least r genomes, among the k_1 distinct k-mers recorded in the index. The observed proportion of k-mers shared by at least r genomes is:


O_r = \frac{k_r}{k_1}.

Under the random-sequence model, the expected proportion is q_r. We can therefore quantify the deviation from random expectations using the difference:


\begin{aligned}
D_r &= O_r-q_r \\
    &= \frac{k_r}{k_1}-q_r.
\end{aligned}

Positive values of D_r indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.

Since both O_r and q_r are proportions between zero and one, D_r is bounded:


-1\leq D_r\leq 1.

More precisely, for a given q_r, its bounds are -q_r and 1-q_r. By construction, D_1=0, since all k-mers recorded in the index occur in at least one genome. Unlike a ratio-based measure, this difference remains defined even when no k-mers are observed in at least r genomes. It thus provides a bounded measure of the excess or deficit of k-mer sharing relative to the random-sequence model.


This probability decreases by a factor of four for every additional nucleotide in the k-mer. Conversely, it increases linearly with the amount of sequence searched.

This simple model already shows why very short k-mers provide limited specificity for genome-scale analyses. With k=11,


4^{11}\simeq4.2\times10^6,

which is a small combinatorial space even compared with the size of very small bacterial genomes. In contrast,


4^{31}\simeq4.6\times10^{18},

which is vastly larger than the number of k-mer positions even in very large genomes. Thus, increasing k from 11 to 31 provides an enormous increase in sequence specificity.

The estimate above is intentionally simplistic. Real genomes do not consist of independent, equiprobable nucleotides: nucleotide composition and sequence dependencies affect the probability of observing particular words. More accurate estimates of the probability of observing matches by chance therefore take sequence length and nucleotide composition into account.

Karlin and Ost provide an analytical framework for this problem based on the distribution of the longest exact match expected between random sequences. If L_{\max} denotes the length of the longest seed observed in random sequences of given size and nucleotide composition, the corresponding seed-level probability is


P_{\mathrm{seeds}}(L)=P(L_{\max}\geq L).

Conversely, for a chosen false-positive probability P, one can determine the minimum seed length L_{\min} such that matches of length L_{\min} or greater are not expected to occur by chance. This is the approach used by Repseek to determine statistically significant seed lengths.

The simple G/4^k estimate and the Karlin–Ost statistics therefore describe the same fundamental effect at different levels of approximation: the probability of a chance match decreases rapidly with k-mer length, while the amount of sequence being searched increases the opportunity for such matches.

Consequently, the appropriate value of k depends on the total amount of sequence being compared and its nucleotide composition. Choosing k appropriately allows the false-positive rate of homology inference to be controlled.

Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today Nasuia deltocephalinicola having a size G=112\,Kb [@Bennett2013-cl], and the largest 13\,Mb in Sorangium cellulosum [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:


\begin{aligned}
P(C\geq1) &= 1- e^{-\frac{112\,000^2}{2 \cdot 4^{11}}} \\
&= 1- e^{-1.5\times10^{3}} \\
&\simeq 1
\end{aligned}

, whereas 21-mers already provide a space of several trillion possible sequences.

Plant genomes illustrate the opposite situation. Plant genome sizes span several orders of magnitude, from the smallest angiosperm genomes present in the carnivorous plans genus [@Fleischmann2014-ed] about 60 Mb to the largest one 160 Gb in Tmesipteris oblanceolata, a New Caledonian fork fern [@Fernandez2024-kd], and large genomes often contain substantial amounts of repeated sequence. Even at k=31, however, the theoretical k-mer space remains vastly larger than any individual genome. At these lengths, the limiting factor is therefore not exhaustion of the theoretical k-mer space but the biological repetitiveness and non-random composition of the genome.

For example, the number of possible k-mers grows as follows:

k 4^k 4^k/2 after canonicalization
11 4.19\times10^6 2.10\times10^6
15 1.07\times10^9 5.37\times10^8
21 4.40\times10^{12} 2.20\times10^{12}
25 1.13\times10^{15} 5.63\times10^{14}
31 4.61\times10^{18} 2.31\times10^{18}

Canonicalization approximately halves the number of possible representations because a k-mer and its reverse complement are represented by the same canonical form.

The distinction between the simple 4^k model and the Karlin--Ost framework is important. The former describes the size of the combinatorial k-mer space; the latter describes the probability of observing long matches in sequences whose composition may depart from the uniform random model. Consequently, the value of k cannot be selected from genome size alone.

These scales have very different implications for different genome sizes.

The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995-lt; @Karlin1994-on]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.

This distinction is important for whole-genome k-mer comparison. Increasing k increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length k\leq11 can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.

The supported range 11\leq k\leq31 should therefore be understood as an engineering compromise rather than as a universal biological optimum. Smaller values increase the probability that unrelated genomic regions share the same k-mers, while larger values increase specificity but also make the representation more sensitive to substitutions, indels, and incomplete homology. The appropriate value within this range consequently depends on the biological scale and comparison task.

Odd k-mer lengths

OBIKmer additionally requires k to be odd. This constraint is independent of the resolution argument above.

For an odd-length DNA sequence, a k-mer cannot be identical to its own reverse complement. Consequently, every k-mer has two distinct orientations, which can be unambiguously collapsed into a canonical representation. For even k, self-reverse-complementary sequences exist, so a k-mer can coincide with its own reverse complement.

Odd k is therefore not required for canonicalization itself: the canonical form


s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)

remains well defined for even k. The odd-k restriction instead guarantees that the two orientations are always distinct, which simplifies the representation and orientation handling used by OBIKmer.

Karlin--Ost (1988) formula

[@Karlin1988] considers s independent stationary letter sequences over a finite alphabet and studies K_{r,s}(N), the length of the longest word common to r of the s sequences, each of length N. For i.i.d. equiprobable letters over an alphabet of size m (here m=4 for DNA), the characteristic local match parameter reduces to


\lambda^{[r]} = m^{1-r},

so for r=2 (a word shared by two occurrences), \lambda = m^{-1} = 1/4.

The expected number of common words of length k among the \binom{s}{r} relevant groups of sequences is


S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1)),

with \gamma=1 exactly in the i.i.d. equiprobable case. Theorem 2.2 gives the limiting distribution of K_{r,s}(N), with k set on the scale k=\left[\dfrac{\log N^{r}}{-\log\lambda^{*}}+x+1\right]:


\lim_{N\to\infty}\left[\Pr\{K_{r,s}(N)<k\} - \exp\{-(1-\lambda^{*})\,S_1(N,k)\}\right] = 0,

equivalently


\frac{-\log\Pr\{K_{r,s}(N)<k\}}{(\lambda^{*})^{\rho(N,x)}} \xrightarrow[N\to\infty]{} (1-\lambda^{*})\,\gamma^{*}(\lambda^{*})^{x}.

The factor (1-\lambda^{*}) multiplying S_1(N,k) is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for s\geq2 independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length R(N), in [@Karlin1985].

Chance sharing across genomes and the FDR of homology

The formulas above ask how likely a match is between two sequences. The question OBIKmer actually has to answer is different: a k-mer is found in r of the S indexed genomes, and one wants to know whether this sharing can be explained by chance or whether it is evidence of homology. This section builds a null model for that question and turns it into a false discovery rate (FDR) for the call "a k-mer shared by at least r genomes is homologous".

The null model

The null hypothesis is that the S genomes are unrelated, that is, that each one is an independent random sequence. Rejecting it for a k-mer means that chance alone cannot explain why the k-mer is shared, which is the signal of homology one is looking for.

The model only records whether a k-mer is present in a genome, not how many times. Two consequences follow. First, repeats inside a genome are irrelevant, and the size G_i of genome i is its number of distinct canonical k-mers. Second, a repeat family shared between genomes is paralogy, which is a form of homology; it is a true positive and not a false one. The only k-mers that could be shared by composition rather than by descent are low-complexity ones, which OBIKmer removes beforehand with its entropy filter.

Everything takes place in canonical space. Because k is odd, no k-mer equals its own reverse complement, so the 4^k k-mers pair up perfectly and the space contains exactly


N=\frac{4^k}{2}

canonical k-mers. A given canonical k-mer is present in genome i by chance with probability


p_i = 1-\left(1-\frac1N\right)^{G_i}\simeq \frac{G_i}{N}=\frac{2\,G_i}{4^k}.

This is twice the G/4^k used earlier in this page, because there G is a number of positions on one strand, whereas G_i here counts canonical k-mers, for which both strands are already merged.

Let R be the number of genomes, out of S, that contain a given canonical k-mer by chance. Genomes are independent under the null, so R is a sum of independent presence indicators, each with its own probability p_i.

Exact tail probability

When genome sizes differ, the p_i differ, and R follows a Poisson-binomial distribution. Writing the probability of exactly j presences as a sum over all subsets of j genomes would be hopeless, since there are up to 2^S subsets. The distribution is instead obtained by adding the genomes one at a time. Let d_j be the probability that exactly j of the genomes processed so far contain the k-mer. Adding genome i leaves j unchanged if the genome does not contain the k-mer, and moves j-1 to j if it does:


d'_j = d_j\,(1-p_i) + d_{j-1}\,p_i, \qquad d_0=1 \text{ before the first genome.}

The table d only needs to remember how many genomes were hit, not which ones, and this brings the cost down to O(S^2). The result does not depend on the order in which the genomes are processed. It does not depend on the k-mer either, so it is computed once per index. The upper tail is then


P_0(R\geq r) = 1-\sum_{j=0}^{r-1} d_j .

Only d_0,\dots,d_{r-1} are needed, so the cost is O(S\,r).

Approximation with equal genome sizes

If all genomes are given the same size, taken as the mean


\bar G=\frac1S\sum_{i=1}^{S}G_i, \qquad \bar p=\frac{2\bar G}{4^k},

then R follows a plain binomial distribution and the tail is


P_0(R\geq r)\simeq\sum_{j=r}^{S}\binom{S}{j}\,\bar p^{\,j}(1-\bar p)^{S-j}.

This sum is the regularized incomplete beta function I_{\bar p}(r,\,S-r+1), available in standard statistical libraries. If it is summed by hand, the terms are obtained one from the next by t_{j+1}=t_j\,\frac{S-j}{j+1}\,\frac{\bar p}{1-\bar p}.

A theorem of Hoeffding [@Hoeffding1956-bx] shows that this approximation errs on the safe side. For a fixed mean, the Poisson-binomial distribution is more concentrated than the binomial one, so for r\geq S\bar p+1 the binomial tail is never smaller than the exact tail. Using the mean genome size therefore overestimates the chance of sharing and can only make the FDR more pessimistic. Gleser [@Gleser1975-yu] refines this result: replacing the p_i by any partial average of themselves, not only by their full mean, also moves the upper tail towards the binomial one, provided r-1\geq S\bar p+2. Grouping genomes of similar size is therefore safe in the same sense. What it loses is power when genome sizes differ by orders of magnitude, since one large genome then weighs as much as several small ones in \bar p.

When all p_i are small, which is the case for large k, the number of hits is also well approximated by a Poisson distribution of mean \lambda=\sum_i p_i, with an error bounded by \sum_i p_i^2 [@Le-Cam1960-pe].

For the numerical evaluation, the tail that does not contain the mode S\bar p should be summed directly. When r>S\bar p, computing the sum from j=r upwards is accurate even for very small probabilities, whereas computing 1 minus the sum of the lower terms loses all precision once the tail falls below the machine epsilon.

False discovery rate

The FDR of the call "present in at least r genomes" is the fraction of retained k-mers expected to be pure chance:


\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P_0(R\geq r)}{n_{\geq r}}\right).

The numerator is the number of k-mers expected to reach r genomes by chance if the genomes were unrelated. Each of the N canonical k-mers has probability P_0(R\geq r) of doing so, and no further correction for multiple testing is needed because the number of tests is already in the product. The denominator n_{\geq r} is the number of distinct k-mers actually observed in at least r genomes, which is read directly from the index. The observed count mixes chance and true homology, so the ratio is an upper bound on the true FDR: it implicitly assumes that all k-mers are null, and it never underestimates the noise. The minimal r for which \widehat{\mathrm{FDR}}(r) falls below the accepted level, for example 5 %, is the sharing threshold to use.

As an illustration, take k=21 (so N=2.2\times10^{12}), S=10 genomes and G_i=10^9 distinct canonical k-mers each, hence p=4.5\times10^{-4}. The number of k-mers expected to be shared by chance is:

r N\,P_0(R\geq r)
2 2\times10^{7}
3 2.5\times10^{4}
4 \approx 20
5 \approx 0.01

Chance alone produces millions of pairwise shared 21-mers at this genome size, a few tens of thousands shared by three genomes, and almost none shared by five or more. Whether a given r is acceptable then depends on how many k-mers the index actually reports at that level.

Limits of the model

  • The FDR is computed per k-mer. Neighbouring k-mers of a shared region overlap and are not independent, so a single homologous region contributes many correlated k-mers. A per-region FDR would count regions instead of k-mers and is not covered here.
  • The probability p_i assumes equiprobable, independent nucleotides. The composition biases discussed earlier in this page violate this assumption, and the resulting error is not accounted for.