Files
obikmer/UserDocMD/theory/kmer_indexing/kmers.md
T
Eric Coissac 36c7247acb feat: add k-mer sharing estimation and Poisson-binomial null model
Add a new `estimate --index` command to compare observed k-mer sharing against
a random-sequence model using a Poisson-binomial distribution. Introduce the
`row_presence_histogram` API to support this analysis across various matrix
types. Update documentation and dependencies accordingly.

- Add `src/obikstats/src/null_model.rs` implementing `SharingNullModel` and
  `SharingRow` for computing cumulative probabilities and false discovery rates
  based on a Poisson-binomial distribution.
- Add `src/obikstats/src/sharing.rs` defining `GenomeSharing` and `SharingSpectrum`
  to compute k-mer sharing spectra using parallel map-reduce.
- Add `src/obikmer/src/cmd/estimate/sharing.rs` with `run` and `write_csv` to
  execute sharing estimation and output CSV results containing observed, expected,
  and deviation metrics.
- Modify `src/obikmer/src/cmd/estimate/mod.rs` to support the new `--index`
  subcommand, adding `EstimateArgs` fields for index path, presence threshold,
  and output file.
- Add `src/obicompactvec/src/row_histogram.rs` implementing `row_presence_histogram`
  for `PersistentIntMatrix`, `PersistentBitMatrix`, and sparse matrices to count
  rows present in exactly $j$ columns.
- Update `src/obikindex/src/layer/content_layer.rs` and `src/obikindex/src/layer/typed_layer.rs`
  to expose the new `row_presence_histogram` method, dispatching based on layer type.
- Add tests in `src/obicompactvec/src/tests/bitmatrix.rs`, `intmatrix.rs`, and
  `sparse.rs` to verify `count_geq` and `row_presence_histogram` against naive
  implementations for various matrix structures.
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to document the Poisson-binomial
  approach, $D_r$ bounds, and FDR calculations, replacing previous binomial approximations.
- Update `UserDocMD/usage/estimate.md` to describe the two uses of `estimate`:
  parameter estimation and index comparison via `--index`.
- Add `wiki` target to `Makefile` for regenerating documentation from user docs.
- Bump `obikmer` package version from 1.4.5 to 1.4.6 in `Cargo.toml` and `Cargo.lock`.
2026-11-19 17:50:29 +01:00

13 KiB

K-mer size

A k-mer is a DNA sequence of length k. OBIKmer is designed to index large k-mers. Here, large refers to a k-mer length for which the probability of observing the same k-mer in several genomes by chance alone is sufficiently small. The k-mer length must therefore be large enough that, when a k-mer is observed in several genomes of the dataset, the probability that this shared occurrence is due to chance alone is sufficiently low for the match to be considered evidence of homology.

OBIKmer restricts k to odd values in the range


11 \leq k \leq 31.

These limits result from two different considerations. The upper bound is imposed by the representation used by OBIKmer, whereas the lower bound defines a practical range of k-mer lengths for genome-scale sequence comparisons. This range covers most practical applications: the lower limit is well below the k-mer size that can be considered large, even for the smallest bacterial genomes, whereas the upper limit provides sufficient specificity for most applications, including those involving very large genomes.

Representation limit

OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer requires 2k bits. To ensure computational efficiency, OBIKmer limits the representation of k-mers to a single 64-bit word, thereby restricting their length.


2k \leq 64,

This gives the hard representation limit: k \leq 32. Furthermore, OBIKmer only considers odd values of k, meaning the largest supported k-mer size is 31.

Odd k-mer lengths

OBIKmer additionally requires k to be odd. This constraint is independent of the specificity considerations discussed below.

For an odd-length DNA sequence, a k-mer cannot be identical to its own reverse complement. Every k-mer therefore belongs to a pair of distinct orientations, which can be represented by a single canonical k-mer. For even k, self-reverse-complementary sequences exist.

Odd k is not required for canonicalization itself: the canonical form


s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)

is well defined for both odd and even lengths. Restricting k to odd values guarantees that the two orientations are always distinct, simplifying the representation and orientation handling used by OBIKmer.

K-mer specificity

A simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are 4^k possible oriented k-mers, so the expected number of occurrences of a particular oriented k-mer in a sequence of length G is approximately G/4^k. For canonical k-mers, the corresponding number of possible sequences is halved, and the expected number of occurrences is approximately 2G/4^k.

Increasing k therefore reduces the probability that a particular k-mer occurs by chance. However, the probability of a match somewhere in a collection of sequences also depends on the amount of sequence searched and on the number of possible k-mers considered. The following sections formalize this distinction and describe how chance sharing can be assessed across an indexed set of genomes.

Choosing an appropriate k-mer size

The theoretical k-mer space grows exponentially with k. Thus, increasing k from 11 to 31 greatly expands the sequence space and reduces the probability that unrelated sequences share a particular k-mer by chance. The following table gives the size of the theoretical k-mer space and, after canonicalization, the number of distinct representations.

k 4^k 4^k/2 after canonicalization
11 4.19\times10^6 2.10\times10^6
15 1.07\times10^9 5.37\times10^8
21 4.40\times10^{12} 2.20\times10^{12}
25 1.13\times10^{15} 5.63\times10^{14}
31 4.61\times10^{18} 2.31\times10^{18}

Genome sizes illustrate the range of sequence scales for which k-mers may be used. Bacterial genomes are typically a few megabases long, although some symbiotic bacteria have much smaller genomes, while other bacteria have genomes exceeding ten megabases. Plant genomes span an even wider range, from tens of megabases to more than one hundred gigabases. In very large and repeat-rich genomes, the main limitation is not necessarily the exhaustion of the theoretical k-mer space, but the non-uniform distribution of k-mers caused by sequence composition and repeated DNA.

The supported range 11\leq k\leq31 should therefore be understood as an engineering and application-oriented compromise, not as a universal biological optimum. Shorter k-mers are more likely to be shared by unrelated regions, whereas longer k-mers provide greater sequence specificity. However, longer k-mers are also more sensitive to substitutions and indels: a single difference can disrupt an exact match across the whole k-mer. The appropriate value within the supported range consequently depends on the size and characteristics of the genomes being compared, as well as on the biological question.

The following section introduces a random-sequence model to quantify the probability of chance sharing across an indexed set of genomes. This provides a formal basis for assessing how the choice of k affects the interpretation of shared k-mers.

Chance sharing across genomes

OBIKmer indexes distinct canonical k-mers and records the genomes in which each k-mer occurs. This makes it possible to compare the observed sharing of k-mers with the sharing expected under a random-sequence model. The same probability model can be used for two related purposes: measuring the excess of sharing among k-mers present in the index, and estimating the number of chance k-mers among those shared by at least a specified number of genomes.

The null model and genome-specific probabilities

The null hypothesis assumes that the S indexed genomes are unrelated and that each genome is an independent random sequence in which nucleotides are independent and equiprobable.

Since OBIKmer uses odd-length canonical k-mers, reverse-complement canonicalization reduces the number of possible k-mers by half. The size of the canonical k-mer space is therefore


M_k=\frac{4^k}{2}.

Let N_i be the number of distinct canonical k-mers recorded for genome i in the index. Under the null model, these can be regarded as a uniform sample without replacement from the M_k possible canonical k-mers. The probability that a given canonical k-mer is present in genome i is then


p_i=\frac{N_i}{M_k}=\frac{2N_i}{4^k}.

This is the inclusion probability for a distinct k-mer. It differs from the probability of at least one occurrence in a positional model, because repeated occurrences within a genome are not counted separately.

Let X denote the number of genomes, among the S indexed genomes, that contain a given canonical k-mer under the null model. Since genomes are assumed to be independent, X is a sum of independent presence indicators with probabilities p_1,\ldots,p_S. When these probabilities differ, X follows a Poisson-binomial distribution.

Computing the Poisson-binomial distribution

The distribution of X can be computed by processing genomes one at a time. Let P_i(j) denote the probability that the k-mer occurs in exactly j of the first i genomes. Initially,


P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).

When genome i is added, the k-mer is either absent, with probability 1-p_i, or present, with probability p_i. Thus,


P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).

This recurrence avoids enumerating the subsets of genomes. Computing the complete distribution requires O(S^2) operations and O(S) memory. To compute the upper-tail probability, only the probabilities for fewer than r presences need to be tracked:


P_{\mathrm{null}}(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).

The computation can therefore be restricted to P_i(0),\ldots,P_i(r-1), requiring O(Sr) operations and O(r) memory. The resulting distribution depends on the genome-specific values N_i, but not on the identity of the k-mer, and can thus be computed once for a given index and k-mer length.

Comparing observed and expected sharing in the index

The index contains only k-mers observed in at least one genome. Consequently, when comparing sharing proportions among the k-mers recorded in the index, the relevant null probability is conditional on at least one presence:


q_r=P(X\geq r\mid X\geq 1)
    =\frac{P_{\mathrm{null}}(X\geq r)}{P_{\mathrm{null}}(X\geq 1)}.

Let n_{\geq r} be the number of distinct canonical k-mers observed in at least r genomes, and let n_{\geq 1} be the total number of distinct canonical k-mers recorded in the index. The observed proportion of indexed k-mers shared by at least r genomes is


O_r=\frac{n_{\geq r}}{n_{\geq 1}}.

Under the random-sequence model, the expected proportion is q_r. The difference


D_r=O_r-q_r

therefore measures the excess or deficit of sharing relative to the null expectation. Positive values indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.

Since both O_r and q_r are proportions between zero and one, D_r is bounded:


-1\leq D_r\leq 1.

More precisely, for a given q_r, its bounds are -q_r and 1-q_r. By construction, D_1=0, since all k-mers recorded in the index occur in at least one genome. Unlike a ratio-based measure, this difference remains defined when no k-mers are observed in at least r genomes.

Estimating the false discovery rate

The probability P_{\mathrm{null}}(X\geq r) applies to one specified canonical k-mer. To estimate the number of k-mers that could reach this sharing threshold by chance across the entire canonical k-mer space, this probability must be considered over all M_k possible canonical k-mers.

The expected number of chance k-mers present in at least r genomes is


E[\mathrm{FP}(r)]=M_kP_{\mathrm{null}}(X\geq r).

The observed number of k-mers passing the threshold is n_{\geq r}. An estimate of the false discovery rate (FDR) for this set is therefore


\widehat{\mathrm{FDR}}(r)=
\begin{cases}
1 & \text{if } n_{\geq r}=0,\\[4pt]
\displaystyle
\min\left(1,
\frac{M_k P_{\mathrm{null}}(X\geq r)}
{n_{\geq r}}\right)
& \text{otherwise}.
\end{cases}

The numerator estimates the number of false positives under the null model, while the denominator is the observed number of k-mers passing the sharing threshold. This estimate assumes that all possible canonical k-mers are subject to the null model. It is therefore conservative in the sense that it does not attempt to estimate the proportion of the k-mer space corresponding to true homology. The estimate is capped at one because a proportion cannot exceed one.

The distinction between D_r and the FDR is important. D_r compares the observed and expected proportions among k-mers already present in the index. The FDR estimate instead compares the expected number of chance k-mers in the entire canonical k-mer space with the observed number passing the sharing threshold.

The smallest value of r for which \widehat{\mathrm{FDR}}(r) falls below a chosen threshold, for example 5%, can be used as a minimum sharing threshold. This interpretation is conditional on the assumptions of the null model and should not be taken as a model-independent guarantee of the biological false discovery rate.

As an illustration, consider k=21, for which M_k\simeq2.2\times10^{12}, and an index containing S=10 genomes. Suppose that the numbers of distinct canonical k-mers in these genomes range from 0.5\times10^9 to 1.4\times10^9, in increments of 0.1\times10^9. Applying the Poisson-binomial model with the corresponding genome-specific probabilities gives the following expected numbers of k-mers shared by chance:

r M_kP_{\mathrm{null}}(X\geq r)
2 1.82\times10^7
3 2.06\times10^4
4 15
5 0.0075

In this example, chance alone can produce many pairwise shared 21-mers, whereas the expected number of k-mers shared by five or more genomes is very small. Whether a given sharing threshold is sufficiently stringent depends on the number of k-mers actually observed at that level.

Limits of the model

The FDR estimate is based on a simplified null model and has two important limitations.

First, the model assumes independent, equiprobable nucleotides. Real genomes exhibit nucleotide-composition biases, oligonucleotide dependencies, and repeated sequences. These properties can increase the probability of particular k-mers beyond the uniform expectation. Removing low-complexity k-mers reduces some of these effects but does not eliminate all compositional biases. Karlin and colleagues showed that relative oligonucleotide abundances, including dinucleotide frequencies, form characteristic genomic signatures that can differ between organisms and remain relatively stable within genomes [@Karlin1995-lt; @Karlin1994-on].

Second, the FDR is computed per k-mer. Consecutive k-mers extracted from the same homologous region overlap and are therefore not independent observations. A single homologous region can contribute many correlated k-mers to the index. The estimate presented here consequently describes the expected fraction of chance k-mers, not the fraction of false homologous regions. A region-level FDR would require a model that treats matching regions, rather than individual k-mers, as the units of observation.