Files
obikmer/UserDocMD/theory/kmer_indexing/kmers.md
T
Eric Coissac 36c7247acb feat: add k-mer sharing estimation and Poisson-binomial null model
Add a new `estimate --index` command to compare observed k-mer sharing against
a random-sequence model using a Poisson-binomial distribution. Introduce the
`row_presence_histogram` API to support this analysis across various matrix
types. Update documentation and dependencies accordingly.

- Add `src/obikstats/src/null_model.rs` implementing `SharingNullModel` and
  `SharingRow` for computing cumulative probabilities and false discovery rates
  based on a Poisson-binomial distribution.
- Add `src/obikstats/src/sharing.rs` defining `GenomeSharing` and `SharingSpectrum`
  to compute k-mer sharing spectra using parallel map-reduce.
- Add `src/obikmer/src/cmd/estimate/sharing.rs` with `run` and `write_csv` to
  execute sharing estimation and output CSV results containing observed, expected,
  and deviation metrics.
- Modify `src/obikmer/src/cmd/estimate/mod.rs` to support the new `--index`
  subcommand, adding `EstimateArgs` fields for index path, presence threshold,
  and output file.
- Add `src/obicompactvec/src/row_histogram.rs` implementing `row_presence_histogram`
  for `PersistentIntMatrix`, `PersistentBitMatrix`, and sparse matrices to count
  rows present in exactly $j$ columns.
- Update `src/obikindex/src/layer/content_layer.rs` and `src/obikindex/src/layer/typed_layer.rs`
  to expose the new `row_presence_histogram` method, dispatching based on layer type.
- Add tests in `src/obicompactvec/src/tests/bitmatrix.rs`, `intmatrix.rs`, and
  `sparse.rs` to verify `count_geq` and `row_presence_histogram` against naive
  implementations for various matrix structures.
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to document the Poisson-binomial
  approach, $D_r$ bounds, and FDR calculations, replacing previous binomial approximations.
- Update `UserDocMD/usage/estimate.md` to describe the two uses of `estimate`:
  parameter estimation and index comparison via `--index`.
- Add `wiki` target to `Makefile` for regenerating documentation from user docs.
- Bump `obikmer` package version from 1.4.5 to 1.4.6 in `Cargo.toml` and `Cargo.lock`.
2026-11-19 17:50:29 +01:00

310 lines
13 KiB
Markdown

# K-mer size
A k-mer is a DNA sequence of length $k$. OBIKmer is designed to index
large k-mers. Here, *large* refers to a k-mer length for which the
probability of observing the same k-mer in several genomes by chance
alone is sufficiently small. The k-mer length must therefore be large
enough that, when a k-mer is observed in several genomes of the dataset,
the probability that this shared occurrence is due to chance alone is
sufficiently low for the match to be considered evidence of homology.
OBIKmer restricts $k$ to odd values in the range
$$
11 \leq k \leq 31.
$$
These limits result from two different considerations. The upper bound
is imposed by the representation used by OBIKmer, whereas the lower
bound defines a practical range of k-mer lengths for genome-scale
sequence comparisons. This range covers most practical applications: the
lower limit is well below the k-mer size that can be considered large,
even for the smallest bacterial genomes, whereas the upper limit
provides sufficient specificity for most applications, including those
involving very large genomes.
## Representation limit
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer
requires $2k$ bits. To ensure computational efficiency, OBIKmer limits
the representation of k-mers to a single 64-bit word, thereby
restricting their length.
$$
2k \leq 64,
$$
This gives the hard representation limit: $k \leq 32.$ Furthermore,
OBIKmer only considers odd values of $k$, meaning the largest supported
k-mer size is $31$.
## Odd k-mer lengths
OBIKmer additionally requires $k$ to be odd. This constraint is
independent of the specificity considerations discussed below.
For an odd-length DNA sequence, a k-mer cannot be identical to its own
reverse complement. Every k-mer therefore belongs to a pair of distinct
orientations, which can be represented by a single canonical k-mer. For
even $k$, self-reverse-complementary sequences exist.
Odd $k$ is not required for canonicalization itself: the canonical form
$$
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
$$
is well defined for both odd and even lengths. Restricting $k$ to odd
values guarantees that the two orientations are always distinct,
simplifying the representation and orientation handling used by OBIKmer.
## K-mer specificity
A simple way to illustrate the effect of k-mer size is to consider a
random DNA sequence in which the four nucleotides are independent and
equally probable. There are $4^k$ possible oriented k-mers, so the
expected number of occurrences of a particular oriented k-mer in a
sequence of length $G$ is approximately $G/4^k$. For canonical k-mers,
the corresponding number of possible sequences is halved, and the
expected number of occurrences is approximately $2G/4^k$.
Increasing $k$ therefore reduces the probability that a particular k-mer
occurs by chance. However, the probability of a match somewhere in a
collection of sequences also depends on the amount of sequence searched
and on the number of possible k-mers considered. The following sections
formalize this distinction and describe how chance sharing can be
assessed across an indexed set of genomes.
## Choosing an appropriate k-mer size
The theoretical k-mer space grows exponentially with $k$. Thus, increasing $k$ from 11 to 31 greatly expands the sequence space and reduces the probability that unrelated sequences share a particular k-mer by chance. The following table gives the size of the theoretical k-mer space and, after canonicalization, the number of distinct representations.
| $k$ | $4^k$ | $4^k/2$ after canonicalization |
|---:|---:|---:|
| 11 | $4.19\times10^6$ | $2.10\times10^6$ |
| 15 | $1.07\times10^9$ | $5.37\times10^8$ |
| 21 | $4.40\times10^{12}$ | $2.20\times10^{12}$ |
| 25 | $1.13\times10^{15}$ | $5.63\times10^{14}$ |
| 31 | $4.61\times10^{18}$ | $2.31\times10^{18}$ |
Genome sizes illustrate the range of sequence scales for which k-mers may be used. Bacterial genomes are typically a few megabases long, although some symbiotic bacteria have much smaller genomes, while other bacteria have genomes exceeding ten megabases. Plant genomes span an even wider range, from tens of megabases to more than one hundred gigabases. In very large and repeat-rich genomes, the main limitation is not necessarily the exhaustion of the theoretical k-mer space, but the non-uniform distribution of k-mers caused by sequence composition and repeated DNA.
The supported range $11\leq k\leq31$ should therefore be understood as an engineering and application-oriented compromise, not as a universal biological optimum. Shorter k-mers are more likely to be shared by unrelated regions, whereas longer k-mers provide greater sequence specificity. However, longer k-mers are also more sensitive to substitutions and indels: a single difference can disrupt an exact match across the whole k-mer. The appropriate value within the supported range consequently depends on the size and characteristics of the genomes being compared, as well as on the biological question.
The following section introduces a random-sequence model to quantify the probability of chance sharing across an indexed set of genomes. This provides a formal basis for assessing how the choice of $k$ affects the interpretation of shared k-mers.
## Chance sharing across genomes
OBIKmer indexes distinct canonical k-mers and records the genomes in
which each k-mer occurs. This makes it possible to compare the observed
sharing of k-mers with the sharing expected under a random-sequence
model. The same probability model can be used for two related purposes:
measuring the excess of sharing among k-mers present in the index, and
estimating the number of chance k-mers among those shared by at least a
specified number of genomes.
### The null model and genome-specific probabilities
The null hypothesis assumes that the $S$ indexed genomes are unrelated
and that each genome is an independent random sequence in which
nucleotides are independent and equiprobable.
Since OBIKmer uses odd-length canonical k-mers, reverse-complement
canonicalization reduces the number of possible k-mers by half. The size
of the canonical k-mer space is therefore
$$
M_k=\frac{4^k}{2}.
$$
Let $N_i$ be the number of distinct canonical k-mers recorded for genome
$i$ in the index. Under the null model, these can be regarded as a
uniform sample without replacement from the $M_k$ possible canonical
k-mers. The probability that a given canonical k-mer is present in
genome $i$ is then
$$
p_i=\frac{N_i}{M_k}=\frac{2N_i}{4^k}.
$$
This is the inclusion probability for a distinct k-mer. It differs from
the probability of at least one occurrence in a positional model,
because repeated occurrences within a genome are not counted separately.
Let $X$ denote the number of genomes, among the $S$ indexed genomes,
that contain a given canonical k-mer under the null model. Since genomes
are assumed to be independent, $X$ is a sum of independent presence
indicators with probabilities $p_1,\ldots,p_S$. When these probabilities
differ, $X$ follows a Poisson-binomial distribution.
### Computing the Poisson-binomial distribution
The distribution of $X$ can be computed by processing genomes one at a
time. Let $P_i(j)$ denote the probability that the k-mer occurs in
exactly $j$ of the first $i$ genomes. Initially,
$$
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
$$
When genome $i$ is added, the k-mer is either absent, with probability
$1-p_i$, or present, with probability $p_i$. Thus,
$$
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
$$
This recurrence avoids enumerating the subsets of genomes. Computing the
complete distribution requires $O(S^2)$ operations and $O(S)$ memory. To
compute the upper-tail probability, only the probabilities for fewer
than $r$ presences need to be tracked:
$$
P_{\mathrm{null}}(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
$$
The computation can therefore be restricted to $P_i(0),\ldots,P_i(r-1)$,
requiring $O(Sr)$ operations and $O(r)$ memory. The resulting
distribution depends on the genome-specific values $N_i$, but not on the
identity of the k-mer, and can thus be computed once for a given index
and k-mer length.
### Comparing observed and expected sharing in the index
The index contains only k-mers observed in at least one genome.
Consequently, when comparing sharing proportions among the k-mers
recorded in the index, the relevant null probability is conditional on
at least one presence:
$$
q_r=P(X\geq r\mid X\geq 1)
=\frac{P_{\mathrm{null}}(X\geq r)}{P_{\mathrm{null}}(X\geq 1)}.
$$
Let $n_{\geq r}$ be the number of distinct canonical k-mers observed in
at least $r$ genomes, and let $n_{\geq 1}$ be the total number of
distinct canonical k-mers recorded in the index. The observed proportion
of indexed k-mers shared by at least $r$ genomes is
$$
O_r=\frac{n_{\geq r}}{n_{\geq 1}}.
$$
Under the random-sequence model, the expected proportion is $q_r$. The
difference
$$
D_r=O_r-q_r
$$
therefore measures the excess or deficit of sharing relative to the null
expectation. Positive values indicate that k-mers are shared across
genomes more frequently than expected by chance, whereas negative values
indicate less frequent sharing. A value of zero indicates agreement with
the random-sequence model.
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$
is bounded:
$$
-1\leq D_r\leq 1.
$$
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By
construction, $D_1=0$, since all k-mers recorded in the index occur in
at least one genome. Unlike a ratio-based measure, this difference
remains defined when no k-mers are observed in at least $r$ genomes.
### Estimating the false discovery rate
The probability $P_{\mathrm{null}}(X\geq r)$ applies to one specified canonical k-mer.
To estimate the number of k-mers that could reach this sharing threshold
by chance across the entire canonical k-mer space, this probability must
be considered over all $M_k$ possible canonical k-mers.
The expected number of chance k-mers present in at least $r$ genomes is
$$
E[\mathrm{FP}(r)]=M_kP_{\mathrm{null}}(X\geq r).
$$
The observed number of k-mers passing the threshold is $n_{\geq r}$. An
estimate of the false discovery rate (FDR) for this set is therefore
$$
\widehat{\mathrm{FDR}}(r)=
\begin{cases}
1 & \text{if } n_{\geq r}=0,\\[4pt]
\displaystyle
\min\left(1,
\frac{M_k P_{\mathrm{null}}(X\geq r)}
{n_{\geq r}}\right)
& \text{otherwise}.
\end{cases}
$$
The numerator estimates the number of false positives under the null
model, while the denominator is the observed number of k-mers passing
the sharing threshold. This estimate assumes that all possible canonical
k-mers are subject to the null model. It is therefore conservative in
the sense that it does not attempt to estimate the proportion of the
k-mer space corresponding to true homology. The estimate is capped at
one because a proportion cannot exceed one.
The distinction between $D_r$ and the FDR is important. $D_r$ compares
the observed and expected proportions among k-mers already present in
the index. The FDR estimate instead compares the expected number of
chance k-mers in the entire canonical k-mer space with the observed
number passing the sharing threshold.
The smallest value of $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls
below a chosen threshold, for example 5%, can be used as a minimum
sharing threshold. This interpretation is conditional on the assumptions
of the null model and should not be taken as a model-independent
guarantee of the biological false discovery rate.
As an illustration, consider $k=21$, for which
$M_k\simeq2.2\times10^{12}$, and an index containing $S=10$ genomes.
Suppose that the numbers of distinct canonical k-mers in these genomes
range from $0.5\times10^9$ to $1.4\times10^9$, in increments of
$0.1\times10^9$. Applying the Poisson-binomial model with the
corresponding genome-specific probabilities gives the following
expected numbers of k-mers shared by chance:
| $r$ | $M_kP_{\mathrm{null}}(X\geq r)$ |
| ---: | ---: |
| 2 | $1.82\times10^7$ |
| 3 | $2.06\times10^4$ |
| 4 | $15$ |
| 5 | $0.0075$ |
In this example, chance alone can produce many pairwise shared 21-mers,
whereas the expected number of k-mers shared by five or more genomes is
very small. Whether a given sharing threshold is sufficiently stringent
depends on the number of k-mers actually observed at that level.
### Limits of the model
The FDR estimate is based on a simplified null model and has two
important limitations.
First, the model assumes independent, equiprobable nucleotides. Real
genomes exhibit nucleotide-composition biases, oligonucleotide
dependencies, and repeated sequences. These properties can increase the
probability of particular k-mers beyond the uniform expectation.
Removing low-complexity k-mers reduces some of these effects but does
not eliminate all compositional biases. Karlin and colleagues showed
that relative oligonucleotide abundances, including dinucleotide
frequencies, form characteristic genomic signatures that can differ
between organisms and remain relatively stable within genomes
[@Karlin1995-lt; @Karlin1994-on].
Second, the FDR is computed per k-mer. Consecutive k-mers extracted from
the same homologous region overlap and are therefore not independent
observations. A single homologous region can contribute many correlated
k-mers to the index. The estimate presented here consequently describes
the expected fraction of chance k-mers, not the fraction of false
homologous regions. A region-level FDR would require a model that treats
matching regions, rather than individual k-mers, as the units of
observation.