Add a new `estimate --index` command to compare observed k-mer sharing against a random-sequence model using a Poisson-binomial distribution. Introduce the `row_presence_histogram` API to support this analysis across various matrix types. Update documentation and dependencies accordingly. - Add `src/obikstats/src/null_model.rs` implementing `SharingNullModel` and `SharingRow` for computing cumulative probabilities and false discovery rates based on a Poisson-binomial distribution. - Add `src/obikstats/src/sharing.rs` defining `GenomeSharing` and `SharingSpectrum` to compute k-mer sharing spectra using parallel map-reduce. - Add `src/obikmer/src/cmd/estimate/sharing.rs` with `run` and `write_csv` to execute sharing estimation and output CSV results containing observed, expected, and deviation metrics. - Modify `src/obikmer/src/cmd/estimate/mod.rs` to support the new `--index` subcommand, adding `EstimateArgs` fields for index path, presence threshold, and output file. - Add `src/obicompactvec/src/row_histogram.rs` implementing `row_presence_histogram` for `PersistentIntMatrix`, `PersistentBitMatrix`, and sparse matrices to count rows present in exactly $j$ columns. - Update `src/obikindex/src/layer/content_layer.rs` and `src/obikindex/src/layer/typed_layer.rs` to expose the new `row_presence_histogram` method, dispatching based on layer type. - Add tests in `src/obicompactvec/src/tests/bitmatrix.rs`, `intmatrix.rs`, and `sparse.rs` to verify `count_geq` and `row_presence_histogram` against naive implementations for various matrix structures. - Update `UserDocMD/theory/kmer_indexing/kmers.md` to document the Poisson-binomial approach, $D_r$ bounds, and FDR calculations, replacing previous binomial approximations. - Update `UserDocMD/usage/estimate.md` to describe the two uses of `estimate`: parameter estimation and index comparison via `--index`. - Add `wiki` target to `Makefile` for regenerating documentation from user docs. - Bump `obikmer` package version from 1.4.5 to 1.4.6 in `Cargo.toml` and `Cargo.lock`.
13 KiB
K-mer size
A k-mer is a DNA sequence of length k. OBIKmer is designed to index
large k-mers. Here, large refers to a k-mer length for which the
probability of observing the same k-mer in several genomes by chance
alone is sufficiently small. The k-mer length must therefore be large
enough that, when a k-mer is observed in several genomes of the dataset,
the probability that this shared occurrence is due to chance alone is
sufficiently low for the match to be considered evidence of homology.
OBIKmer restricts k to odd values in the range
11 \leq k \leq 31.
These limits result from two different considerations. The upper bound is imposed by the representation used by OBIKmer, whereas the lower bound defines a practical range of k-mer lengths for genome-scale sequence comparisons. This range covers most practical applications: the lower limit is well below the k-mer size that can be considered large, even for the smallest bacterial genomes, whereas the upper limit provides sufficient specificity for most applications, including those involving very large genomes.
Representation limit
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer
requires 2k bits. To ensure computational efficiency, OBIKmer limits
the representation of k-mers to a single 64-bit word, thereby
restricting their length.
2k \leq 64,
This gives the hard representation limit: k \leq 32. Furthermore,
OBIKmer only considers odd values of k, meaning the largest supported
k-mer size is 31.
Odd k-mer lengths
OBIKmer additionally requires k to be odd. This constraint is
independent of the specificity considerations discussed below.
For an odd-length DNA sequence, a k-mer cannot be identical to its own
reverse complement. Every k-mer therefore belongs to a pair of distinct
orientations, which can be represented by a single canonical k-mer. For
even k, self-reverse-complementary sequences exist.
Odd k is not required for canonicalization itself: the canonical form
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
is well defined for both odd and even lengths. Restricting k to odd
values guarantees that the two orientations are always distinct,
simplifying the representation and orientation handling used by OBIKmer.
K-mer specificity
A simple way to illustrate the effect of k-mer size is to consider a
random DNA sequence in which the four nucleotides are independent and
equally probable. There are 4^k possible oriented k-mers, so the
expected number of occurrences of a particular oriented k-mer in a
sequence of length G is approximately G/4^k. For canonical k-mers,
the corresponding number of possible sequences is halved, and the
expected number of occurrences is approximately 2G/4^k.
Increasing k therefore reduces the probability that a particular k-mer
occurs by chance. However, the probability of a match somewhere in a
collection of sequences also depends on the amount of sequence searched
and on the number of possible k-mers considered. The following sections
formalize this distinction and describe how chance sharing can be
assessed across an indexed set of genomes.
Choosing an appropriate k-mer size
The theoretical k-mer space grows exponentially with k. Thus, increasing k from 11 to 31 greatly expands the sequence space and reduces the probability that unrelated sequences share a particular k-mer by chance. The following table gives the size of the theoretical k-mer space and, after canonicalization, the number of distinct representations.
k |
4^k |
4^k/2 after canonicalization |
|---|---|---|
| 11 | 4.19\times10^6 |
2.10\times10^6 |
| 15 | 1.07\times10^9 |
5.37\times10^8 |
| 21 | 4.40\times10^{12} |
2.20\times10^{12} |
| 25 | 1.13\times10^{15} |
5.63\times10^{14} |
| 31 | 4.61\times10^{18} |
2.31\times10^{18} |
Genome sizes illustrate the range of sequence scales for which k-mers may be used. Bacterial genomes are typically a few megabases long, although some symbiotic bacteria have much smaller genomes, while other bacteria have genomes exceeding ten megabases. Plant genomes span an even wider range, from tens of megabases to more than one hundred gigabases. In very large and repeat-rich genomes, the main limitation is not necessarily the exhaustion of the theoretical k-mer space, but the non-uniform distribution of k-mers caused by sequence composition and repeated DNA.
The supported range 11\leq k\leq31 should therefore be understood as an engineering and application-oriented compromise, not as a universal biological optimum. Shorter k-mers are more likely to be shared by unrelated regions, whereas longer k-mers provide greater sequence specificity. However, longer k-mers are also more sensitive to substitutions and indels: a single difference can disrupt an exact match across the whole k-mer. The appropriate value within the supported range consequently depends on the size and characteristics of the genomes being compared, as well as on the biological question.
The following section introduces a random-sequence model to quantify the probability of chance sharing across an indexed set of genomes. This provides a formal basis for assessing how the choice of k affects the interpretation of shared k-mers.
Chance sharing across genomes
OBIKmer indexes distinct canonical k-mers and records the genomes in which each k-mer occurs. This makes it possible to compare the observed sharing of k-mers with the sharing expected under a random-sequence model. The same probability model can be used for two related purposes: measuring the excess of sharing among k-mers present in the index, and estimating the number of chance k-mers among those shared by at least a specified number of genomes.
The null model and genome-specific probabilities
The null hypothesis assumes that the S indexed genomes are unrelated
and that each genome is an independent random sequence in which
nucleotides are independent and equiprobable.
Since OBIKmer uses odd-length canonical k-mers, reverse-complement canonicalization reduces the number of possible k-mers by half. The size of the canonical k-mer space is therefore
M_k=\frac{4^k}{2}.
Let N_i be the number of distinct canonical k-mers recorded for genome
i in the index. Under the null model, these can be regarded as a
uniform sample without replacement from the M_k possible canonical
k-mers. The probability that a given canonical k-mer is present in
genome i is then
p_i=\frac{N_i}{M_k}=\frac{2N_i}{4^k}.
This is the inclusion probability for a distinct k-mer. It differs from the probability of at least one occurrence in a positional model, because repeated occurrences within a genome are not counted separately.
Let X denote the number of genomes, among the S indexed genomes,
that contain a given canonical k-mer under the null model. Since genomes
are assumed to be independent, X is a sum of independent presence
indicators with probabilities p_1,\ldots,p_S. When these probabilities
differ, X follows a Poisson-binomial distribution.
Computing the Poisson-binomial distribution
The distribution of X can be computed by processing genomes one at a
time. Let P_i(j) denote the probability that the k-mer occurs in
exactly j of the first i genomes. Initially,
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
When genome i is added, the k-mer is either absent, with probability
1-p_i, or present, with probability p_i. Thus,
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
This recurrence avoids enumerating the subsets of genomes. Computing the
complete distribution requires O(S^2) operations and O(S) memory. To
compute the upper-tail probability, only the probabilities for fewer
than r presences need to be tracked:
P_{\mathrm{null}}(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
The computation can therefore be restricted to P_i(0),\ldots,P_i(r-1),
requiring O(Sr) operations and O(r) memory. The resulting
distribution depends on the genome-specific values N_i, but not on the
identity of the k-mer, and can thus be computed once for a given index
and k-mer length.
Comparing observed and expected sharing in the index
The index contains only k-mers observed in at least one genome. Consequently, when comparing sharing proportions among the k-mers recorded in the index, the relevant null probability is conditional on at least one presence:
q_r=P(X\geq r\mid X\geq 1)
=\frac{P_{\mathrm{null}}(X\geq r)}{P_{\mathrm{null}}(X\geq 1)}.
Let n_{\geq r} be the number of distinct canonical k-mers observed in
at least r genomes, and let n_{\geq 1} be the total number of
distinct canonical k-mers recorded in the index. The observed proportion
of indexed k-mers shared by at least r genomes is
O_r=\frac{n_{\geq r}}{n_{\geq 1}}.
Under the random-sequence model, the expected proportion is q_r. The
difference
D_r=O_r-q_r
therefore measures the excess or deficit of sharing relative to the null expectation. Positive values indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.
Since both O_r and q_r are proportions between zero and one, D_r
is bounded:
-1\leq D_r\leq 1.
More precisely, for a given q_r, its bounds are -q_r and 1-q_r. By
construction, D_1=0, since all k-mers recorded in the index occur in
at least one genome. Unlike a ratio-based measure, this difference
remains defined when no k-mers are observed in at least r genomes.
Estimating the false discovery rate
The probability P_{\mathrm{null}}(X\geq r) applies to one specified canonical k-mer.
To estimate the number of k-mers that could reach this sharing threshold
by chance across the entire canonical k-mer space, this probability must
be considered over all M_k possible canonical k-mers.
The expected number of chance k-mers present in at least r genomes is
E[\mathrm{FP}(r)]=M_kP_{\mathrm{null}}(X\geq r).
The observed number of k-mers passing the threshold is n_{\geq r}. An
estimate of the false discovery rate (FDR) for this set is therefore
\widehat{\mathrm{FDR}}(r)=
\begin{cases}
1 & \text{if } n_{\geq r}=0,\\[4pt]
\displaystyle
\min\left(1,
\frac{M_k P_{\mathrm{null}}(X\geq r)}
{n_{\geq r}}\right)
& \text{otherwise}.
\end{cases}
The numerator estimates the number of false positives under the null model, while the denominator is the observed number of k-mers passing the sharing threshold. This estimate assumes that all possible canonical k-mers are subject to the null model. It is therefore conservative in the sense that it does not attempt to estimate the proportion of the k-mer space corresponding to true homology. The estimate is capped at one because a proportion cannot exceed one.
The distinction between D_r and the FDR is important. D_r compares
the observed and expected proportions among k-mers already present in
the index. The FDR estimate instead compares the expected number of
chance k-mers in the entire canonical k-mer space with the observed
number passing the sharing threshold.
The smallest value of r for which \widehat{\mathrm{FDR}}(r) falls
below a chosen threshold, for example 5%, can be used as a minimum
sharing threshold. This interpretation is conditional on the assumptions
of the null model and should not be taken as a model-independent
guarantee of the biological false discovery rate.
As an illustration, consider k=21, for which
M_k\simeq2.2\times10^{12}, and an index containing S=10 genomes.
Suppose that the numbers of distinct canonical k-mers in these genomes
range from 0.5\times10^9 to 1.4\times10^9, in increments of
0.1\times10^9. Applying the Poisson-binomial model with the
corresponding genome-specific probabilities gives the following
expected numbers of k-mers shared by chance:
r |
M_kP_{\mathrm{null}}(X\geq r) |
|---|---|
| 2 | 1.82\times10^7 |
| 3 | 2.06\times10^4 |
| 4 | 15 |
| 5 | 0.0075 |
In this example, chance alone can produce many pairwise shared 21-mers, whereas the expected number of k-mers shared by five or more genomes is very small. Whether a given sharing threshold is sufficiently stringent depends on the number of k-mers actually observed at that level.
Limits of the model
The FDR estimate is based on a simplified null model and has two important limitations.
First, the model assumes independent, equiprobable nucleotides. Real genomes exhibit nucleotide-composition biases, oligonucleotide dependencies, and repeated sequences. These properties can increase the probability of particular k-mers beyond the uniform expectation. Removing low-complexity k-mers reduces some of these effects but does not eliminate all compositional biases. Karlin and colleagues showed that relative oligonucleotide abundances, including dinucleotide frequencies, form characteristic genomic signatures that can differ between organisms and remain relatively stable within genomes [@Karlin1995-lt; @Karlin1994-on].
Second, the FDR is computed per k-mer. Consecutive k-mers extracted from the same homologous region overlap and are therefore not independent observations. A single homologous region can contribute many correlated k-mers to the index. The estimate presented here consequently describes the expected fraction of chance k-mers, not the fraction of false homologous regions. A region-level FDR would require a model that treats matching regions, rather than individual k-mers, as the units of observation.