Add a new `estimate --index` command to compare observed k-mer sharing against a random-sequence model using a Poisson-binomial distribution. Introduce the `row_presence_histogram` API to support this analysis across various matrix types. Update documentation and dependencies accordingly. - Add `src/obikstats/src/null_model.rs` implementing `SharingNullModel` and `SharingRow` for computing cumulative probabilities and false discovery rates based on a Poisson-binomial distribution. - Add `src/obikstats/src/sharing.rs` defining `GenomeSharing` and `SharingSpectrum` to compute k-mer sharing spectra using parallel map-reduce. - Add `src/obikmer/src/cmd/estimate/sharing.rs` with `run` and `write_csv` to execute sharing estimation and output CSV results containing observed, expected, and deviation metrics. - Modify `src/obikmer/src/cmd/estimate/mod.rs` to support the new `--index` subcommand, adding `EstimateArgs` fields for index path, presence threshold, and output file. - Add `src/obicompactvec/src/row_histogram.rs` implementing `row_presence_histogram` for `PersistentIntMatrix`, `PersistentBitMatrix`, and sparse matrices to count rows present in exactly $j$ columns. - Update `src/obikindex/src/layer/content_layer.rs` and `src/obikindex/src/layer/typed_layer.rs` to expose the new `row_presence_histogram` method, dispatching based on layer type. - Add tests in `src/obicompactvec/src/tests/bitmatrix.rs`, `intmatrix.rs`, and `sparse.rs` to verify `count_geq` and `row_presence_histogram` against naive implementations for various matrix structures. - Update `UserDocMD/theory/kmer_indexing/kmers.md` to document the Poisson-binomial approach, $D_r$ bounds, and FDR calculations, replacing previous binomial approximations. - Update `UserDocMD/usage/estimate.md` to describe the two uses of `estimate`: parameter estimation and index comparison via `--index`. - Add `wiki` target to `Makefile` for regenerating documentation from user docs. - Bump `obikmer` package version from 1.4.5 to 1.4.6 in `Cargo.toml` and `Cargo.lock`.
310 lines
13 KiB
Markdown
310 lines
13 KiB
Markdown
# K-mer size
|
|
|
|
A k-mer is a DNA sequence of length $k$. OBIKmer is designed to index
|
|
large k-mers. Here, *large* refers to a k-mer length for which the
|
|
probability of observing the same k-mer in several genomes by chance
|
|
alone is sufficiently small. The k-mer length must therefore be large
|
|
enough that, when a k-mer is observed in several genomes of the dataset,
|
|
the probability that this shared occurrence is due to chance alone is
|
|
sufficiently low for the match to be considered evidence of homology.
|
|
|
|
OBIKmer restricts $k$ to odd values in the range
|
|
|
|
$$
|
|
11 \leq k \leq 31.
|
|
$$
|
|
|
|
These limits result from two different considerations. The upper bound
|
|
is imposed by the representation used by OBIKmer, whereas the lower
|
|
bound defines a practical range of k-mer lengths for genome-scale
|
|
sequence comparisons. This range covers most practical applications: the
|
|
lower limit is well below the k-mer size that can be considered large,
|
|
even for the smallest bacterial genomes, whereas the upper limit
|
|
provides sufficient specificity for most applications, including those
|
|
involving very large genomes.
|
|
|
|
## Representation limit
|
|
|
|
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer
|
|
requires $2k$ bits. To ensure computational efficiency, OBIKmer limits
|
|
the representation of k-mers to a single 64-bit word, thereby
|
|
restricting their length.
|
|
|
|
$$
|
|
2k \leq 64,
|
|
$$
|
|
|
|
This gives the hard representation limit: $k \leq 32.$ Furthermore,
|
|
OBIKmer only considers odd values of $k$, meaning the largest supported
|
|
k-mer size is $31$.
|
|
|
|
## Odd k-mer lengths
|
|
|
|
OBIKmer additionally requires $k$ to be odd. This constraint is
|
|
independent of the specificity considerations discussed below.
|
|
|
|
For an odd-length DNA sequence, a k-mer cannot be identical to its own
|
|
reverse complement. Every k-mer therefore belongs to a pair of distinct
|
|
orientations, which can be represented by a single canonical k-mer. For
|
|
even $k$, self-reverse-complementary sequences exist.
|
|
|
|
Odd $k$ is not required for canonicalization itself: the canonical form
|
|
|
|
$$
|
|
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
|
|
$$
|
|
|
|
is well defined for both odd and even lengths. Restricting $k$ to odd
|
|
values guarantees that the two orientations are always distinct,
|
|
simplifying the representation and orientation handling used by OBIKmer.
|
|
|
|
## K-mer specificity
|
|
|
|
A simple way to illustrate the effect of k-mer size is to consider a
|
|
random DNA sequence in which the four nucleotides are independent and
|
|
equally probable. There are $4^k$ possible oriented k-mers, so the
|
|
expected number of occurrences of a particular oriented k-mer in a
|
|
sequence of length $G$ is approximately $G/4^k$. For canonical k-mers,
|
|
the corresponding number of possible sequences is halved, and the
|
|
expected number of occurrences is approximately $2G/4^k$.
|
|
|
|
Increasing $k$ therefore reduces the probability that a particular k-mer
|
|
occurs by chance. However, the probability of a match somewhere in a
|
|
collection of sequences also depends on the amount of sequence searched
|
|
and on the number of possible k-mers considered. The following sections
|
|
formalize this distinction and describe how chance sharing can be
|
|
assessed across an indexed set of genomes.
|
|
|
|
## Choosing an appropriate k-mer size
|
|
|
|
The theoretical k-mer space grows exponentially with $k$. Thus, increasing $k$ from 11 to 31 greatly expands the sequence space and reduces the probability that unrelated sequences share a particular k-mer by chance. The following table gives the size of the theoretical k-mer space and, after canonicalization, the number of distinct representations.
|
|
|
|
| $k$ | $4^k$ | $4^k/2$ after canonicalization |
|
|
|---:|---:|---:|
|
|
| 11 | $4.19\times10^6$ | $2.10\times10^6$ |
|
|
| 15 | $1.07\times10^9$ | $5.37\times10^8$ |
|
|
| 21 | $4.40\times10^{12}$ | $2.20\times10^{12}$ |
|
|
| 25 | $1.13\times10^{15}$ | $5.63\times10^{14}$ |
|
|
| 31 | $4.61\times10^{18}$ | $2.31\times10^{18}$ |
|
|
|
|
Genome sizes illustrate the range of sequence scales for which k-mers may be used. Bacterial genomes are typically a few megabases long, although some symbiotic bacteria have much smaller genomes, while other bacteria have genomes exceeding ten megabases. Plant genomes span an even wider range, from tens of megabases to more than one hundred gigabases. In very large and repeat-rich genomes, the main limitation is not necessarily the exhaustion of the theoretical k-mer space, but the non-uniform distribution of k-mers caused by sequence composition and repeated DNA.
|
|
|
|
The supported range $11\leq k\leq31$ should therefore be understood as an engineering and application-oriented compromise, not as a universal biological optimum. Shorter k-mers are more likely to be shared by unrelated regions, whereas longer k-mers provide greater sequence specificity. However, longer k-mers are also more sensitive to substitutions and indels: a single difference can disrupt an exact match across the whole k-mer. The appropriate value within the supported range consequently depends on the size and characteristics of the genomes being compared, as well as on the biological question.
|
|
|
|
The following section introduces a random-sequence model to quantify the probability of chance sharing across an indexed set of genomes. This provides a formal basis for assessing how the choice of $k$ affects the interpretation of shared k-mers.
|
|
|
|
## Chance sharing across genomes
|
|
|
|
OBIKmer indexes distinct canonical k-mers and records the genomes in
|
|
which each k-mer occurs. This makes it possible to compare the observed
|
|
sharing of k-mers with the sharing expected under a random-sequence
|
|
model. The same probability model can be used for two related purposes:
|
|
measuring the excess of sharing among k-mers present in the index, and
|
|
estimating the number of chance k-mers among those shared by at least a
|
|
specified number of genomes.
|
|
|
|
### The null model and genome-specific probabilities
|
|
|
|
The null hypothesis assumes that the $S$ indexed genomes are unrelated
|
|
and that each genome is an independent random sequence in which
|
|
nucleotides are independent and equiprobable.
|
|
|
|
Since OBIKmer uses odd-length canonical k-mers, reverse-complement
|
|
canonicalization reduces the number of possible k-mers by half. The size
|
|
of the canonical k-mer space is therefore
|
|
|
|
$$
|
|
M_k=\frac{4^k}{2}.
|
|
$$
|
|
|
|
Let $N_i$ be the number of distinct canonical k-mers recorded for genome
|
|
$i$ in the index. Under the null model, these can be regarded as a
|
|
uniform sample without replacement from the $M_k$ possible canonical
|
|
k-mers. The probability that a given canonical k-mer is present in
|
|
genome $i$ is then
|
|
|
|
$$
|
|
p_i=\frac{N_i}{M_k}=\frac{2N_i}{4^k}.
|
|
$$
|
|
|
|
This is the inclusion probability for a distinct k-mer. It differs from
|
|
the probability of at least one occurrence in a positional model,
|
|
because repeated occurrences within a genome are not counted separately.
|
|
|
|
Let $X$ denote the number of genomes, among the $S$ indexed genomes,
|
|
that contain a given canonical k-mer under the null model. Since genomes
|
|
are assumed to be independent, $X$ is a sum of independent presence
|
|
indicators with probabilities $p_1,\ldots,p_S$. When these probabilities
|
|
differ, $X$ follows a Poisson-binomial distribution.
|
|
|
|
### Computing the Poisson-binomial distribution
|
|
|
|
The distribution of $X$ can be computed by processing genomes one at a
|
|
time. Let $P_i(j)$ denote the probability that the k-mer occurs in
|
|
exactly $j$ of the first $i$ genomes. Initially,
|
|
|
|
$$
|
|
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
|
|
$$
|
|
|
|
When genome $i$ is added, the k-mer is either absent, with probability
|
|
$1-p_i$, or present, with probability $p_i$. Thus,
|
|
|
|
$$
|
|
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
|
|
$$
|
|
|
|
This recurrence avoids enumerating the subsets of genomes. Computing the
|
|
complete distribution requires $O(S^2)$ operations and $O(S)$ memory. To
|
|
compute the upper-tail probability, only the probabilities for fewer
|
|
than $r$ presences need to be tracked:
|
|
|
|
$$
|
|
P_{\mathrm{null}}(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
|
|
$$
|
|
|
|
The computation can therefore be restricted to $P_i(0),\ldots,P_i(r-1)$,
|
|
requiring $O(Sr)$ operations and $O(r)$ memory. The resulting
|
|
distribution depends on the genome-specific values $N_i$, but not on the
|
|
identity of the k-mer, and can thus be computed once for a given index
|
|
and k-mer length.
|
|
|
|
### Comparing observed and expected sharing in the index
|
|
|
|
The index contains only k-mers observed in at least one genome.
|
|
Consequently, when comparing sharing proportions among the k-mers
|
|
recorded in the index, the relevant null probability is conditional on
|
|
at least one presence:
|
|
|
|
$$
|
|
q_r=P(X\geq r\mid X\geq 1)
|
|
=\frac{P_{\mathrm{null}}(X\geq r)}{P_{\mathrm{null}}(X\geq 1)}.
|
|
$$
|
|
|
|
Let $n_{\geq r}$ be the number of distinct canonical k-mers observed in
|
|
at least $r$ genomes, and let $n_{\geq 1}$ be the total number of
|
|
distinct canonical k-mers recorded in the index. The observed proportion
|
|
of indexed k-mers shared by at least $r$ genomes is
|
|
|
|
$$
|
|
O_r=\frac{n_{\geq r}}{n_{\geq 1}}.
|
|
$$
|
|
|
|
Under the random-sequence model, the expected proportion is $q_r$. The
|
|
difference
|
|
|
|
$$
|
|
D_r=O_r-q_r
|
|
$$
|
|
|
|
therefore measures the excess or deficit of sharing relative to the null
|
|
expectation. Positive values indicate that k-mers are shared across
|
|
genomes more frequently than expected by chance, whereas negative values
|
|
indicate less frequent sharing. A value of zero indicates agreement with
|
|
the random-sequence model.
|
|
|
|
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$
|
|
is bounded:
|
|
|
|
$$
|
|
-1\leq D_r\leq 1.
|
|
$$
|
|
|
|
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By
|
|
construction, $D_1=0$, since all k-mers recorded in the index occur in
|
|
at least one genome. Unlike a ratio-based measure, this difference
|
|
remains defined when no k-mers are observed in at least $r$ genomes.
|
|
|
|
### Estimating the false discovery rate
|
|
|
|
The probability $P_{\mathrm{null}}(X\geq r)$ applies to one specified canonical k-mer.
|
|
To estimate the number of k-mers that could reach this sharing threshold
|
|
by chance across the entire canonical k-mer space, this probability must
|
|
be considered over all $M_k$ possible canonical k-mers.
|
|
|
|
The expected number of chance k-mers present in at least $r$ genomes is
|
|
|
|
$$
|
|
E[\mathrm{FP}(r)]=M_kP_{\mathrm{null}}(X\geq r).
|
|
$$
|
|
|
|
The observed number of k-mers passing the threshold is $n_{\geq r}$. An
|
|
estimate of the false discovery rate (FDR) for this set is therefore
|
|
|
|
$$
|
|
\widehat{\mathrm{FDR}}(r)=
|
|
\begin{cases}
|
|
1 & \text{if } n_{\geq r}=0,\\[4pt]
|
|
\displaystyle
|
|
\min\left(1,
|
|
\frac{M_k P_{\mathrm{null}}(X\geq r)}
|
|
{n_{\geq r}}\right)
|
|
& \text{otherwise}.
|
|
\end{cases}
|
|
$$
|
|
|
|
The numerator estimates the number of false positives under the null
|
|
model, while the denominator is the observed number of k-mers passing
|
|
the sharing threshold. This estimate assumes that all possible canonical
|
|
k-mers are subject to the null model. It is therefore conservative in
|
|
the sense that it does not attempt to estimate the proportion of the
|
|
k-mer space corresponding to true homology. The estimate is capped at
|
|
one because a proportion cannot exceed one.
|
|
|
|
The distinction between $D_r$ and the FDR is important. $D_r$ compares
|
|
the observed and expected proportions among k-mers already present in
|
|
the index. The FDR estimate instead compares the expected number of
|
|
chance k-mers in the entire canonical k-mer space with the observed
|
|
number passing the sharing threshold.
|
|
|
|
The smallest value of $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls
|
|
below a chosen threshold, for example 5%, can be used as a minimum
|
|
sharing threshold. This interpretation is conditional on the assumptions
|
|
of the null model and should not be taken as a model-independent
|
|
guarantee of the biological false discovery rate.
|
|
|
|
As an illustration, consider $k=21$, for which
|
|
$M_k\simeq2.2\times10^{12}$, and an index containing $S=10$ genomes.
|
|
Suppose that the numbers of distinct canonical k-mers in these genomes
|
|
range from $0.5\times10^9$ to $1.4\times10^9$, in increments of
|
|
$0.1\times10^9$. Applying the Poisson-binomial model with the
|
|
corresponding genome-specific probabilities gives the following
|
|
expected numbers of k-mers shared by chance:
|
|
|
|
| $r$ | $M_kP_{\mathrm{null}}(X\geq r)$ |
|
|
| ---: | ---: |
|
|
| 2 | $1.82\times10^7$ |
|
|
| 3 | $2.06\times10^4$ |
|
|
| 4 | $15$ |
|
|
| 5 | $0.0075$ |
|
|
|
|
In this example, chance alone can produce many pairwise shared 21-mers,
|
|
whereas the expected number of k-mers shared by five or more genomes is
|
|
very small. Whether a given sharing threshold is sufficiently stringent
|
|
depends on the number of k-mers actually observed at that level.
|
|
|
|
### Limits of the model
|
|
|
|
The FDR estimate is based on a simplified null model and has two
|
|
important limitations.
|
|
|
|
First, the model assumes independent, equiprobable nucleotides. Real
|
|
genomes exhibit nucleotide-composition biases, oligonucleotide
|
|
dependencies, and repeated sequences. These properties can increase the
|
|
probability of particular k-mers beyond the uniform expectation.
|
|
Removing low-complexity k-mers reduces some of these effects but does
|
|
not eliminate all compositional biases. Karlin and colleagues showed
|
|
that relative oligonucleotide abundances, including dinucleotide
|
|
frequencies, form characteristic genomic signatures that can differ
|
|
between organisms and remain relatively stable within genomes
|
|
[@Karlin1995-lt; @Karlin1994-on].
|
|
|
|
Second, the FDR is computed per k-mer. Consecutive k-mers extracted from
|
|
the same homologous region overlap and are therefore not independent
|
|
observations. A single homologous region can contribute many correlated
|
|
k-mers to the index. The estimate presented here consequently describes
|
|
the expected fraction of chance k-mers, not the fraction of false
|
|
homologous regions. A region-level FDR would require a model that treats
|
|
matching regions, rather than individual k-mers, as the units of
|
|
observation.
|