Push punmmmpkuymy #83

Merged
coissac merged 4 commits from push-punmmmpkuymy into main 2026-11-20 13:24:07 +00:00
17 changed files with 948 additions and 356 deletions
Showing only changes of commit 36c7247acb - Show all commits

No files matched your search

+11
View File
@@ -91,6 +91,17 @@ doc-user-serve: $(MKDOCS)
clean-doc-user:
rm -rf $(DOC_USER_SITE)/
# Regenerates the Gitea/Forgejo wiki (a separate repository, checked out next
# to this one) from the user documentation. Only rewrites its working tree:
# committing and pushing it stays a manual step. Not part of `clean`.
DOC_WIKI_DIR := ../obikmer.wiki
.PHONY: wiki
wiki:
@command -v pmotransdoc >/dev/null || { echo "pmotransdoc not found in PATH" >&2; exit 1; }
@test -d $(DOC_WIKI_DIR)/.git || { echo "$(DOC_WIKI_DIR) is not a git checkout of the wiki" >&2; exit 1; }
pmotransdoc $(DOC_USER_DIR) $(DOC_WIKI_DIR)
.PHONY: clean
clean: clean-doc clean-doc-user
rm -rf $(VENV)
+204 -352
View File
@@ -1,6 +1,12 @@
# K-mer size
A k-mer is a DNA sequence of length $k$. OBIKmer is designed to index large k-mers. Here, *large* refers to a k-mer length for which the probability of observing the same k-mer in several genomes by chance alone is sufficiently small. The k-mer length must therefore be large enough that, when a k-mer is observed in several genomes of the dataset, the probability that this shared occurrence is due to chance alone is sufficiently low for the match to be considered evidence of homology.
A k-mer is a DNA sequence of length $k$. OBIKmer is designed to index
large k-mers. Here, *large* refers to a k-mer length for which the
probability of observing the same k-mer in several genomes by chance
alone is sufficiently small. The k-mer length must therefore be large
enough that, when a k-mer is observed in several genomes of the dataset,
the probability that this shared occurrence is due to chance alone is
sufficiently low for the match to be considered evidence of homology.
OBIKmer restricts $k$ to odd values in the range
@@ -8,450 +14,296 @@ $$
11 \leq k \leq 31.
$$
These limits result from two different considerations. The upper bound is imposed by the representation used by OBIKmer, whereas the lower bound defines a practical range of k-mer lengths for genome-scale sequence comparisons. This range covers most practical applications: the lower limit is well below the k-mer size that can be considered large, even for the smallest bacterial genomes, whereas the upper limit provides sufficient specificity for most applications, including those involving very large genomes.
These limits result from two different considerations. The upper bound
is imposed by the representation used by OBIKmer, whereas the lower
bound defines a practical range of k-mer lengths for genome-scale
sequence comparisons. This range covers most practical applications: the
lower limit is well below the k-mer size that can be considered large,
even for the smallest bacterial genomes, whereas the upper limit
provides sufficient specificity for most applications, including those
involving very large genomes.
## Representation limit
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer requires $2k$ bits. To ensure computational efficiency, OBIKmer limits the representation of k-mers to a single 64-bit word, thereby restricting their length.
OBIKmer uses two bits to encode each nucleotide. Therefore, a k-mer
requires $2k$ bits. To ensure computational efficiency, OBIKmer limits
the representation of k-mers to a single 64-bit word, thereby
restricting their length.
$$
2k \leq 64,
$$
This gives the hard representation limit: $k \leq 32.$ Furthermore, OBIKmer only considers odd values of $k$, meaning the largest supported k-mer size is $31$.
This gives the hard representation limit: $k \leq 32.$ Furthermore,
OBIKmer only considers odd values of $k$, meaning the largest supported
k-mer size is $31$.
### K-mer specificity
## Odd k-mer lengths
One simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are $4^k$ possible k-mers, so a particular k-mer is expected to occur once every $4^k$ positions on average. In a sequence of length $G$, the expected number of occurrences of a particular k-mer is approximately $G \cdot 4^{-k}$. When k is large enough to ensure $4^k >> G$, the chance of a second occurrence of the same k-mer becomes close to zero. Consequently, observing the same k-mer twice can only be explained by a homology relationship, except for low-complexity k-mers, which are overrepresented in genomes.
OBIKmer additionally requires $k$ to be odd. This constraint is
independent of the specificity considerations discussed below.
### Estimating False discovery rate on homology inference
For an odd-length DNA sequence, a k-mer cannot be identical to its own
reverse complement. Every k-mer therefore belongs to a pair of distinct
orientations, which can be represented by a single canonical k-mer. For
even $k$, self-reverse-complementary sequences exist.
OBIKmer uses k-mers long enough that their occurrence in multiple indexed genomes provides evidence of homology. The corresponding false discovery rate (FDR) can be expressed in terms of the probability of observing a k-mer in at least $R$ of the $S$ indexed genomes under a purely random model.
#### The random model
The random model, or null hypothesis, assumes that the $S$ genomes are unrelated and that each genome is an independent random sequence in which each nucleotide occurs with equal probability. Rejecting this hypothesis for a given k-mer indicates that its presence in multiple genomes is unlikely to be explained by chance alone, providing evidence for homology.
For a genome of size $G$, the number of occurrences of a given canonical k-mer $K$ is approximated by a Poisson distribution with parameter
$$ \lambda = 2\frac{G}{4^k}, $$
which is also the expected number of occurrences under the random model. The factor of two accounts for k-mer canonicalisation, which halves the k-mer space when $k$ is odd.
The probability of observing at least one occurrence of this k-mer is therefore
Odd $k$ is not required for canonicalization itself: the canonical form
$$
\begin{aligned}
P(n_K > 0) &= 1-e^{-\lambda} \\
&= 1-e^{-2G/4^k}.
\end{aligned}
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
$$
##### Assuming equal genome lengths
is well defined for both odd and even lengths. Restricting $k$ to odd
values guarantees that the two orientations are always distinct,
simplifying the representation and orientation handling used by OBIKmer.
The number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
## K-mer specificity
A simple way to illustrate the effect of k-mer size is to consider a
random DNA sequence in which the four nucleotides are independent and
equally probable. There are $4^k$ possible oriented k-mers, so the
expected number of occurrences of a particular oriented k-mer in a
sequence of length $G$ is approximately $G/4^k$. For canonical k-mers,
the corresponding number of possible sequences is halved, and the
expected number of occurrences is approximately $2G/4^k$.
$$
P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right)^{S-j}.
$$
Increasing $k$ therefore reduces the probability that a particular k-mer
occurs by chance. However, the probability of a match somewhere in a
collection of sequences also depends on the amount of sequence searched
and on the number of possible k-mers considered. The following sections
formalize this distinction and describe how chance sharing can be
assessed across an indexed set of genomes.
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
## Choosing an appropriate k-mer size
$$
I_p(r,S-r+1)
=
\frac{1}{B(r,S-r+1)}
\int_0^p t^{r-1}(1-t)^{S-r}\,dt
$$
and
$$
p = 1-e^{-2G/4^k}.
$$
the regularized incomplete beta function is available in standard statistical libraries.
##### Different genome sizes
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
$$ \bar G = \frac{1}{S}\sum_{i=1}^{S}G_i. $$
This gives a common probability
$$ \bar p = 1-e^{-2\bar G/4^k}, $$
and the number of genomes containing the k-mer can then be approximated by a binomial distribution:
$$ P(X\geq r)=\sum_{j=r}^{S}\binom{S}{j}\bar p^j(1-\bar p)^{S-j}. $$
This approximation is conservative for the upper tail. Hoeffding [-@Hoeffding1956-bx, p. 717, Eq. 26] shows that, for a fixed mean, the binomial upper tail is at least as large as the corresponding Poisson-binomial tail for
$$ r\geq S\bar p+1. $$
Using the mean genome size therefore provides a conservative estimate of the FDR.
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
##### Exact tail probability
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
Let $P_i(j)$ denote the probability that the k-mer occurs in exactly $j$ of the first $i$ genomes. Initially,
$$
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
$$
When genome $i$ is added, the k-mer is either absent, with probability $1-p_i$, or present, with probability $p_i$. The probability of observing exactly $j$ presences among the first $i$ genomes is therefore
$$
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
$$
This dynamic programming algorithm requires $O(S^2)$ operations and $O(S)$ memory. However, to compute $P(X\geq r)$, only the probabilities for fewer than $r$ presences need to be tracked, since
$$
P(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
$$
This reduces the computational cost to $O(Sr)$ operations and $O(r)$ memory.
#### Distinguishing random from non-random k-mer sharing
The probability $P(X\geq r)$ describes the occurrence of a given k-mer in at least $r$ genomes under the random-sequence model. However, the index contains only k-mers observed in at least one genome. Therefore, when considering the k-mers recorded in the index, the relevant probability is the conditional probability:
$$
q_r = P(X\geq r\mid X\geq 1)
= \frac{P(X\geq r)}{P(X\geq 1)}.
$$
Let $k_r$ be the number of distinct k-mers observed in at least $r$ genomes, among the $k_1$ distinct k-mers recorded in the index. The observed proportion of k-mers shared by at least $r$ genomes is:
$$
O_r = \frac{k_r}{k_1}.
$$
Under the random-sequence model, the expected proportion is $q_r$. We can therefore quantify the deviation from random expectations using the difference:
$$
\begin{aligned}
D_r &= O_r-q_r \\
&= \frac{k_r}{k_1}-q_r.
\end{aligned}
$$
Positive values of $D_r$ indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$ is bounded:
$$
-1\leq D_r\leq 1.
$$
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By construction, $D_1=0$, since all k-mers recorded in the index occur in at least one genome. Unlike a ratio-based measure, this difference remains defined even when no k-mers are observed in at least $r$ genomes. It thus provides a bounded measure of the excess or deficit of k-mer sharing relative to the random-sequence model.
*******
This probability decreases by a factor of four for every additional nucleotide in the k-mer. Conversely, it increases linearly with the amount of sequence searched.
This simple model already shows why very short k-mers provide limited specificity for genome-scale analyses. With $k=11$,
$$
4^{11}\simeq4.2\times10^6,
$$
which is a small combinatorial space even compared with the size of very small bacterial genomes. In contrast,
$$
4^{31}\simeq4.6\times10^{18},
$$
which is vastly larger than the number of k-mer positions even in very large genomes. Thus, increasing $k$ from 11 to 31 provides an enormous increase in sequence specificity.
The estimate above is intentionally simplistic. Real genomes do not consist of independent, equiprobable nucleotides: nucleotide composition and sequence dependencies affect the probability of observing particular words. More accurate estimates of the probability of observing matches by chance therefore take sequence length and nucleotide composition into account.
Karlin and Ost provide an analytical framework for this problem based on the distribution of the longest exact match expected between random sequences. If $L_{\max}$ denotes the length of the longest seed observed in random sequences of given size and nucleotide composition, the corresponding seed-level probability is
$$
P_{\mathrm{seeds}}(L)=P(L_{\max}\geq L).
$$
Conversely, for a chosen false-positive probability $P$, one can determine the minimum seed length $L_{\min}$ such that matches of length $L_{\min}$ or greater are not expected to occur by chance. This is the approach used by Repseek to determine statistically significant seed lengths.
The simple $G/4^k$ estimate and the Karlin–Ost statistics therefore describe the same fundamental effect at different levels of approximation: the probability of a chance match decreases rapidly with k-mer length, while the amount of sequence being searched increases the opportunity for such matches.
Consequently, the appropriate value of $k$ depends on the total amount of sequence being compared and its nucleotide composition. Choosing $k$ appropriately allows the false-positive rate of homology inference to be controlled.
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Nasuia deltocephalinicola* having a size $G=112\,Kb$ [@Bennett2013-cl], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
$$
\begin{aligned}
P(C\geq1) &= 1- e^{-\frac{112\,000^2}{2 \cdot 4^{11}}} \\
&= 1- e^{-1.5\times10^{3}} \\
&\simeq 1
\end{aligned}
$$
, whereas 21-mers already provide a space of several trillion possible sequences.
Plant genomes illustrate the opposite situation. Plant genome sizes span several orders of magnitude, from the smallest angiosperm genomes present in the carnivorous plans genus [@Fleischmann2014-ed] about 60 Mb to the largest one 160 Gb in Tmesipteris oblanceolata, a New Caledonian fork fern [@Fernandez2024-kd], and large genomes often contain substantial amounts of repeated sequence. Even at $k=31$, however, the theoretical k-mer space remains vastly larger than any individual genome. At these lengths, the limiting factor is therefore not exhaustion of the theoretical k-mer space but the biological repetitiveness and non-random composition of the genome.
For example, the number of possible k-mers grows as follows:
The theoretical k-mer space grows exponentially with $k$. Thus, increasing $k$ from 11 to 31 greatly expands the sequence space and reduces the probability that unrelated sequences share a particular k-mer by chance. The following table gives the size of the theoretical k-mer space and, after canonicalization, the number of distinct representations.
| $k$ | $4^k$ | $4^k/2$ after canonicalization |
| ----: | --------------------: | -------------------------------: |
|---:|---:|---:|
| 11 | $4.19\times10^6$ | $2.10\times10^6$ |
| 15 | $1.07\times10^9$ | $5.37\times10^8$ |
| 21 | $4.40\times10^{12}$ | $2.20\times10^{12}$ |
| 25 | $1.13\times10^{15}$ | $5.63\times10^{14}$ |
| 31 | $4.61\times10^{18}$ | $2.31\times10^{18}$ |
Canonicalization approximately halves the number of possible representations because a k-mer and its reverse complement are represented by the same canonical form.
Genome sizes illustrate the range of sequence scales for which k-mers may be used. Bacterial genomes are typically a few megabases long, although some symbiotic bacteria have much smaller genomes, while other bacteria have genomes exceeding ten megabases. Plant genomes span an even wider range, from tens of megabases to more than one hundred gigabases. In very large and repeat-rich genomes, the main limitation is not necessarily the exhaustion of the theoretical k-mer space, but the non-uniform distribution of k-mers caused by sequence composition and repeated DNA.
The supported range $11\leq k\leq31$ should therefore be understood as an engineering and application-oriented compromise, not as a universal biological optimum. Shorter k-mers are more likely to be shared by unrelated regions, whereas longer k-mers provide greater sequence specificity. However, longer k-mers are also more sensitive to substitutions and indels: a single difference can disrupt an exact match across the whole k-mer. The appropriate value within the supported range consequently depends on the size and characteristics of the genomes being compared, as well as on the biological question.
The following section introduces a random-sequence model to quantify the probability of chance sharing across an indexed set of genomes. This provides a formal basis for assessing how the choice of $k$ affects the interpretation of shared k-mers.
The distinction between the simple $4^k$ model and the Karlin--Ost framework is important. The former describes the size of the combinatorial k-mer space; the latter describes the probability of observing long matches in sequences whose composition may depart from the uniform random model. Consequently, the value of $k$ cannot be selected from genome size alone.
## Chance sharing across genomes
OBIKmer indexes distinct canonical k-mers and records the genomes in
which each k-mer occurs. This makes it possible to compare the observed
sharing of k-mers with the sharing expected under a random-sequence
model. The same probability model can be used for two related purposes:
measuring the excess of sharing among k-mers present in the index, and
estimating the number of chance k-mers among those shared by at least a
specified number of genomes.
These scales have very different implications for different genome sizes.
### The null model and genome-specific probabilities
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995-lt; @Karlin1994-on]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
The null hypothesis assumes that the $S$ indexed genomes are unrelated
and that each genome is an independent random sequence in which
nucleotides are independent and equiprobable.
This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.
The supported range $11\leq k\leq31$ should therefore be understood as an engineering compromise rather than as a universal biological optimum. Smaller values increase the probability that unrelated genomic regions share the same k-mers, while larger values increase specificity but also make the representation more sensitive to substitutions, indels, and incomplete homology. The appropriate value within this range consequently depends on the biological scale and comparison task.
## Odd k-mer lengths
OBIKmer additionally requires $k$ to be odd. This constraint is independent of the resolution argument above.
For an odd-length DNA sequence, a k-mer cannot be identical to its own reverse complement. Consequently, every k-mer has two distinct orientations, which can be unambiguously collapsed into a canonical representation. For even $k$, self-reverse-complementary sequences exist, so a k-mer can coincide with its own reverse complement.
Odd $k$ is therefore **not required for canonicalization itself**: the canonical form
Since OBIKmer uses odd-length canonical k-mers, reverse-complement
canonicalization reduces the number of possible k-mers by half. The size
of the canonical k-mer space is therefore
$$
s^c=\min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)
M_k=\frac{4^k}{2}.
$$
remains well defined for even $k$. The odd-$k$ restriction instead guarantees that the two orientations are always distinct, which simplifies the representation and orientation handling used by OBIKmer.
## Karlin--Ost (1988) formula
[@Karlin1988] considers $s$ independent stationary letter sequences over a finite alphabet and studies $K_{r,s}(N)$, the length of the longest word common to $r$ of the $s$ sequences, each of length $N$. For i.i.d. equiprobable letters over an alphabet of size $m$ (here $m=4$ for DNA), the characteristic local match parameter reduces to
Let $N_i$ be the number of distinct canonical k-mers recorded for genome
$i$ in the index. Under the null model, these can be regarded as a
uniform sample without replacement from the $M_k$ possible canonical
k-mers. The probability that a given canonical k-mer is present in
genome $i$ is then
$$
\lambda^{[r]} = m^{1-r},
p_i=\frac{N_i}{M_k}=\frac{2N_i}{4^k}.
$$
so for $r=2$ (a word shared by two occurrences), $\lambda = m^{-1} = 1/4$.
This is the inclusion probability for a distinct k-mer. It differs from
the probability of at least one occurrence in a positional model,
because repeated occurrences within a genome are not counted separately.
The expected number of common words of length $k$ among the $\binom{s}{r}$ relevant groups of sequences is
Let $X$ denote the number of genomes, among the $S$ indexed genomes,
that contain a given canonical k-mer under the null model. Since genomes
are assumed to be independent, $X$ is a sum of independent presence
indicators with probabilities $p_1,\ldots,p_S$. When these probabilities
differ, $X$ follows a Poisson-binomial distribution.
### Computing the Poisson-binomial distribution
The distribution of $X$ can be computed by processing genomes one at a
time. Let $P_i(j)$ denote the probability that the k-mer occurs in
exactly $j$ of the first $i$ genomes. Initially,
$$
S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1)),
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
$$
with $\gamma=1$ exactly in the i.i.d. equiprobable case. Theorem 2.2 gives the limiting distribution of $K_{r,s}(N)$, with $k$ set on the scale $k=\left[\dfrac{\log N^{r}}{-\log\lambda^{*}}+x+1\right]$:
When genome $i$ is added, the k-mer is either absent, with probability
$1-p_i$, or present, with probability $p_i$. Thus,
$$
\lim_{N\to\infty}\left[\Pr\{K_{r,s}(N)<k\} - \exp\{-(1-\lambda^{*})\,S_1(N,k)\}\right] = 0,
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
$$
equivalently
This recurrence avoids enumerating the subsets of genomes. Computing the
complete distribution requires $O(S^2)$ operations and $O(S)$ memory. To
compute the upper-tail probability, only the probabilities for fewer
than $r$ presences need to be tracked:
$$
\frac{-\log\Pr\{K_{r,s}(N)<k\}}{(\lambda^{*})^{\rho(N,x)}} \xrightarrow[N\to\infty]{} (1-\lambda^{*})\,\gamma^{*}(\lambda^{*})^{x}.
P_{\mathrm{null}}(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
$$
The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985].
The computation can therefore be restricted to $P_i(0),\ldots,P_i(r-1)$,
requiring $O(Sr)$ operations and $O(r)$ memory. The resulting
distribution depends on the genome-specific values $N_i$, but not on the
identity of the k-mer, and can thus be computed once for a given index
and k-mer length.
<!--
$o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ».
### Comparing observed and expected sharing in the index
Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ :
- $N^{r}\binom{s}{r}\gamma\,\lambda^{k}$ est la valeur **exacte à la limite**, celle qu'on utilise en pratique (c'est elle qui redonne $E[C]$ de notre discussion).
- $(1+o(1))$ dit : « la vraie valeur de $S_1(N,k)$ n'est pas *exactement* égale à ce produit pour $N$ fini, mais le rapport entre les deux tend vers 1 quand $N\to\infty$ ». Autrement dit $S_1(N,k) / \left[N^{r}\binom{s}{r}\gamma\lambda^{k}\right] \to 1$.
Pourquoi ce terme existe : la formule $F_\sigma(k)/\lambda^k \to \gamma$ (éq. 2.8 du papier) est elle-même une limite quand $k\to\infty$ — pour $k$ fini, il y a un petit écart entre $F_\sigma(k)$ et $\gamma\lambda^k$. C'est cet écart-là qui devient le $o(1)$.
Ce que ça implique pour l'usage pratique : dans le cas iid équiprobable que tu utilises (ADN, $\gamma=1$ exactement, pas seulement à la limite — je l'ai vérifié directement sur la dérivation : $F_\sigma(k)=m^{-k}=\lambda^k$ pour tout $k$, pas juste asymptotiquement), le $o(1)$ est en réalité **nul** dans ce cas précis, pas juste négligeable. La formule $S_1(N,k)=N^r\binom{s}{r}\lambda^k$ est donc exacte (à l'asymptotique en $N$ près, qui elle vient d'ailleurs — de l'approximation de Poisson elle-même, pas de ce $\gamma$).
Les deux approches sont légitimes, mais elles répondent à des questions différentes — et il y a une relation algébrique exacte entre les deux qui vaut la peine d'être explicitée.
**Ce que calcule l'approche naïve.** Pour **un k-mer $w$ déjà observé et fixé**, $p_i\simeq1-\exp(-G_i/4^k)$ est la probabilité que *ce mot précis* apparaisse quelque part dans le génome $i$ par hasard, et $P(R\geq r)$ (Poisson-binomiale ou binomiale) est la probabilité que *ce mot précis* soit partagé par au moins $r$ des $S$ génomes par hasard. C'est une **p-value par observation** : « j'ai vu ce k-mer particulier dans 5 génomes sur 10, est-ce surprenant pour lui ? »
**Ce que calcule Karlin-Ost.** $S_1(k)$ et $P(K_{r,s}\geq k)$ répondent à : « parmi **tous les $4^k$ mots possibles**, quelle est la probabilité qu'au moins un seul d'entre eux atteigne ce seuil de partage par hasard, n'importe où dans les génomes ? » C'est une question **globale sur tout l'espace des k-mers**, pas sur un mot déjà identifié.
**La relation exacte entre les deux.** Reprenons le cas $r=s=2$, génomes égaux de taille $G$ :
- Naïf (mot $w$ fixé) : $P(R=2\mid w)\simeq p^2=(G\cdot4^{-k})^2=G^2\cdot4^{-2k}$
- Karlin-Ost : $S_1=G^2\cdot4^{-k}$
Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nombre de mots possibles. Ce n'est pas une coïncidence : $S_1=N^r\binom{s}{r}\lambda^k$ vient de $\lambda^k=\sum_w \Pr\{w\}^r$, une somme explicite sur les $4^k$ mots, alors que le calcul naïf tel que présenté ne somme sur rien — il traite $w$ comme donné.
**Donc l'approche naïve, formellement, c'est Karlin-Ost auquel il manque le facteur « nombre de k-mers testés ».** Pour la rendre comparable à la question « dois-je m'attendre à *une* collision fortuite n'importe où dans mon jeu de données », il faudrait une correction à la Bonferroni : multiplier $P(R\geq r\mid w)$ par le nombre effectif de k-mers canoniques testables ($\approx4^k/2$). Karlin-Ost fait cette correction analytiquement et correctement (avec en prime le facteur de déclumping $(1-\lambda)$ qu'on a discuté, qui corrige le sur-comptage des matches chevauchants — une union bound brute comme le Bonferroni décrit ci-dessus le surestimerait légèrement).
**Ce qui veut dire, en pratique, que les deux outils ne sont pas concurrents mais complémentaires :**
- **Choix a priori de $k$ / FPR global du dataset** (la question de `kmers.md` depuis le début de cette conversation) → Karlin-Ost, parce qu'il intègre nativement la multiplicité sur tous les mots possibles.
- **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant.
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
-->
## Chance sharing across genomes and the FDR of homology
The formulas above ask how likely a match is between two sequences. The question OBIKmer actually has to answer is different: a k-mer is found in $r$ of the $S$ indexed genomes, and one wants to know whether this sharing can be explained by chance or whether it is evidence of homology. This section builds a null model for that question and turns it into a false discovery rate (FDR) for the call "a k-mer shared by at least $r$ genomes is homologous".
### The null model
The null hypothesis is that the $S$ genomes are unrelated, that is, that each one is an independent random sequence. Rejecting it for a k-mer means that chance alone cannot explain why the k-mer is shared, which is the signal of homology one is looking for.
The model only records whether a k-mer is present in a genome, not how many times. Two consequences follow. First, repeats inside a genome are irrelevant, and the size $G_i$ of genome $i$ is its number of **distinct canonical k-mers**. Second, a repeat family shared between genomes is paralogy, which is a form of homology; it is a true positive and not a false one. The only k-mers that could be shared by composition rather than by descent are low-complexity ones, which OBIKmer removes beforehand with its entropy filter.
Everything takes place in canonical space. Because $k$ is odd, no k-mer equals its own reverse complement, so the $4^k$ k-mers pair up perfectly and the space contains exactly
The index contains only k-mers observed in at least one genome.
Consequently, when comparing sharing proportions among the k-mers
recorded in the index, the relevant null probability is conditional on
at least one presence:
$$
N=\frac{4^k}{2}
q_r=P(X\geq r\mid X\geq 1)
=\frac{P_{\mathrm{null}}(X\geq r)}{P_{\mathrm{null}}(X\geq 1)}.
$$
canonical k-mers. A given canonical k-mer is present in genome $i$ by chance with probability
Let $n_{\geq r}$ be the number of distinct canonical k-mers observed in
at least $r$ genomes, and let $n_{\geq 1}$ be the total number of
distinct canonical k-mers recorded in the index. The observed proportion
of indexed k-mers shared by at least $r$ genomes is
$$
p_i = 1-\left(1-\frac1N\right)^{G_i}\simeq \frac{G_i}{N}=\frac{2\,G_i}{4^k}.
O_r=\frac{n_{\geq r}}{n_{\geq 1}}.
$$
This is twice the $G/4^k$ used earlier in this page, because there $G$ is a number of positions on one strand, whereas $G_i$ here counts canonical k-mers, for which both strands are already merged.
Let $R$ be the number of genomes, out of $S$, that contain a given canonical k-mer by chance. Genomes are independent under the null, so $R$ is a sum of independent presence indicators, each with its own probability $p_i$.
### Exact tail probability
When genome sizes differ, the $p_i$ differ, and $R$ follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be hopeless, since there are up to $2^S$ subsets. The distribution is instead obtained by adding the genomes one at a time. Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
Under the random-sequence model, the expected proportion is $q_r$. The
difference
$$
d'_j = d_j\,(1-p_i) + d_{j-1}\,p_i, \qquad d_0=1 \text{ before the first genome.}
D_r=O_r-q_r
$$
The table $d$ only needs to remember how many genomes were hit, not which ones, and this brings the cost down to $O(S^2)$. The result does not depend on the order in which the genomes are processed. It does not depend on the k-mer either, so it is computed once per index. The upper tail is then
therefore measures the excess or deficit of sharing relative to the null
expectation. Positive values indicate that k-mers are shared across
genomes more frequently than expected by chance, whereas negative values
indicate less frequent sharing. A value of zero indicates agreement with
the random-sequence model.
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$
is bounded:
$$
P_0(R\geq r) = 1-\sum_{j=0}^{r-1} d_j .
-1\leq D_r\leq 1.
$$
Only $d_0,\dots,d_{r-1}$ are needed, so the cost is $O(S\,r)$.
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By
construction, $D_1=0$, since all k-mers recorded in the index occur in
at least one genome. Unlike a ratio-based measure, this difference
remains defined when no k-mers are observed in at least $r$ genomes.
### Approximation with equal genome sizes
### Estimating the false discovery rate
If all genomes are given the same size, taken as the mean
The probability $P_{\mathrm{null}}(X\geq r)$ applies to one specified canonical k-mer.
To estimate the number of k-mers that could reach this sharing threshold
by chance across the entire canonical k-mer space, this probability must
be considered over all $M_k$ possible canonical k-mers.
The expected number of chance k-mers present in at least $r$ genomes is
$$
\bar G=\frac1S\sum_{i=1}^{S}G_i, \qquad \bar p=\frac{2\bar G}{4^k},
E[\mathrm{FP}(r)]=M_kP_{\mathrm{null}}(X\geq r).
$$
then $R$ follows a plain binomial distribution and the tail is
The observed number of k-mers passing the threshold is $n_{\geq r}$. An
estimate of the false discovery rate (FDR) for this set is therefore
$$
P_0(R\geq r)\simeq\sum_{j=r}^{S}\binom{S}{j}\,\bar p^{\,j}(1-\bar p)^{S-j}.
\widehat{\mathrm{FDR}}(r)=
\begin{cases}
1 & \text{if } n_{\geq r}=0,\\[4pt]
\displaystyle
\min\left(1,
\frac{M_k P_{\mathrm{null}}(X\geq r)}
{n_{\geq r}}\right)
& \text{otherwise}.
\end{cases}
$$
This sum is the regularized incomplete beta function $I_{\bar p}(r,\,S-r+1)$, available in standard statistical libraries. If it is summed by hand, the terms are obtained one from the next by $t_{j+1}=t_j\,\frac{S-j}{j+1}\,\frac{\bar p}{1-\bar p}$.
The numerator estimates the number of false positives under the null
model, while the denominator is the observed number of k-mers passing
the sharing threshold. This estimate assumes that all possible canonical
k-mers are subject to the null model. It is therefore conservative in
the sense that it does not attempt to estimate the proportion of the
k-mer space corresponding to true homology. The estimate is capped at
one because a proportion cannot exceed one.
A theorem of Hoeffding [@Hoeffding1956-bx] shows that this approximation errs on the safe side. For a fixed mean, the Poisson-binomial distribution is more concentrated than the binomial one, so for $r\geq S\bar p+1$ the binomial tail is never smaller than the exact tail. Using the mean genome size therefore overestimates the chance of sharing and can only make the FDR more pessimistic. Gleser [@Gleser1975-yu] refines this result: replacing the $p_i$ by any partial average of themselves, not only by their full mean, also moves the upper tail towards the binomial one, provided $r-1\geq S\bar p+2$. Grouping genomes of similar size is therefore safe in the same sense. What it loses is power when genome sizes differ by orders of magnitude, since one large genome then weighs as much as several small ones in $\bar p$.
The distinction between $D_r$ and the FDR is important. $D_r$ compares
the observed and expected proportions among k-mers already present in
the index. The FDR estimate instead compares the expected number of
chance k-mers in the entire canonical k-mer space with the observed
number passing the sharing threshold.
When all $p_i$ are small, which is the case for large $k$, the number of hits is also well approximated by a Poisson distribution of mean $\lambda=\sum_i p_i$, with an error bounded by $\sum_i p_i^2$ [@Le-Cam1960-pe].
The smallest value of $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls
below a chosen threshold, for example 5%, can be used as a minimum
sharing threshold. This interpretation is conditional on the assumptions
of the null model and should not be taken as a model-independent
guarantee of the biological false discovery rate.
For the numerical evaluation, the tail that does not contain the mode $S\bar p$ should be summed directly. When $r>S\bar p$, computing the sum from $j=r$ upwards is accurate even for very small probabilities, whereas computing $1$ minus the sum of the lower terms loses all precision once the tail falls below the machine epsilon.
As an illustration, consider $k=21$, for which
$M_k\simeq2.2\times10^{12}$, and an index containing $S=10$ genomes.
Suppose that the numbers of distinct canonical k-mers in these genomes
range from $0.5\times10^9$ to $1.4\times10^9$, in increments of
$0.1\times10^9$. Applying the Poisson-binomial model with the
corresponding genome-specific probabilities gives the following
expected numbers of k-mers shared by chance:
### False discovery rate
The FDR of the call "present in at least $r$ genomes" is the fraction of retained k-mers expected to be pure chance:
$$
\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P_0(R\geq r)}{n_{\geq r}}\right).
$$
The numerator is the number of k-mers expected to reach $r$ genomes by chance if the genomes were unrelated. Each of the $N$ canonical k-mers has probability $P_0(R\geq r)$ of doing so, and no further correction for multiple testing is needed because the number of tests is already in the product. The denominator $n_{\geq r}$ is the number of distinct k-mers actually observed in at least $r$ genomes, which is read directly from the index. The observed count mixes chance and true homology, so the ratio is an upper bound on the true FDR: it implicitly assumes that all k-mers are null, and it never underestimates the noise. The minimal $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls below the accepted level, for example 5 %, is the sharing threshold to use.
As an illustration, take $k=21$ (so $N=2.2\times10^{12}$), $S=10$ genomes and $G_i=10^9$ distinct canonical k-mers each, hence $p=4.5\times10^{-4}$. The number of k-mers expected to be shared by chance is:
| $r$ | $N\,P_0(R\geq r)$ |
| $r$ | $M_kP_{\mathrm{null}}(X\geq r)$ |
| ---: | ---: |
| 2 | $2\times10^{7}$ |
| 3 | $2.5\times10^{4}$ |
| 4 | $\approx 20$ |
| 5 | $\approx 0.01$ |
| 2 | $1.82\times10^7$ |
| 3 | $2.06\times10^4$ |
| 4 | $15$ |
| 5 | $0.0075$ |
Chance alone produces millions of pairwise shared 21-mers at this genome size, a few tens of thousands shared by three genomes, and almost none shared by five or more. Whether a given $r$ is acceptable then depends on how many k-mers the index actually reports at that level.
In this example, chance alone can produce many pairwise shared 21-mers,
whereas the expected number of k-mers shared by five or more genomes is
very small. Whether a given sharing threshold is sufficiently stringent
depends on the number of k-mers actually observed at that level.
### Limits of the model
- The FDR is computed per k-mer. Neighbouring k-mers of a shared region overlap and are not independent, so a single homologous region contributes many correlated k-mers. A per-region FDR would count regions instead of k-mers and is not covered here.
- The probability $p_i$ assumes equiprobable, independent nucleotides. The composition biases discussed earlier in this page violate this assumption, and the resulting error is not accounted for.
The FDR estimate is based on a simplified null model and has two
important limitations.
---
First, the model assumes independent, equiprobable nucleotides. Real
genomes exhibit nucleotide-composition biases, oligonucleotide
dependencies, and repeated sequences. These properties can increase the
probability of particular k-mers beyond the uniform expectation.
Removing low-complexity k-mers reduces some of these effects but does
not eliminate all compositional biases. Karlin and colleagues showed
that relative oligonucleotide abundances, including dinucleotide
frequencies, form characteristic genomic signatures that can differ
between organisms and remain relatively stable within genomes
[@Karlin1995-lt; @Karlin1994-on].
<!-- Notes perso (bac à sable), non publiées dans le HTML
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
## 1. FDR d'un ensemble de k-mers sélectionnés
**Estimation de l'espérance du nombre de faux positifs.** Soit $M$ le nombre de k-mers canoniques testés, $M = 4^k/2$ pour k impair. Soit $q(r)=P(X\ge r)$ la queue du modèle nul. Le nombre attendu de k-mers présents par hasard dans au moins $r$ génomes vaut
$$
E[\mathrm{FP}(r)] = M\,q(r).
$$
Cette égalité est exacte par linéarité de l'espérance, même si les k-mers chevauchants sont dépendants. La dépendance n'affecte que la variance, pas l'espérance. Seule l'approximation de Poisson par génome reste une approximation.
**FDR.** Soit $N(r)=\#\{i : r_i\ge r\}$ le nombre observé de k-mers couvrant au moins $r$ génomes. Alors
$$
\mathrm{FDR}(r)=\min\!\Big(1,\ \frac{M\,q(r)}{N(r)}\Big).
$$
Cette estimation est légèrement conservatrice. Elle suppose $\pi_0=1$, c'est-à-dire que presque tout l'espace $M$ est nul, ce qui est vrai pour k grand.
**Ensemble hétérogène en $r_i$.** Si le jeu n'est pas défini par un seuil unique, on utilise le local fdr par classe de couverture :
$$
\mathrm{fdr}(r)=\frac{M\,P(X=r)}{N_r},\qquad N_r=\#\{i : r_i=r\}.
$$
Le FDR de l'ensemble est alors la moyenne $\frac1N\sum_i \mathrm{fdr}(r_i)$. Pour un ensemble défini par seuil $r_{\min}$, cela redonne exactement la formule précédente.
**Remarque.** $M\,p\approx G$, d'où $E[\mathrm{FP}(r)]\approx\binom{S}{r}\,G^r/M^{r-1}$ en régime de petits $p$. Cela donne une lecture directe de la quantité.
## 2. Markov d'ordre 0 avec fréquences inégales
Soit $\pi_b$ la fréquence du nucléotide $b$. Pour un k-mer $w$ de composition $n=(n_A,n_C,n_G,n_T)$ avec $\sum n_b=k$, on a $P(w)=\prod_b \pi_b^{n_b}$.
**Attendu par génome.** Le k-mer canonique $K=\{w,\overline w\}$ est vu quand le brin direct porte $w$ ou $\overline w$, donc
$$
\lambda_K = G\,\big(P(w)+P(\overline w)\big)=G\Big(\prod_b\pi_b^{n_b}+\pi_A^{n_T}\pi_T^{n_A}\pi_C^{n_G}\pi_G^{n_C}\Big).
$$
Cela ne dépend que de la composition. Si $\pi_A=\pi_T$ et $\pi_C=\pi_G$, on retrouve $2P(w)$.
**Faux positifs.** Le test n'est plus homogène : chaque composition a sa propre $p(n)=1-e^{-\lambda(n)}$. Il faut donc sommer sur les classes de composition (un k-mer et son complément inverse partagent le même $\lambda$, d'où le facteur ½) :
$$
E[\mathrm{FP}(r)]=\frac12\sum_{n}\binom{k}{n_A,n_C,n_G,n_T}\,P\big(X\ge r\mid p(n)\big).
$$
Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Elle est donc légère avec la binomiale de la moyenne, ou avec une Poisson-binomiale par composition si les $G_i$ diffèrent. Si les $\pi_b$ varient d'un génome à l'autre (GC différent), les $p_i(n)$ se construisent par génome.
**Effet attendu.** La moyenne de $\lambda_w$ sur tous les k-mers reste $2G/4^k$, car $\sum_w P(w)=1$. Dans le régime de petits $p$, le terme dominant $\binom{S}{r}p^r$ est convexe en $\lambda$. Par Jensen, le modèle équiprobable est donc le minimum de $E[\mathrm{FP}]$, et la déviation de composition ne peut qu'augmenter le FDR. Les k-mers riches en AT ou GC dominent la somme, ce qui rejoint ton remarque sur la faible complexité. C'est un résultat sur le terme dominant, pas une preuve pour la queue exacte.
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
-->
Second, the FDR is computed per k-mer. Consecutive k-mers extracted from
the same homologous region overlap and are therefore not independent
observations. A single homologous region can contribute many correlated
k-mers to the index. The estimate presented here consequently describes
the expected fraction of chance k-mers, not the fraction of false
homologous regions. A region-level FDR would require a model that treats
matching regions, rather than individual k-mers, as the units of
observation.
+33 -1
View File
@@ -1,6 +1,6 @@
# estimate
Estimate approximate-index parameters (z, evidence bits, false-positive rate) before building an index with `--approx`, without touching any files.
`estimate` has two independent uses. Without `--index`, it estimates approximate-index parameters (z, evidence bits, false-positive rate) before building an index with `--approx`, without touching any files. With `--index`, it compares the k-mer sharing observed in an existing index with the sharing expected by chance (see the last section).
```bash
obikmer estimate [OPTIONS]
@@ -16,3 +16,35 @@ obikmer estimate [OPTIONS]
| `--fp` | none | Target false-positive rate per z-window |
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `convert --approx-evidence` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
## Comparing an index with random sharing
```bash
obikmer estimate --index INDEX [--presence-threshold N] [-o FILE]
```
With `--index`, `estimate` reads an existing index instead of computing approximate-index parameters, and it cannot be combined with `-k`, `-z`, `--evidence-bits` or `--fp`. Its purpose is to tell how much of the k-mer sharing between the genomes of the index goes beyond what chance alone would produce, which is the question behind the choice of $k$ (see [K-mer size](../theory/kmer_indexing/kmers.md)).
The command writes a CSV, to standard output or to `FILE`, with one row for each $r$ from 1 to the number of genomes $S$:
| Column | Meaning |
|---|---|
| `r` | Minimum number of genomes sharing a k-mer |
| `k_r` | Number of k-mers present in at least `r` genomes |
| `O_r` | Observed proportion $k_r / k_1$ |
| `q_r` | Proportion expected by chance, $P(X \geq r \mid X \geq 1)$, where $X$ is the number of genomes containing a given k-mer under the random-sequence model |
| `D_r` | Deviation $O_r - q_r$ |
| `FP_r` | Number of k-mers expected to be present in at least `r` genomes by chance over the whole canonical k-mer space, $M_k \, P(X \geq r)$ |
| `FDR_r` | Estimated false discovery rate of the k-mers shared by at least `r` genomes: 1 when `k_r` is 0, otherwise $\min(1, \mathrm{FP}_r / k_r)$ |
A positive `D_r` means that k-mers are shared by at least `r` genomes more often than random sequences would share them; a negative value means less often. The first row always has `D_1 = 0`, because $k_1$ counts only k-mers present in at least one genome.
When no k-mer is shared by at least `r` genomes (`k_r = 0`), there is no set of k-mers whose false discovery rate could be estimated, and `FDR_r` is set to 1 by convention. This value means that nothing can be claimed at this level; it does not mean that the observed k-mers are false positives. The `FP_r` column of the same row shows how many k-mers chance alone would have produced.
The expected proportion is computed under the random-sequence model of the theory page: the genomes are unrelated, and each genome is a set of $N_i$ distinct canonical k-mers drawn without replacement and uniformly from the $M_k = 4^k/2$ canonical k-mers. A given k-mer is then present in genome $i$ with probability $p_i = N_i / M_k = 2N_i/4^k$, where $N_i$ is the number of distinct canonical k-mers considered as present in genome $i$. Genome sizes are rarely known, in particular when the index was built from unassembled reads, and the number of distinct k-mers has the advantage of being measured on the index itself and of not depending on the sequencing coverage.
Because the $N_i$ differ between genomes, the number $X$ of genomes containing a k-mer follows an exact Poisson-binomial distribution, from which $P(X \geq r)$ is obtained for every $r$ in a single pass. The approximation by a binomial distribution based on the mean genome size is not used.
`--presence-threshold` (default 1) has the same meaning as in `phylo`: on an index that stores counts, a k-mer is present in a genome only when its count reaches the threshold. Both $N_i$ and $k_r$ are computed with this same threshold, so the observed and expected proportions describe the same data. It has no effect on an index without counts.
The `k` used is the k-mer size stored in the index (for an approximate index, the effective indexed size $k-z+1$).
+1 -1
View File
@@ -1531,7 +1531,7 @@ dependencies = [
[[package]]
name = "obikmer"
version = "1.4.5"
version = "1.4.6"
dependencies = [
"clap",
"csv",
+1
View File
@@ -14,6 +14,7 @@ mod meta;
mod mmap_file;
mod rankselect;
mod reader;
mod row_histogram;
mod sparse_intmatrix;
mod sparse_intmatrix_columnar;
mod storage_kind;
+77
View File
@@ -0,0 +1,77 @@
//! Row-wise presence histogram — how many k-mers (matrix rows) are present
//! in exactly `j` genomes (columns), for every `j`.
//!
//! Row-major for `Sparse` (no on-disk column representation: a `col_view`
//! there is a deliberate `panic!`, so a column could only be synthesised by
//! scanning every row anyway), column-major otherwise — same split, same
//! reasons as [`crate::batch_presence_counts`] and
//! [`crate::batch_int_group_stats`], which this reuses and mirrors.
//!
//! The result is additive across layers and partitions: summing the vectors
//! gives the histogram of the whole index.
use std::io;
use crate::bitmatrix::batch_presence_counts;
use crate::colgroup::{ColGroup, chunked_presence_count};
use crate::intmatrix::PersistentIntMatrix;
use crate::tempintvec::TempCompactIntVec;
use crate::PersistentBitMatrix;
/// Tallies a per-row presence-count vector into `hist[j]` = number of rows
/// whose count is `j`. `hist` has `n_cols + 1` entries.
fn tally(counts: &TempCompactIntVec, n_cols: usize) -> Vec<u64> {
let mut hist = vec![0u64; n_cols + 1];
for c in counts.view().iter() {
hist[c as usize] += 1;
}
hist
}
impl PersistentIntMatrix {
/// `hist[j]` = number of rows (k-mers) present in exactly `j` columns
/// (genomes), a cell being present when its count reaches `threshold`.
/// Length `n_cols() + 1`; `hist[0]` counts rows present nowhere, which
/// is possible when `threshold > 1`. `threshold == 0` makes every cell
/// present: all rows land in `hist[n_cols()]`.
pub fn row_presence_histogram(&self, threshold: u32) -> io::Result<Vec<u64>> {
let n_cols = self.n_cols();
if threshold == 0 {
let mut hist = vec![0u64; n_cols + 1];
hist[n_cols] = self.n() as u64;
return Ok(hist);
}
match self {
Self::Sparse(m) => {
let mut hist = vec![0u64; n_cols + 1];
for slot in 0..m.n() {
let mut present = 0usize;
m.for_each_cell_in_row(slot, |_, v| present += usize::from(v >= threshold));
hist[present] += 1;
}
Ok(hist)
}
Self::Columnar(_) | Self::Packed(_) => {
let cols: Vec<usize> = (0..n_cols).collect();
let counts = chunked_presence_count(self.n(), &cols, |b, c| {
b.inc_predicate_fast(self.col_view(c), |v| v >= threshold)
})?;
Ok(tally(&counts, n_cols))
}
}
}
}
impl PersistentBitMatrix {
/// `hist[j]` = number of rows (k-mers) present in exactly `j` columns
/// (genomes). Length `n_cols() + 1`. A presence matrix holds no counts,
/// so there is no threshold: every set bit is present.
pub fn row_presence_histogram(&self) -> io::Result<Vec<u64>> {
let n_cols = self.n_cols();
let all = ColGroup::new("all", (0..n_cols).collect());
let counts = batch_presence_counts(self, std::slice::from_ref(&all))?
.pop()
.expect("one group in, one count vector out");
Ok(tally(&counts, n_cols))
}
}
+28
View File
@@ -304,3 +304,31 @@ fn col_view_packed_matches_columnar() {
}
drop(dir_col);
}
// ── count_geq (ColumnWeights) ─────────────────────────────────────────────────
/// A presence matrix holds no counts: `count_geq` is `count_ones` for any
/// threshold, so a presence index is unaffected by `--presence-threshold`.
#[test]
fn count_geq_ignores_threshold() {
use crate::traits::ColumnWeights;
let (_d, m) = make_matrix(&[
&[true, false, true, true],
&[false, false, false, false],
&[true, true, true, false],
]);
for t in [0u32, 1, 2, 100] {
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), vec![3, 0, 3], "threshold {t}");
}
}
#[test]
fn row_presence_histogram_dense() {
let (_d, m) = make_matrix(&[
&[true, false, true, true],
&[false, false, true, false],
&[true, false, true, false],
]);
// Rows carry 2, 0, 3, 1 genomes.
assert_eq!(m.row_presence_histogram().unwrap(), vec![1, 1, 1, 1]);
}
+131 -1
View File
@@ -1,6 +1,6 @@
use tempfile::tempdir;
use crate::traits::CountPartials;
use crate::traits::{ColumnWeights, CountPartials};
use crate::{
PersistentCompactIntMatrixBuilder, PersistentCompactIntVec, PersistentCompactIntVecBuilder,
PersistentIntMatrix, StorageKind, pack_compact_int_matrix, pack_sparse_compact_int_matrix,
@@ -682,3 +682,133 @@ fn intslice_view_collect_slots_values() {
assert_eq!(view.collect_slots_values(&[0, 1, 2]), vec![10, 255, 300]);
assert_eq!(view.collect_slots_values(&[3, 1]), vec![1000, 255]);
}
// ── count_geq (ColumnWeights) ─────────────────────────────────────────────────
/// Naive per-column `count(v >= threshold)` over column-major `data`.
fn naive_count_geq(data: &[&[u32]], threshold: u32) -> Vec<u64> {
data.iter().map(|col| col.iter().filter(|&&v| v >= threshold).count() as u64).collect()
}
/// Thresholds straddling 0, 1, the primary-byte limit (255) and the overflow values.
const GEQ_THRESHOLDS: [u32; 10] = [0, 1, 2, 5, 10, 254, 255, 256, 500, 1001];
#[test]
fn count_geq_columnar_matches_naive() {
let data: &[&[u32]] = &[&[0, 1, 2, 300, 255], &[5, 0, 0, 0, 1000], &[0, 0, 0, 0, 0]];
let (_d, m) = make_matrix(data);
assert_eq!(m.storage_kind(), StorageKind::Columnar);
for t in GEQ_THRESHOLDS {
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), naive_count_geq(data, t), "threshold {t}");
}
}
#[test]
fn count_geq_packed_matches_naive() {
let data: &[&[u32]] = &[&[10, 300, 500, 0], &[200, 50, 1000, 1]];
let (dir, _col) = make_matrix(data);
pack_compact_int_matrix(&dir.path().join("counts")).unwrap();
let m = PersistentIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.storage_kind(), StorageKind::Packed);
for t in GEQ_THRESHOLDS {
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), naive_count_geq(data, t), "threshold {t}");
}
}
#[test]
fn count_geq_sparse_enum_matches_naive() {
let data: &[&[u32]] = &[&[0, 5, 0, 3, 700], &[2, 0, 0, 4, 0], &[0, 0, 9, 0, 1]];
let (dir, _col) = make_matrix(data);
pack_sparse_compact_int_matrix(&dir.path().join("counts")).unwrap();
let m = PersistentIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.storage_kind(), StorageKind::Sparse);
for t in GEQ_THRESHOLDS {
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), naive_count_geq(data, t), "threshold {t}");
}
}
#[test]
fn partial_kmer_counts_is_count_geq_one() {
let data: &[&[u32]] = &[&[0, 1, 2, 300], &[5, 0, 0, 0]];
let (_d, m) = make_matrix(data);
assert_eq!(m.partial_kmer_counts().to_vec(), m.count_nonzero().to_vec());
assert_eq!(m.partial_kmer_counts().to_vec(), ColumnWeights::count_geq(&m, 1).to_vec());
}
#[test]
fn pciv_count_geq_matches_naive() {
let counts = [0u32, 1, 2, 254, 255, 256, 1000];
let (_dir, v) = make_pciv(&counts);
for t in GEQ_THRESHOLDS {
let expected = counts.iter().filter(|&&c| c >= t).count() as u64;
assert_eq!(v.count_geq(t), expected, "threshold {t}");
}
}
// ── row_presence_histogram ────────────────────────────────────────────────────
/// Naive reference: for each row, how many columns reach `threshold`.
fn naive_row_hist(data: &[&[u32]], threshold: u32) -> Vec<u64> {
let n = data[0].len();
let mut hist = vec![0u64; data.len() + 1];
for slot in 0..n {
hist[data.iter().filter(|col| col[slot] >= threshold).count()] += 1;
}
hist
}
const HIST_DATA: &[&[u32]] = &[
&[0, 1, 2, 300, 255, 0],
&[5, 0, 0, 0, 1000, 0],
&[0, 7, 0, 3, 1, 0],
];
#[test]
fn row_presence_histogram_columnar_matches_naive() {
let (_d, m) = make_matrix(HIST_DATA);
for t in GEQ_THRESHOLDS {
assert_eq!(m.row_presence_histogram(t).unwrap(), naive_row_hist(HIST_DATA, t), "threshold {t}");
}
}
#[test]
fn row_presence_histogram_packed_matches_naive() {
let (dir, _m) = make_matrix(HIST_DATA);
pack_compact_int_matrix(&dir.path().join("counts")).unwrap();
let m = PersistentIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.storage_kind(), StorageKind::Packed);
for t in GEQ_THRESHOLDS {
assert_eq!(m.row_presence_histogram(t).unwrap(), naive_row_hist(HIST_DATA, t), "threshold {t}");
}
}
#[test]
fn row_presence_histogram_sparse_matches_naive() {
let (dir, _m) = make_matrix(HIST_DATA);
pack_sparse_compact_int_matrix(&dir.path().join("counts")).unwrap();
let m = PersistentIntMatrix::open(dir.path()).unwrap();
assert_eq!(m.storage_kind(), StorageKind::Sparse);
for t in GEQ_THRESHOLDS {
assert_eq!(m.row_presence_histogram(t).unwrap(), naive_row_hist(HIST_DATA, t), "threshold {t}");
}
}
/// More than 254 columns exercises `chunked_presence_count`'s chunked path
/// (per-row counts above the `u8` primary range).
#[test]
fn row_presence_histogram_many_columns() {
// 300 columns × 4 rows: row r is present (value 1) in the first
// [0, 300, 7, 255] columns respectively.
let present = [0usize, 300, 7, 255];
let cols: Vec<Vec<u32>> = (0..300)
.map(|c| present.iter().map(|&p| u32::from(c < p)).collect())
.collect();
let col_refs: Vec<&[u32]> = cols.iter().map(Vec::as_slice).collect();
let (_d, m) = make_matrix(&col_refs);
let hist = m.row_presence_histogram(1).unwrap();
assert_eq!(hist.len(), 301);
for p in present {
assert_eq!(hist[p], 1, "rows present in {p} columns");
}
assert_eq!(hist.iter().sum::<u64>(), 4);
}
+27
View File
@@ -443,3 +443,30 @@ fn build_permuted_follows_the_permutation() {
assert_eq!(row_as_bool(&permuted, new), row_as_bool(&src, old as usize), "new row {new}");
}
}
#[test]
fn sparse_count_geq_ignores_threshold() {
use crate::traits::ColumnWeights;
let rows: &[&[bool]] = &[&[true, false, true], &[false, false, true], &[true, false, false]];
let (_d, m) = make_sparse(rows, 3);
for t in [0u32, 1, 2, 100] {
assert_eq!(ColumnWeights::count_geq(&m, t).to_vec(), vec![2, 0, 2], "threshold {t}");
}
}
#[test]
fn sparse_row_presence_histogram_matches_dense() {
let cols: &[&[bool]] = &[
&[true, false, true, true, false],
&[false, false, true, false, false],
&[true, false, true, false, false],
];
let (_dd, dense) = make_dense(cols);
let rows: Vec<Vec<bool>> = (0..5).map(|s| cols.iter().map(|c| c[s]).collect()).collect();
let row_refs: Vec<&[bool]> = rows.iter().map(Vec::as_slice).collect();
let (_ds, sparse) = make_sparse(&row_refs, 3);
let sparse = PersistentBitMatrix::Sparse(sparse);
assert_eq!(sparse.row_presence_histogram().unwrap(), dense.row_presence_histogram().unwrap());
// Rows carry 2, 0, 3, 1, 0 genomes.
assert_eq!(sparse.row_presence_histogram().unwrap(), vec![2, 1, 1, 1]);
}
+12
View File
@@ -266,6 +266,18 @@ impl KmerLayer {
}
}
/// `hist[j]` = number of k-mers of this layer present in exactly `j`
/// genomes (length `n_cols() + 1`), a k-mer being present when its count
/// reaches `threshold` — presence layers hold no counts, so `threshold`
/// has no effect there. Additive across layers.
pub fn row_presence_histogram(&self, threshold: u32) -> OKIResult<Vec<u64>> {
match self {
KmerLayer::Count { layer, .. } => layer.row_presence_histogram(threshold),
KmerLayer::Presence { layer, .. } => layer.row_presence_histogram(),
KmerLayer::Empty { .. } => panic!("Layer::row_presence_histogram() called on an Empty layer"),
}
}
/// Batch, genome-major "carries" for a set of `slots` — `out[g][i]` =
/// whether genome `g` (0..`out.len()`) carries `slots[i]`. `out` must
/// have one entry per genome column, each resized to `slots.len()`.
+13
View File
@@ -395,6 +395,13 @@ impl TypedLayer<PersistentIntMatrix> {
self.data.n_cols()
}
/// `hist[j]` = number of k-mers of this layer present in exactly `j`
/// genomes, a k-mer being present in a genome when its count reaches
/// `threshold` (see `PersistentIntMatrix::row_presence_histogram`).
pub fn row_presence_histogram(&self, threshold: u32) -> OKIResult<Vec<u64>> {
self.data.row_presence_histogram(threshold).map_err(OKIError::Io)
}
/// Extract a sub-matrix of counts for the rows at `slots`.
///
/// Returns a column-first `Vec<Vec<u32>>`: one inner `Vec` per genome
@@ -518,6 +525,12 @@ impl<D: LayerData<Item = Box<[bool]>> + BinaryMatrix + obicompactvec::ColumnWeig
}
impl TypedLayer<PersistentBitMatrix> {
/// `hist[j]` = number of k-mers of this layer present in exactly `j`
/// genomes (see `PersistentBitMatrix::row_presence_histogram`).
pub fn row_presence_histogram(&self) -> OKIResult<Vec<u64>> {
self.data.row_presence_histogram().map_err(OKIError::Io)
}
/// Every nonzero `(idx into slots, col, value)` triple among `slots` —
/// delegates to `PersistentBitMatrix::nonzero_iter`.
pub fn nonzero_iter<'a>(
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "obikmer"
version = "1.4.5"
version = "1.4.6"
edition = "2024"
[[bin]]
+33
View File
@@ -1,9 +1,38 @@
use std::path::PathBuf;
use clap::Args;
use super::index::resolve_approx_params;
mod sharing;
#[derive(Args)]
pub struct EstimateArgs {
/// Compare the k-mer sharing of an index with a random-sequence model
///
/// Instead of the approximate-index report, reads INDEX and writes a CSV
/// with one row per r = 1..S (S = number of genomes): `r`; `k_r`, the
/// number of k-mers present in at least r genomes; `O_r = k_r/k_1`, the
/// observed proportion; `q_r`, the proportion expected by chance when
/// each genome is a random set of k-mers of its own size; and
/// `D_r = O_r - q_r`, positive when k-mers are shared more than chance
/// predicts. Cannot be combined with -k, -z, --evidence-bits or --fp.
#[arg(long, value_name = "INDEX",
conflicts_with_all = ["kmer_size", "findere_z", "evidence_bits", "fp"])]
pub index: Option<PathBuf>,
/// Minimum k-mer count to count as present (with --index)
///
/// Same meaning as in `phylo`: on an index that stores counts, a k-mer
/// is present in a genome only when its count reaches this threshold.
/// It has no effect on indexes without counts.
#[arg(long, default_value = "1", requires = "index")]
pub presence_threshold: u32,
/// Write the CSV to this file instead of standard output (with --index)
#[arg(short = 'o', long, value_name = "FILE", requires = "index")]
pub output: Option<PathBuf>,
/// k-mer size used for querying (same as --kmer-size in index)
#[arg(short = 'k', long, default_value_t = 31)]
pub kmer_size: usize,
@@ -23,6 +52,10 @@ pub struct EstimateArgs {
}
pub fn run(args: EstimateArgs) {
if let Some(index) = &args.index {
sharing::run(index, args.presence_threshold, args.output.as_deref());
return;
}
let (z, b, fp_window) = resolve_approx_params(args.findere_z, args.evidence_bits, args.fp);
let k_query = args.kmer_size;
+74
View File
@@ -0,0 +1,74 @@
//! `estimate --index`: observed k-mer sharing against the random-sequence
//! null model, as a CSV (see `DevDocMD`/`UserDocMD` `kmers.md`).
use std::fs::File;
use std::io::{self, BufWriter, Write};
use std::path::Path;
use std::sync::Arc;
use obikalgorithm::Algorithm;
use obikindex::KmerIndex;
use obikstats::{GenomeSharing, SharingNullModel, SharingRow};
use tracing::info;
pub(super) fn run(index_path: &Path, presence_threshold: u32, output: Option<&Path>) {
let fail = |what: &str, e: &dyn std::fmt::Display| -> ! {
eprintln!("error {what}: {e}");
std::process::exit(1);
};
let idx = Arc::new(KmerIndex::open(index_path).unwrap_or_else(|e| fail("opening index", &e)));
let k = idx.kmer_size();
let spectrum = GenomeSharing::new(Arc::clone(&idx), presence_threshold)
.run()
.unwrap_or_else(|e| fail("computing k-mer sharing", &e));
let rows = SharingNullModel::new(&spectrum.genome_kmers, k)
.compare(&spectrum)
.unwrap_or_else(|e| fail("comparing with the null model", &e));
let result = match output {
Some(path) => File::create(path)
.and_then(|f| write_csv(&rows, BufWriter::new(f)))
.map(|()| info!("sharing CSV written to {}", path.display())),
None => write_csv(&rows, io::stdout().lock()),
};
result.unwrap_or_else(|e| fail("writing CSV", &e));
}
pub(super) fn write_csv(rows: &[SharingRow], mut out: impl Write) -> io::Result<()> {
writeln!(out, "r,k_r,O_r,q_r,D_r,FP_r,FDR_r")?;
for row in rows {
writeln!(
out,
"{},{},{:.6e},{:.6e},{:.6e},{:.6e},{:.6e}",
row.r,
row.k_r,
row.observed,
row.expected,
row.deviation,
row.expected_false_positives,
row.fdr
)?;
}
out.flush()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn csv_has_header_and_one_row_per_r() {
let rows = vec![
SharingRow { r: 1, k_r: 20, observed: 1.0, expected: 1.0, deviation: 0.0, expected_false_positives: 30.0, fdr: 1.0 },
SharingRow { r: 2, k_r: 5, observed: 0.25, expected: 0.01, deviation: 0.24, expected_false_positives: 0.5, fdr: 0.1 },
];
let mut buf = Vec::new();
write_csv(&rows, &mut buf).unwrap();
let text = String::from_utf8(buf).unwrap();
let lines: Vec<&str> = text.lines().collect();
assert_eq!(lines[0], "r,k_r,O_r,q_r,D_r,FP_r,FDR_r");
assert_eq!(lines.len(), 3);
assert_eq!(lines[2], "2,5,2.500000e-1,1.000000e-2,2.400000e-1,5.000000e-1,1.000000e-1");
}
}
+4
View File
@@ -10,6 +10,10 @@
//! `run`), built on `obikidxcache::IndexCache` rather than hand-opening
//! each layer's files.
mod null_model;
mod sharing;
mod stats;
pub use null_model::{SharingNullModel, SharingRow};
pub use sharing::{GenomeSharing, SharingSpectrum};
pub use stats::{BitsPerKmer, GenomeKmerCounts, IndexBitsPerKmer};
+186
View File
@@ -0,0 +1,186 @@
//! Random-sequence null model for k-mer sharing (`kmers.md`, "Distinguishing
//! random from non-random k-mer sharing").
//!
//! Each genome `i` is modelled as a uniform random set of `N_i` distinct
//! canonical k-mers drawn from the `M = 4^k / 2` canonical k-mers (exact for
//! odd `k`: no k-mer is its own reverse complement). A given k-mer is then in
//! genome `i` with probability `p_i = N_i / M`, independently across genomes,
//! so the number `X` of genomes holding it is Poisson-binomial.
use crate::SharingSpectrum;
/// One row of the comparison between observed and expected sharing.
#[derive(Debug, Clone, PartialEq)]
pub struct SharingRow {
pub r: usize,
/// `k_r`: k-mers present in at least `r` genomes.
pub k_r: u64,
/// `O_r = k_r / k_1`.
pub observed: f64,
/// `q_r = P(X >= r | X >= 1)` under the null model.
pub expected: f64,
/// `D_r = O_r - q_r`.
pub deviation: f64,
/// `M_k P_null(X >= r)`: k-mers expected to reach `r` genomes by chance
/// over the whole canonical k-mer space.
pub expected_false_positives: f64,
/// `FDR(r)`: 1 when `k_r = 0`, else `min(1, M_k P_null(X >= r) / k_r)`.
pub fdr: f64,
}
/// Poisson-binomial law of the number of genomes holding a random k-mer.
pub struct SharingNullModel {
/// Size `M_k = 4^k / 2` of the canonical k-mer space.
m: f64,
p: Vec<f64>,
}
impl SharingNullModel {
/// `genome_kmers[i]` is `N_i`; `k` is the (indexed) k-mer size, odd.
pub fn new(genome_kmers: &[u64], k: usize) -> Self {
let m = 4f64.powi(k as i32) / 2.0;
Self { m, p: genome_kmers.iter().map(|&n| (n as f64 / m).min(1.0)).collect() }
}
/// `P(X >= r)` for `r = 0..=S`, by the dynamic programme of `kmers.md`
/// (genomes added one at a time, `O(S^2)`), then summed from the upper
/// end so that tiny tail probabilities keep their precision (no
/// `1 - Σ` cancellation).
pub fn tail(&self) -> Vec<f64> {
let s = self.p.len();
let mut dist = vec![0.0f64; s + 1];
dist[0] = 1.0;
for (i, &p) in self.p.iter().enumerate() {
for j in (1..=i + 1).rev() {
dist[j] = dist[j] * (1.0 - p) + dist[j - 1] * p;
}
dist[0] *= 1.0 - p;
}
let mut acc = 0.0;
for v in dist.iter_mut().rev() {
acc += *v;
*v = acc;
}
dist
}
/// The rows `r = 1..=S` of observed against expected sharing.
///
/// Errors when the index records no k-mer (`k_1 = 0`): `O_r` is undefined.
/// `q_r` is taken as 0 when `P(X >= 1)` underflows to 0 (no genome can
/// hold anything).
pub fn compare(&self, spectrum: &SharingSpectrum) -> Result<Vec<SharingRow>, String> {
let k_at_least = spectrum.k_at_least();
let k1 = *k_at_least.get(1).ok_or("the index has no genome")?;
if k1 == 0 {
return Err("the index records no k-mer present in any genome".into());
}
let tail = self.tail();
let p1 = tail[1];
Ok((1..tail.len())
.map(|r| {
let observed = k_at_least[r] as f64 / k1 as f64;
let expected = if p1 > 0.0 { tail[r] / p1 } else { 0.0 };
let k_r = k_at_least[r];
let expected_false_positives = self.m * tail[r];
let fdr = if k_r == 0 { 1.0 } else { (expected_false_positives / k_r as f64).min(1.0) };
SharingRow {
r,
k_r,
observed,
expected,
deviation: observed - expected,
expected_false_positives,
fdr,
}
})
.collect())
}
}
#[cfg(test)]
mod tests {
use super::*;
fn binom(n: usize, k: usize) -> f64 {
(1..=k).fold(1.0, |a, i| a * (n + 1 - i) as f64 / i as f64)
}
#[test]
fn equal_probabilities_give_the_binomial_tail() {
// k = 3: M = 32, N_i = 8 → p = 0.25 for all four genomes.
let m = SharingNullModel::new(&[8, 8, 8, 8], 3);
let tail = m.tail();
let (p, s) = (0.25f64, 4usize);
for r in 0..=s {
let expected: f64 =
(r..=s).map(|j| binom(s, j) * p.powi(j as i32) * (1.0 - p).powi((s - j) as i32)).sum();
assert!((tail[r] - expected).abs() < 1e-12, "r={r}: {} vs {expected}", tail[r]);
}
assert!((tail[0] - 1.0).abs() < 1e-12);
}
#[test]
fn unequal_probabilities_two_genomes() {
// p = (0.5, 0.25): P(X>=1) = 1 - 0.5*0.75, P(X>=2) = 0.125.
let m = SharingNullModel { m: 1.0, p: vec![0.5, 0.25] };
let t = m.tail();
assert!((t[1] - 0.625).abs() < 1e-15);
assert!((t[2] - 0.125).abs() < 1e-15);
}
#[test]
fn tiny_tail_keeps_its_precision() {
// k = 31, N = 1e9: p ≈ 2.2e-10; P(X>=3) over 3 genomes = p^3 ≈ 1e-29,
// far below what `1 - Σ` could resolve.
let m = SharingNullModel::new(&[1_000_000_000; 3], 31);
let p = 1e9 / (4f64.powi(31) / 2.0);
assert!((m.tail()[3] / p.powi(3) - 1.0).abs() < 1e-9);
}
#[test]
fn compare_first_row_has_zero_deviation() {
let spectrum = SharingSpectrum { genome_kmers: vec![10, 10], hist: vec![0, 15, 5] };
let rows = SharingNullModel::new(&spectrum.genome_kmers, 11).compare(&spectrum).unwrap();
assert_eq!(rows.len(), 2);
assert_eq!(rows[0].k_r, 20);
assert_eq!(rows[1].k_r, 5);
assert!((rows[0].observed - 1.0).abs() < 1e-15);
assert!(rows[0].deviation.abs() < 1e-12); // D_1 = 0 by construction
assert!((rows[1].observed - 0.25).abs() < 1e-15);
}
#[test]
fn fdr_is_expected_false_positives_over_k_r_capped_at_one() {
// k = 3: M = 32; N = (8, 8) → p = 0.25 each; P(X>=2) = 0.0625.
let model = SharingNullModel::new(&[8, 8], 3);
// 10 k-mers in exactly 1 genome, 2 in both → k_1 = 12, k_2 = 2.
let spectrum = SharingSpectrum { genome_kmers: vec![8, 8], hist: vec![0, 10, 2] };
let rows = model.compare(&spectrum).unwrap();
// r = 2: FP = 32 * 0.0625 = 2; FDR = 2 / 2 = 1 (not below the cap).
assert!((rows[1].expected_false_positives - 2.0).abs() < 1e-12);
assert!((rows[1].fdr - 1.0).abs() < 1e-12);
// r = 1: FP = 32 * (1 - 0.75^2) = 14 > k_1 = 12 → capped at 1.
assert!((rows[0].expected_false_positives - 14.0).abs() < 1e-12);
assert!((rows[0].fdr - 1.0).abs() < 1e-12);
// Fewer expected false positives than observed: 20 of 22 k-mers shared.
let spectrum = SharingSpectrum { genome_kmers: vec![8, 8], hist: vec![0, 2, 20] };
let rows = model.compare(&spectrum).unwrap();
assert!((rows[1].fdr - 2.0 / 20.0).abs() < 1e-12);
}
#[test]
fn fdr_is_one_when_no_kmer_reaches_r() {
let model = SharingNullModel::new(&[8, 8, 8], 3);
let spectrum = SharingSpectrum { genome_kmers: vec![8, 8, 8], hist: vec![0, 12, 0, 0] };
let rows = model.compare(&spectrum).unwrap();
assert_eq!(rows[2].k_r, 0);
assert_eq!(rows[2].fdr, 1.0);
}
#[test]
fn empty_index_is_an_error() {
let spectrum = SharingSpectrum { genome_kmers: vec![0, 0], hist: vec![0, 0, 0] };
assert!(SharingNullModel::new(&spectrum.genome_kmers, 11).compare(&spectrum).is_err());
}
}
+112
View File
@@ -0,0 +1,112 @@
//! How the k-mers of an index are shared across its genomes — the data the
//! null-model comparison of `DevDocMD`/`UserDocMD` `kmers.md` needs: the
//! number `N_i` of distinct k-mers of each genome and the number `k_r` of
//! k-mers present in at least `r` genomes.
use std::sync::Arc;
use obikalgorithm::Algorithm;
use obikidxcache::index_cache::IndexCache;
use obikindex::KmerIndex;
/// Layers scanned at once per `par_map_reduce` call — see
/// `obikidxcache::IndexCache::par_map_reduce`'s own docs.
const MAX_CONCURRENT_LAYERS: usize = 8;
/// Result of [`GenomeSharing`].
pub struct SharingSpectrum {
/// `N_i`: distinct k-mers present in genome `i`, one entry per genome.
pub genome_kmers: Vec<u64>,
/// `hist[j]`: number of k-mers present in exactly `j` genomes;
/// `n_genomes + 1` entries. `hist[0]` is non-zero only when the presence
/// threshold discards every occurrence of some k-mer.
pub hist: Vec<u64>,
}
impl SharingSpectrum {
pub fn n_genomes(&self) -> usize {
self.genome_kmers.len()
}
/// `k_r` for `r = 0..=n_genomes`: number of k-mers present in at least
/// `r` genomes. `k_at_least()[1]` is `k_1`, the number of k-mers
/// present in at least one genome.
pub fn k_at_least(&self) -> Vec<u64> {
let mut k = vec![0u64; self.hist.len()];
let mut acc = 0u64;
for r in (0..self.hist.len()).rev() {
acc += self.hist[r];
k[r] = acc;
}
k
}
}
/// Single pass over every layer computing a [`SharingSpectrum`]. A k-mer is
/// present in a genome when its value reaches `threshold` (on a presence
/// index every recorded k-mer is present, whatever `threshold`), so both `N_i`
/// and the histogram are taken at the same threshold. A k-mer lives in a
/// single layer of its partition, so per-layer results add up exactly.
pub struct GenomeSharing {
index: Arc<KmerIndex>,
threshold: u32,
}
impl GenomeSharing {
pub fn new(index: Arc<KmerIndex>, threshold: u32) -> Self {
Self { index, threshold }
}
}
/// `acc[i] += add[i]`, growing `acc` when `add` is longer.
fn add_into(acc: &mut Vec<u64>, add: &[u64]) {
if acc.len() < add.len() {
acc.resize(add.len(), 0);
}
for (a, b) in acc.iter_mut().zip(add) {
*a += b;
}
}
type Partial = obikalgorithm::Result<(Vec<u64>, Vec<u64>)>;
impl Algorithm for GenomeSharing {
type Output = SharingSpectrum;
fn run(&mut self) -> obikalgorithm::Result<SharingSpectrum> {
let n_genomes = self.index.meta().genomes()?.len();
let threshold = self.threshold;
let cache = IndexCache::new(Arc::clone(&self.index), None);
let (genome_kmers, hist) = cache.par_map_reduce(
MAX_CONCURRENT_LAYERS,
|| -> Partial { Ok((Vec::new(), Vec::new())) },
|layer| -> Partial {
let per_genome = layer.count_geq(threshold).to_vec();
let hist = layer.row_presence_histogram(threshold)?;
Ok((per_genome, hist))
},
|a, b| -> Partial {
let (mut ga, mut ha) = a?;
let (gb, hb) = b?;
add_into(&mut ga, &gb);
add_into(&mut ha, &hb);
Ok((ga, ha))
},
)?;
let mut genome_kmers = genome_kmers;
let mut hist = hist;
if genome_kmers.len() > n_genomes || hist.len() > n_genomes + 1 {
return Err(format!(
"a layer has more genome columns than the index declares ({n_genomes} genomes)"
)
.into());
}
// A layer may carry fewer columns than the index has genomes (and an
// index may have no layer at all): the missing columns are absent.
genome_kmers.resize(n_genomes, 0);
hist.resize(n_genomes + 1, 0);
Ok(SharingSpectrum { genome_kmers, hist })
}
}