Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
This commit is contained in:
1 parent
9da0876ee6
commit
e715528d34
19 files changed
+582
-712
No files matched your search
@@ -0,0 +1,28 @@
|
||||
# DNA encoding
|
||||
|
||||
## 2-bit nucleotide encoding
|
||||
|
||||
Every nucleotide is encoded on 2 bits, most-significant-bit first within each word:
|
||||
|
||||
| Base | Encoding |
|
||||
|------|----------|
|
||||
| A | `00` |
|
||||
| C | `01` |
|
||||
| G | `10` |
|
||||
| T | `11` |
|
||||
|
||||
The Watson-Crick complement of a base is its bitwise NOT on 2 bits: $\text{complement}(base) = \lnot base \mathbin{\&} \texttt{0b11}$.
|
||||
|
||||
## Kmer encoding
|
||||
|
||||
A kmer of length $k$ ($k \le 31$) fits in a single 64-bit word. The first nucleotide occupies the two most significant bits, each following nucleotide occupies the next two bits, and unused low-order bits are zero. Extracting nucleotide i (0-indexed from the 5′ end) is a shift-and-mask operation.
|
||||
|
||||
Reverse complement is computed by bit manipulation directly on the packed word, without any lookup table: complement every base, reverse the byte order, then reverse the order of 2-bit groups within each byte in two more passes, and finally realign the result to the most-significant bits.
|
||||
|
||||
## Canonical form
|
||||
|
||||
The canonical form of a packed 2-bit sequence is the lexicographic minimum of the sequence and its reverse complement:
|
||||
|
||||
$$\text{canonical}(x) = \min\big(x,\ \text{revcomp}(x)\big)$$
|
||||
|
||||
This is a single operation on the packed representation, so it applies identically regardless of what `x` represents — a kmer (see [Kmers](kmers.md)), an m-mer (see [Minimizer selection](minimizer_selection.md)), or a super-kmer (see [Super-kmers](superkmers.md)). Using the canonical form halves the relevant space and makes counting/selection strand-independent: a sequence and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Low-complexity kmer filter
|
||||
|
||||
Low-complexity kmers (homopolymer runs, tandem repeats) can dominate an index without carrying useful information. `obikmer` detects and excludes them during index construction using a normalized Shannon entropy score.
|
||||
|
||||
## Sub-word frequencies
|
||||
|
||||
For a kmer of length $k$ and a sub-word size $ws$ ($1 \le ws \le ws_{\max}$, default $ws_{\max} = 6$), the kmer is decomposed into its $k - ws + 1$ overlapping sub-words of length $ws$ by sliding a window across it. Each sub-word is tallied under its raw 2-bit-packed value, with no canonicalization.
|
||||
|
||||
## Corrected Shannon entropy
|
||||
|
||||
Let $f_j$ be the observed count of raw sub-word $j$, and $n_{\text{words}} = k - ws + 1$ the total number of sub-words. The entropy is:
|
||||
|
||||
$$H_{\text{corr}} = \log(n_{\text{words}}) - \frac{1}{n_{\text{words}}} \sum_j f_j \log f_j$$
|
||||
|
||||
## Small-sample correction
|
||||
|
||||
Because only $n_{\text{words}}$ sub-words are observed among up to $4^{ws}$ possible values, the achievable maximum entropy $H_{\max}$ is bounded below $\log(4^{ws})$ for small samples. $H_{\max}$ is computed from the most uniform integer distribution achievable with $n_{\text{words}}$ observations over $4^{ws}$ categories. The normalized entropy is:
|
||||
|
||||
$$\hat{H}(ws) = \frac{H_{\text{corr}}}{H_{\max}} \in [0, 1]$$
|
||||
|
||||
A value near 0 indicates low complexity (e.g. a homopolymer run); near 1 indicates high complexity, characteristic of a random sequence.
|
||||
|
||||
## Final score
|
||||
|
||||
The filter evaluates $\hat{H}(ws)$ for every word size from 1 to ws_max and keeps the minimum:
|
||||
|
||||
$$\text{entropy}(kmer) = \min_{ws=1}^{ws_{\max}} \hat{H}(ws)$$
|
||||
|
||||
Taking the minimum across word sizes ensures that repetition at any scale is detected: a homopolymer is caught at $ws=1$, a dinucleotide repeat at $ws=2$, and so on. A kmer is rejected if its entropy score falls below a threshold $\theta$ (default 0.7), a configurable collection parameter.
|
||||
|
||||
## Properties
|
||||
|
||||
The entropy score depends only on the kmer sequence itself, not on where or how many times it occurs:
|
||||
|
||||
- **Orientation invariance**: a kmer and its reverse complement always receive the same score.
|
||||
- **Context independence**: a given kmer is always accepted or always rejected, regardless of which genome or read it appears in. The filter defines a fixed partition of the kmer space into low-complexity and valid kmers.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Partitioning and indexing architecture
|
||||
|
||||
An index is split into a fixed number of **partitions**, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
|
||||
|
||||
## Routing
|
||||
|
||||
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition — the low $p$ bits of the minimizer's hash:
|
||||
|
||||
```
|
||||
canonical minimizer → hash(minimizer) → p-bit value → partition index
|
||||
```
|
||||
|
||||
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
|
||||
|
||||
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
|
||||
|
||||
## Parameter guidance
|
||||
|
||||
Even though $H$ already makes minimizer values well-distributed (see [Minimizer selection](minimizer_selection.md)), choosing $p$ well below the number of bits available in the minimizer ($2m$) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
|
||||
|
||||
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|
||||
|----|-----------|-----------|------------|
|
||||
| 9 | 18 | 6–8 | 64–256 |
|
||||
| 11 | 22 | 8–10 | 256–1 024 |
|
||||
| 13 | 26 | 10–12 | 1 024–4 096|
|
||||
| 15 | 30 | 10–14 | 1 024–16 384|
|
||||
|
||||
The number of partitions must satisfy $p \le 2m$, and in practice $p$ is chosen well below that bound to leave a comfortable entropy margin. For $k=31$, $m=13$, $p=10$ (1024 partitions), partition load is well balanced on real genomic data.
|
||||
@@ -0,0 +1,6 @@
|
||||
# Kmers
|
||||
|
||||
A **kmer** is a DNA subsequence of fixed length $k$. Two constraints apply to $k$, both enforced when a command starts (an invalid value exits immediately with an error):
|
||||
|
||||
- $k \in [11, 31]$: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base ($k \le 32$ is the hard limit; $k < 11$ gives insufficient specificity).
|
||||
- $k$ **is odd**: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see [DNA encoding](encoding.md)) to be well defined.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Minimizer selection
|
||||
|
||||
## Definition
|
||||
|
||||
The **minimizer** of a k-mer is the canonical form of the m-mer that is smallest according to a chosen ordering among the $k-m+1$ overlapping m-mers contained in the k-mer, with $m<k$ [@Roberts2004-rz]. Canonical form is the same operation defined for a full kmer (see [DNA encoding](encoding.md)), applied here to an m-mer $s$ instead: $s^c = \min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)$.
|
||||
|
||||
In **OBIkmer**, canonical m-mers are ordered according to a deterministic hash function $H$ applied to their canonical form:
|
||||
|
||||
$$
|
||||
s_1^c <_h s_2^c
|
||||
\quad\Longleftrightarrow\quad
|
||||
H(s_1^c) < H(s_2^c).
|
||||
$$
|
||||
|
||||
Thus, for a k-mer $K$, let
|
||||
|
||||
$$
|
||||
M(K)=(s_1,\ldots,s_{k-m+1})
|
||||
$$
|
||||
|
||||
be its sequence of overlapping m-mers. The minimizer is
|
||||
|
||||
$$
|
||||
\operatorname{minimizer}(K) = s_j^c,
|
||||
\qquad
|
||||
j=\underset{i}{\operatorname{argmin}}\;H(s_i^c).
|
||||
$$
|
||||
|
||||
In other words, the hash function defines the ordering of canonical m-mers, while the minimizer itself is the canonical form of the m-mer selected by that ordering.
|
||||
|
||||
Applying the minimizer to consecutive k-mers partitions a sequence into super-kmers (see [Super-kmers](superkmers.md)).
|
||||
|
||||
## Hash-based minimizer ordering
|
||||
|
||||
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf].
|
||||
|
||||
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijective mixing function with good avalanche properties [@Steele2014-cz], its output behaves approximately as a composition-independent pseudorandom ordering of distinct m-mers. Under the usual random-hash assumption, each distinct m-mer in a window has approximately the same probability of being the minimum, independently of nucleotide composition.
|
||||
|
||||
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
|
||||
|
||||
### Hash function
|
||||
|
||||
The hash function \(H\) applies Stafford's 64-bit mixing function, variant 13 [@Stafford2011], to the m-mer encoding XORed with a seed $\sigma$:
|
||||
|
||||
|
||||
```text
|
||||
H(x):
|
||||
x ← x ⊕ σ
|
||||
x ← x ⊕ (x >> 30)
|
||||
x ← x × 0xbf58476d1ce4e5b9
|
||||
x ← x ⊕ (x >> 27)
|
||||
x ← x × 0x94d049bb133111eb
|
||||
return x ⊕ (x >> 31)
|
||||
```
|
||||
|
||||
The hash uses a fixed non-zero seed
|
||||
|
||||
$$ \sigma=\texttt{0x9e3779b97f4a7c15}. $$
|
||||
|
||||
The choice of $\sigma$ is not arbitrary. Low-complexity m-mers (homopolymers and short tandem repeats) are disproportionately abundant in real genomes. If one of them happened to be the global argmin of $H$, it could therefore occur as the minimizer in far more windows than expected from the composition-uniform behavior described above. This is not a bias of the mixing function itself, but a consequence of the repeated occurrence of the same input in real sequence data.
|
||||
|
||||
In particular, with $\sigma=0$, the all-A m-mer is encoded as $0$, and
|
||||
|
||||
$$ \operatorname{Mix13}(0)=0, $$
|
||||
|
||||
This makes it the global argmin of the hash function. The non-zero seed eliminates the particular pathological correspondence between the all-A m-mer and the zero output of the mixer.
|
||||
|
||||
Exhaustive enumeration of all $4^m$ m-mers confirms that, with $\sigma=\texttt{0x9e3779b97f4a7c15}$, the global argmin is neither a homopolymer nor a periodic repeat for any tested \(m\):
|
||||
|
||||
| $m$ | argmin (canonical) | decoded sequence | minimal period |
|
||||
|---|---|---|---|
|
||||
| 3 | 16 | `CAA` | 3 |
|
||||
| 5 | 78 | `ACATG` | 5 |
|
||||
| 7 | 5512 | `CCCGAGA` | 7 |
|
||||
| 9 | 108760 | `CGGGATCGA` | 9 |
|
||||
| 11 | 179014 | `AAGGTGTCACG` | 11 |
|
||||
| 13 | 33759044 | `GAAATACTTCACA` | 13 |
|
||||
| 15 | 29869313 | `AACTACTTACCAAAC` | 15 |
|
||||
|
||||
When the minimal period of the sequence is $m$, as observed in the table above, the sequence is primitive: it cannot be represented as repetitions of a shorter sequence.
|
||||
|
||||
See [Partitioning and indexing architecture](indexing_architecture.md) for how this hash routes a super-kmer to a partition.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Super-kmers
|
||||
|
||||
A **super-kmer** is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see [Minimizer selection](minimizer_selection.md)). Each kmer in the run overlaps the next by $k-1$ nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
|
||||
|
||||
For a random minimizer of length $m$ over kmers of length $k$, the expected length of a super-kmer is approximately [@Zheng2020-ji; @Golan2025-xf]:
|
||||
|
||||
$$L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1$$
|
||||
|
||||
For $k=31$, $m=13$ this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.
|
||||
|
||||
## Canonical super-kmers
|
||||
|
||||
A **canonical super-kmer** is the canonical form of a super-kmer (see [DNA encoding](encoding.md)): the lexicographic minimum of the super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.
|
||||
|
||||
Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.
|
||||
Reference in new issue
Block a user