Refactor indexing logic using hash-based minimizer selection.
Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
This commit is contained in:
1 parent
755cf85334
commit
9da0876ee6
5 files changed
+686
-23
No files matched your search
@@ -2,27 +2,58 @@
|
||||
|
||||
## Definition
|
||||
|
||||
A **minimizer** of a kmer window is the m-mer ($m < k$) that is smallest, among all $k - m + 1$ overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent.
|
||||
The **minimizer** of a k-mer is the canonical form of the m-mer that is smallest according to a chosen ordering among the $k-m+1$ overlapping m-mers contained in the k-mer, with $m<k$ [@Roberts2004-rz].
|
||||
|
||||
The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
|
||||
For an m-mer $s$, its **canonical form** is defined as
|
||||
|
||||
## Hash-based ("random") minimizer
|
||||
$$
|
||||
s^c = \min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right),
|
||||
$$
|
||||
|
||||
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions.
|
||||
where $\operatorname{RC}(s)$ denotes the reverse complement of $s$.
|
||||
|
||||
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition.
|
||||
In **OBIkmer**, canonical m-mers are ordered according to a deterministic hash function $H$ applied to their canonical form:
|
||||
|
||||
$$
|
||||
s_1^c <_h s_2^c
|
||||
\quad\Longleftrightarrow\quad
|
||||
H(s_1^c) < H(s_2^c).
|
||||
$$
|
||||
|
||||
Thus, for a k-mer $K$, let
|
||||
|
||||
$$
|
||||
M(K)=(s_1,\ldots,s_{k-m+1})
|
||||
$$
|
||||
|
||||
be its sequence of overlapping m-mers. The minimizer is
|
||||
|
||||
$$
|
||||
\operatorname{minimizer}(K) = s_j^c,
|
||||
\qquad
|
||||
j=\underset{i}{\operatorname{argmin}}\;H(s_i^c).
|
||||
$$
|
||||
|
||||
In other words, the hash function defines the ordering of canonical m-mers, while the minimizer itself is the canonical form of the m-mer selected by that ordering.
|
||||
|
||||
Applying the minimizer to consecutive k-mers partitions a sequence into super-kmers, defined as maximal runs of consecutive k-mers sharing the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
|
||||
|
||||
## Hash-based minimizer ordering
|
||||
|
||||
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf].
|
||||
|
||||
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijective mixing function with good avalanche properties [@Steele2014-cz], its output behaves approximately as a composition-independent pseudorandom ordering of distinct m-mers. Under the usual random-hash assumption, each distinct m-mer in a window has approximately the same probability of being the minimum, independently of nucleotide composition.
|
||||
|
||||
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
|
||||
|
||||
### Hash function
|
||||
|
||||
The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed:
|
||||
The hash function \(H\) applies Stafford's 64-bit mixing function, variant 13 [@Stafford2011], to the m-mer encoding XORed with a seed $\sigma$:
|
||||
|
||||
$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
|
||||
|
||||
```
|
||||
```text
|
||||
H(x):
|
||||
x ← x ⊕ 0x9e3779b97f4a7c15
|
||||
x ← x ⊕ σ
|
||||
x ← x ⊕ (x >> 30)
|
||||
x ← x × 0xbf58476d1ce4e5b9
|
||||
x ← x ⊕ (x >> 27)
|
||||
@@ -30,13 +61,36 @@ H(x):
|
||||
return x ⊕ (x >> 31)
|
||||
```
|
||||
|
||||
The XOR seed avoids the finalizer's fixed point at 0 ($\text{mix64}(0) = 0$), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison.
|
||||
The hash uses a fixed non-zero seed
|
||||
|
||||
## Partition routing is independent of minimizer selection
|
||||
$$ \sigma=\texttt{0x9e3779b97f4a7c15}. $$
|
||||
|
||||
The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately:
|
||||
The choice of $\sigma$ is not arbitrary. Low-complexity m-mers (homopolymers and short tandem repeats) are disproportionately abundant in real genomes. If one of them happened to be the global argmin of $H$, it could therefore occur as the minimizer in far more windows than expected from the composition-uniform behavior described above. This is not a bias of the mixing function itself, but a consequence of the repeated occurrence of the same input in real sequence data.
|
||||
|
||||
- **Selection** uses $H$ applied to every candidate m-mer in the window, keeping the minimum.
|
||||
- **Partition routing** recomputes $H$ on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.
|
||||
In particular, with $\sigma=0$, the all-A m-mer is encoded as $0$, and
|
||||
|
||||
See [Partitioning and indexing architecture](indexing_architecture.md) for how the routing value is turned into a partition index.
|
||||
$$ \operatorname{Mix13}(0)=0, $$
|
||||
|
||||
This makes it the global argmin of the hash function. The non-zero seed eliminates the particular pathological correspondence between the all-A m-mer and the zero output of the mixer.
|
||||
|
||||
Exhaustive enumeration of all $4^m$ m-mers confirms that, with $\sigma=\texttt{0x9e3779b97f4a7c15}$, the global argmin is neither a homopolymer nor a periodic repeat for any tested \(m\):
|
||||
|
||||
| $m$ | argmin (canonical) | decoded sequence | minimal period |
|
||||
|---|---|---|---|
|
||||
| 3 | 16 | `CAA` | 3 |
|
||||
| 5 | 78 | `ACATG` | 5 |
|
||||
| 7 | 5512 | `CCCGAGA` | 7 |
|
||||
| 9 | 108760 | `CGGGATCGA` | 9 |
|
||||
| 11 | 179014 | `AAGGTGTCACG` | 11 |
|
||||
| 13 | 33759044 | `GAAATACTTCACA` | 13 |
|
||||
| 15 | 29869313 | `AACTACTTACCAAAC` | 15 |
|
||||
|
||||
When the minimal period of the sequence is $m$, as observed in the table above, the sequence is primitive: it cannot be represented as repetitions of a shorter sequence.
|
||||
|
||||
## Partition routing
|
||||
|
||||
A super-kmer's partition is the low $p$ bits of its minimizer's hash:
|
||||
|
||||
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
|
||||
|
||||
See [Partitioning and indexing architecture](indexing_architecture.md) for more details.
|
||||
Reference in new issue
Block a user