Refactor indexing logic using hash-based minimizer selection.

Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
This commit is contained in:
Eric Coissac committed 2026-10-13 08:34:51 +02:00
1 parent 755cf85334
commit 9da0876ee6
5 files changed
+686 -23

No files matched your search

+2 -8
View File
@@ -10,18 +10,12 @@ The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_sel
canonical minimizer → hash(minimizer) → p-bit value → partition index
```
The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
## Why hashing is necessary
A canonical minimizer is an m-mer ($m \in \{9, 11, 13, 15\}$), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.
Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits $p$ is chosen well below the number of bits available in the minimizer ($2m$): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
## Parameter guidance
Even though $H$ already makes minimizer values well-distributed (see [Minimizer selection](minimizer_selection.md)), choosing $p$ well below the number of bits available in the minimizer ($2m$) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|----|-----------|-----------|------------|
| 9 | 18 | 6–8 | 64–256 |
+69 -15
View File
@@ -2,27 +2,58 @@
## Definition
A **minimizer** of a kmer window is the m-mer ($m < k$) that is smallest, among all $k - m + 1$ overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent.
The **minimizer** of a k-mer is the canonical form of the m-mer that is smallest according to a chosen ordering among the $k-m+1$ overlapping m-mers contained in the k-mer, with $m<k$ [@Roberts2004-rz].
The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
For an m-mer $s$, its **canonical form** is defined as
## Hash-based ("random") minimizer
$$
s^c = \min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right),
$$
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions.
where $\operatorname{RC}(s)$ denotes the reverse complement of $s$.
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition.
In **OBIkmer**, canonical m-mers are ordered according to a deterministic hash function $H$ applied to their canonical form:
$$
s_1^c <_h s_2^c
\quad\Longleftrightarrow\quad
H(s_1^c) < H(s_2^c).
$$
Thus, for a k-mer $K$, let
$$
M(K)=(s_1,\ldots,s_{k-m+1})
$$
be its sequence of overlapping m-mers. The minimizer is
$$
\operatorname{minimizer}(K) = s_j^c,
\qquad
j=\underset{i}{\operatorname{argmin}}\;H(s_i^c).
$$
In other words, the hash function defines the ordering of canonical m-mers, while the minimizer itself is the canonical form of the m-mer selected by that ordering.
Applying the minimizer to consecutive k-mers partitions a sequence into super-kmers, defined as maximal runs of consecutive k-mers sharing the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
## Hash-based minimizer ordering
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf].
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijective mixing function with good avalanche properties [@Steele2014-cz], its output behaves approximately as a composition-independent pseudorandom ordering of distinct m-mers. Under the usual random-hash assumption, each distinct m-mer in a window has approximately the same probability of being the minimum, independently of nucleotide composition.
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
### Hash function
The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed:
The hash function \(H\) applies Stafford's 64-bit mixing function, variant 13 [@Stafford2011], to the m-mer encoding XORed with a seed $\sigma$:
$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
```
```text
H(x):
x ← x ⊕ 0x9e3779b97f4a7c15
x ← x ⊕ σ
x ← x ⊕ (x >> 30)
x ← x × 0xbf58476d1ce4e5b9
x ← x ⊕ (x >> 27)
@@ -30,13 +61,36 @@ H(x):
return x ⊕ (x >> 31)
```
The XOR seed avoids the finalizer's fixed point at 0 ($\text{mix64}(0) = 0$), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison.
The hash uses a fixed non-zero seed
## Partition routing is independent of minimizer selection
$$ \sigma=\texttt{0x9e3779b97f4a7c15}. $$
The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately:
The choice of $\sigma$ is not arbitrary. Low-complexity m-mers (homopolymers and short tandem repeats) are disproportionately abundant in real genomes. If one of them happened to be the global argmin of $H$, it could therefore occur as the minimizer in far more windows than expected from the composition-uniform behavior described above. This is not a bias of the mixing function itself, but a consequence of the repeated occurrence of the same input in real sequence data.
- **Selection** uses $H$ applied to every candidate m-mer in the window, keeping the minimum.
- **Partition routing** recomputes $H$ on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.
In particular, with $\sigma=0$, the all-A m-mer is encoded as $0$, and
See [Partitioning and indexing architecture](indexing_architecture.md) for how the routing value is turned into a partition index.
$$ \operatorname{Mix13}(0)=0, $$
This makes it the global argmin of the hash function. The non-zero seed eliminates the particular pathological correspondence between the all-A m-mer and the zero output of the mixer.
Exhaustive enumeration of all $4^m$ m-mers confirms that, with $\sigma=\texttt{0x9e3779b97f4a7c15}$, the global argmin is neither a homopolymer nor a periodic repeat for any tested \(m\):
| $m$ | argmin (canonical) | decoded sequence | minimal period |
|---|---|---|---|
| 3 | 16 | `CAA` | 3 |
| 5 | 78 | `ACATG` | 5 |
| 7 | 5512 | `CCCGAGA` | 7 |
| 9 | 108760 | `CGGGATCGA` | 9 |
| 11 | 179014 | `AAGGTGTCACG` | 11 |
| 13 | 33759044 | `GAAATACTTCACA` | 13 |
| 15 | 29869313 | `AACTACTTACCAAAC` | 15 |
When the minimal period of the sequence is $m$, as observed in the table above, the sequence is primitive: it cannot be represented as repetitions of a shorter sequence.
## Partition routing
A super-kmer's partition is the low $p$ bits of its minimizer's hash:
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
See [Partitioning and indexing architecture](indexing_architecture.md) for more details.