Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
27 lines
1.7 KiB
Markdown
27 lines
1.7 KiB
Markdown
# Partitioning and indexing architecture
|
|
|
|
An index is split into a fixed number of **partitions**, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
|
|
|
|
## Routing
|
|
|
|
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition:
|
|
|
|
```
|
|
canonical minimizer → hash(minimizer) → p-bit value → partition index
|
|
```
|
|
|
|
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
|
|
|
|
## Parameter guidance
|
|
|
|
Even though $H$ already makes minimizer values well-distributed (see [Minimizer selection](minimizer_selection.md)), choosing $p$ well below the number of bits available in the minimizer ($2m$) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
|
|
|
|
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|
|
|----|-----------|-----------|------------|
|
|
| 9 | 18 | 6–8 | 64–256 |
|
|
| 11 | 22 | 8–10 | 256–1 024 |
|
|
| 13 | 26 | 10–12 | 1 024–4 096|
|
|
| 15 | 30 | 10–14 | 1 024–16 384|
|
|
|
|
The number of partitions must satisfy $p \le 2m$, and in practice $p$ is chosen well below that bound to leave a comfortable entropy margin. For $k=31$, $m=13$, $p=10$ (1024 partitions), partition load is well balanced on real genomic data.
|