Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
1.7 KiB
Partitioning and indexing architecture
An index is split into a fixed number of partitions, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
Routing
The canonical minimizer of a super-kmer (see Minimizer selection) is hashed to produce a $p$-bit routing value that selects the destination partition:
canonical minimizer → hash(minimizer) → p-bit value → partition index
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see On-disk storage); the minimizer plays no further role once a super-kmer has reached its partition.
Parameter guidance
Even though H already makes minimizer values well-distributed (see Minimizer selection), choosing p well below the number of bits available in the minimizer (2m) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
Minimizer size m |
Minimizer bits (2m) |
Typical partition-index bits p |
Partitions |
|---|---|---|---|
| 9 | 18 | 6–8 | 64–256 |
| 11 | 22 | 8–10 | 256–1 024 |
| 13 | 26 | 10–12 | 1 024–4 096 |
| 15 | 30 | 10–14 | 1 024–16 384 |
The number of partitions must satisfy p \le 2m, and in practice p is chosen well below that bound to leave a comfortable entropy margin. For k=31, m=13, p=10 (1024 partitions), partition load is well balanced on real genomic data.