Eric Coissac
6acafa7f2c
Introduces a comprehensive documentation set covering theoretical foundations, CLI usage, installation, and system architecture. Adds MkDocs configuration and Makefile targets to generate, serve with live reload, and clean the documentation site. Includes citation styles and bibliography files for academic references.
2.4 KiB
Partitioning and indexing architecture
An index is split into a fixed number of partitions, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
Routing
The canonical minimizer of a super-kmer (see Minimizer selection) is hashed to produce a $p$-bit routing value that selects the destination partition:
canonical minimizer → hash(minimizer) → p-bit value → partition index
The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see On-disk storage); the minimizer plays no further role once a super-kmer has reached its partition.
Why hashing is necessary
A canonical minimizer is an m-mer (m \in \{9, 11, 13, 15\}), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.
Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits p is chosen well below the number of bits available in the minimizer (2m): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
Parameter guidance
Minimizer size m |
Minimizer bits (2m) |
Typical partition-index bits p |
Partitions |
|---|---|---|---|
| 9 | 18 | 6–8 | 64–256 |
| 11 | 22 | 8–10 | 256–1 024 |
| 13 | 26 | 10–12 | 1 024–4 096 |
| 15 | 30 | 10–14 | 1 024–16 384 |
The number of partitions must satisfy p \le 2m, and in practice p is chosen well below that bound to leave a comfortable entropy margin. For k=31, m=13, p=10 (1024 partitions), partition load is well balanced on real genomic data.