Files
obikmer/UserDocMD/theory/kmer_indexing/indexing_architecture.md
T
Eric Coissac e715528d34 Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
2026-10-13 08:38:49 +02:00

29 lines
1.8 KiB
Markdown

# Partitioning and indexing architecture
An index is split into a fixed number of **partitions**, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
## Routing
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition — the low $p$ bits of the minimizer's hash:
```
canonical minimizer → hash(minimizer) → p-bit value → partition index
```
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
## Parameter guidance
Even though $H$ already makes minimizer values well-distributed (see [Minimizer selection](minimizer_selection.md)), choosing $p$ well below the number of bits available in the minimizer ($2m$) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|----|-----------|-----------|------------|
| 9 | 18 | 6–8 | 64–256 |
| 11 | 22 | 8–10 | 256–1 024 |
| 13 | 26 | 10–12 | 1 024–4 096|
| 15 | 30 | 10–14 | 1 024–16 384|
The number of partitions must satisfy $p \le 2m$, and in practice $p$ is chosen well below that bound to leave a comfortable entropy margin. For $k=31$, $m=13$, $p=10$ (1024 partitions), partition load is well balanced on real genomic data.