# Index construction and on-disk layout ## Construction pipeline Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)): 1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition. The number of partitions is always a power of two, so this routing amounts to keeping the low bits of the hash. 2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it. 3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. To accumulate these counts, a provisional minimal perfect hash function is built over the distinct kmers of the partition and a temporary count file is filled through it. Both are working files: they are deleted at the end of the build unless `--keep-intermediate` is given. A per-genome kmer frequency spectrum is produced at this stage, from the counts as they are before any abundance filtering. 4. **Quorum filtering and local assembly.** The kmers of the dereplicated super-kmers are fed one by one into a local de Bruijn graph of the partition. A kmer whose count falls outside the `--min-abundance`/`--max-abundance` range is dropped at this point, and never enters the graph; the super-kmers themselves are not rewritten. The kmers that remain are assembled into unitigs — maximal non-branching runs of the graph — such that every kmer appears exactly once, at one (unitig, offset) location. 5. **MPHF and evidence construction.** The final minimal perfect hash function is built over the canonical kmers that survived, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested; the counts come from the counting phase. Within each phase, the partitions are independent of one another and are processed in parallel. The phases themselves follow one another over the whole index, and the index records in its `index.meta` file how far the build has gone (after scatter, after counting, and when fully built). A build that was interrupted can therefore be resumed: the phases already completed are skipped. ## Minimal perfect hash function (MPHF) Each partition's surviving kmers are mapped to a dense range of integer slots by a minimal perfect hash function: no collisions, near-optimal space (a few bits per key), O(1) lookup. Because an MPHF maps *any* input to some slot — including kmers that were never indexed — a lookup alone cannot distinguish a genuinely indexed kmer from an arbitrary one; every lookup is followed by an evidence check. ## Evidence: exact vs. approximate Three verification modes exist. The exact and approximate modes are selected at build time (`index --approx`), and any of the three can be reached afterwards with [`convert`](../usage/convert.md): - **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup. - **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building. - **Hybrid** (`convert --hybrid-evidence`): both the fingerprint and the exact evidence are kept. A lookup uses the fingerprint, which is fast and approximate, while the exact evidence remains stored alongside it. ## On-disk layout ``` 📁 / ├── 📄 index.meta configuration (k, minimizer size, partition count, │ evidence mode, counts stored or not), genome list │ and metadata, build state (empty, scattered, │ counted, indexed) ├── 📄 .obikmer.lock lock files: commands that read the index share it, ├── 📄 .obikmer.gate commands that write into it hold it exclusively ├── 📁 spectrums/ │ └── 📄