Files
obikmer/UserDocMD/formats/index_layout.md
T

67 lines
9.0 KiB
Markdown
Raw Normal View History

# Index construction and on-disk layout
## Construction pipeline
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)):
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition. The number of partitions is always a power of two, so this routing amounts to keeping the low bits of the hash.
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. To accumulate these counts, a provisional minimal perfect hash function is built over the distinct kmers of the partition and a temporary count file is filled through it. Both are working files: they are deleted at the end of the build unless `--keep-intermediate` is given. A per-genome kmer frequency spectrum is produced at this stage, from the counts as they are before any abundance filtering.
4. **Quorum filtering and local assembly.** The kmers of the dereplicated super-kmers are fed one by one into a local de Bruijn graph of the partition. A kmer whose count falls outside the `--min-abundance`/`--max-abundance` range is dropped at this point, and never enters the graph; the super-kmers themselves are not rewritten. The kmers that remain are assembled into unitigs — maximal non-branching runs of the graph — such that every kmer appears exactly once, at one (unitig, offset) location.
5. **MPHF and evidence construction.** The final minimal perfect hash function is built over the canonical kmers that survived, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested; the counts come from the counting phase.
Within each phase, the partitions are independent of one another and are processed in parallel. The phases themselves follow one another over the whole index, and the index records in its `index.meta` file how far the build has gone (after scatter, after counting, and when fully built). A build that was interrupted can therefore be resumed: the phases already completed are skipped.
## Minimal perfect hash function (MPHF)
Each partition's surviving kmers are mapped to a dense range of integer slots by a minimal perfect hash function: no collisions, near-optimal space (a few bits per key), O(1) lookup. Because an MPHF maps *any* input to some slot — including kmers that were never indexed — a lookup alone cannot distinguish a genuinely indexed kmer from an arbitrary one; every lookup is followed by an evidence check.
## Evidence: exact vs. approximate
Three verification modes exist. The exact and approximate modes are selected at build time (`index --approx`), and any of the three can be reached afterwards with [`convert`](../usage/convert.md):
- **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
- **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
- **Hybrid** (`convert --hybrid-evidence`): both the fingerprint and the exact evidence are kept. A lookup uses the fingerprint, which is fast and approximate, while the exact evidence remains stored alongside it.
## On-disk layout
```
📁 <index_root>/
├── 📄 index.meta configuration (k, minimizer size, partition count,
│ evidence mode, counts stored or not), genome list
│ and metadata, build state (empty, scattered,
│ counted, indexed)
├── 📄 .obikmer.lock lock files: commands that read the index share it,
├── 📄 .obikmer.gate commands that write into it hold it exclusively
├── 📁 spectrums/
│ └── 📄 <label>.json per-genome kmer frequency histogram
└── 📁 partitions/
├── 📁 part_00000/
│ └── 📁 index/
│ ├── 📁 layer_0/
│ │ ├── 📄 layer_meta.json number of kmers in this layer
│ │ ├── 📄 unitigs.bin reconstructible kmer sequence data — always kept
│ │ ├── 📄 unitigs.bin.idx random access into unitigs.bin (exact, hybrid)
│ │ ├── 📄 mphf.bin the minimal perfect hash function
│ │ ├── 📄 evidence.bin exact evidence (exact, hybrid)
│ │ ├── 📄 fingerprint.bin approximate evidence (approximate, hybrid)
│ │ ├── 📁 counts/ per-genome kmer counts (if requested)
│ │ └── 📁 presence/ per-genome presence/absence bits
│ └── 📁 layer_1/, layer_2/, … added by later merges, same structure
├── 📁 part_00001/
│ └── …
└── 📁 part_NNNNN/
```
The build state is stored in `index.meta` and not in separate marker files: it is what allows an interrupted build to resume. The number of layers of a partition is not stored anywhere either; it is found by looking for `layer_0`, `layer_1`, ... until one is missing. A partition that received no kmer at all (which happens with small inputs and many partitions) has no layer on disk.
The matrix directory of a layer, `counts/` or `presence/`, exists in three forms, which hold the same information. A freshly built index stores one file per genome, with a small metadata file (for counts, `col_000000.pciv`, `col_000001.pciv`, ... and `meta.json`). The dense packed form, produced by [`pack`](../usage/pack.md), stores the whole matrix in a single file. The sparse packed form, produced by `pack --sparse` and, by default, by [`merge`](../usage/merge.md), stores for each kmer only the list of genomes that carry it, with the lists shared between kmers deduplicated (for counts, the directory then holds files such as `dict_values.bin`, `singleton.pfiv`, `multi.pfiv` and `sparse_meta.json`). Which form a layer uses is detected from the files found in the directory.
`unitigs.bin` is the only file from which the indexed kmer content can be fully recovered; it is always retained. Every other file (MPHF, evidence, counts) is derived from it.
A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. The genomes of the merged sources are added as new columns to the count/presence data of the existing layers; only genuinely new kmer content, that is kmers absent from every existing layer of the partition, is assembled into a new layer. A kmer therefore belongs to exactly one layer of its partition. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
Sources merged together must share the same kmer size, minimizer size and partition count; any mismatch is rejected rather than silently reconciled. The evidence mode, on the other hand, may differ between sources: as soon as one source is approximate or hybrid, the destination is chosen among the approximate or hybrid sources, and the merged index takes its evidence mode. A count merge additionally requires every source to store counts. Finally, an index whose matrices are already packed, which includes any merge output, can be merged in as a source but not used as the destination, so at least one source must still be unpacked.
[`pack`](../usage/pack.md) consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time; with `--sparse`, it also switches to the sparse form described above.