From 1119af85ba3f1d74ed1f51dc287f92606faa91ab Mon Sep 17 00:00:00 2001 From: Eric Coissac Date: Fri, 20 Nov 2026 14:20:44 +0100 Subject: [PATCH] feat: update indexing pipeline and bump obikmer version to 1.4.7 Refactor the indexing process to support parallel phases, resumable builds, and three verification modes (Exact, Approximate, Hybrid). Update the on-disk layout to manage evidence-specific files and state in `index.meta`, and refine layer merging logic. Bump the obikmer package version from 1.4.6 to 1.4.7. - Modify indexing routing to use low bits of hash and enable parallel, independent phases for resumable builds - Update counting phase to build provisional MPHF and temporary counts, deleted unless `--keep-intermediate` is set - Combine quorum filtering with local assembly by filtering kmers during de Bruijn graph construction before unitig assembly - Expand verification modes to Exact, Approximate, and Hybrid, allowing post-build conversion between them - Update on-disk layout to include specific files for each evidence type and manage build state in `index.meta` - Refine layer merging to create new layers only for genuinely new kmer content and determine destination evidence mode from sources - Add `--sparse` option to packing logic for sparse representation support - Bump obikmer package version from 1.4.6 to 1.4.7 in Cargo.toml --- UserDocMD/formats/index_layout.md | 67 ++++++++++++++++++------------- src/obikmer/Cargo.toml | 2 +- 2 files changed, 40 insertions(+), 29 deletions(-) diff --git a/UserDocMD/formats/index_layout.md b/UserDocMD/formats/index_layout.md index 46303b39..da020c03 100644 --- a/UserDocMD/formats/index_layout.md +++ b/UserDocMD/formats/index_layout.md @@ -4,14 +4,13 @@ Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)): -1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition. +1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition. The number of partitions is always a power of two, so this routing amounts to keeping the low bits of the hash. 2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it. -3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage. -4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set. -5. **Local assembly.** The surviving kmers of each partition are assembled into unitigs — maximal non-branching runs of a local de Bruijn graph — such that every kmer appears exactly once, at one (unitig, offset) location. -6. **MPHF and evidence construction.** A minimal perfect hash function is built over the canonical kmers of each partition, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested. +3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. To accumulate these counts, a provisional minimal perfect hash function is built over the distinct kmers of the partition and a temporary count file is filled through it. Both are working files: they are deleted at the end of the build unless `--keep-intermediate` is given. A per-genome kmer frequency spectrum is produced at this stage, from the counts as they are before any abundance filtering. +4. **Quorum filtering and local assembly.** The kmers of the dereplicated super-kmers are fed one by one into a local de Bruijn graph of the partition. A kmer whose count falls outside the `--min-abundance`/`--max-abundance` range is dropped at this point, and never enters the graph; the super-kmers themselves are not rewritten. The kmers that remain are assembled into unitigs — maximal non-branching runs of the graph — such that every kmer appears exactly once, at one (unitig, offset) location. +5. **MPHF and evidence construction.** The final minimal perfect hash function is built over the canonical kmers that survived, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested; the counts come from the counting phase. -Phases 1–5 are independent per partition and run in parallel; phase 6 finalizes each partition once its kmer set is fixed. +Within each phase, the partitions are independent of one another and are processed in parallel. The phases themselves follow one another over the whole index, and the index records in its `index.meta` file how far the build has gone (after scatter, after counting, and when fully built). A build that was interrupted can therefore be resumed: the phases already completed are skipped. ## Minimal perfect hash function (MPHF) @@ -19,38 +18,50 @@ Each partition's surviving kmers are mapped to a dense range of integer slots by ## Evidence: exact vs. approximate -Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)): +Three verification modes exist. The exact and approximate modes are selected at build time (`index --approx`), and any of the three can be reached afterwards with [`convert`](../usage/convert.md): - **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup. - **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building. +- **Hybrid** (`convert --hybrid-evidence`): both the fingerprint and the exact evidence are kept. A lookup uses the fingerprint, which is fast and approximate, while the exact evidence remains stored alongside it. ## On-disk layout ``` -/ - index.meta global configuration (k, minimizer size, partition count, - evidence mode, whether counts are stored) and genome list/metadata - scatter.done / count.done / index.done build-progress sentinels - spectrums/