feat: update indexing pipeline and bump obikmer version to 1.4.7
Release / create-release (push) Successful in 1m15s
Release / build-linux-x86_64 (push) Successful in 6m43s
Release / build-macos-arm64 (push) Successful in 1m57s
ci.yml / build (pull_request) Successful in 3m42s

Refactor the indexing process to support parallel phases, resumable builds, and three verification modes (Exact, Approximate, Hybrid). Update the on-disk layout to manage evidence-specific files and state in `index.meta`, and refine layer merging logic. Bump the obikmer package version from 1.4.6 to 1.4.7.

- Modify indexing routing to use low bits of hash and enable parallel, independent phases for resumable builds
- Update counting phase to build provisional MPHF and temporary counts, deleted unless `--keep-intermediate` is set
- Combine quorum filtering with local assembly by filtering kmers during de Bruijn graph construction before unitig assembly
- Expand verification modes to Exact, Approximate, and Hybrid, allowing post-build conversion between them
- Update on-disk layout to include specific files for each evidence type and manage build state in `index.meta`
- Refine layer merging to create new layers only for genuinely new kmer content and determine destination evidence mode from sources
- Add `--sparse` option to packing logic for sparse representation support
- Bump obikmer package version from 1.4.6 to 1.4.7 in Cargo.toml
This commit is contained in:
Eric Coissac committed 2026-11-20 14:21:01 +01:00
1 parent 36c7247acb
commit 1119af85ba
2 files changed
+40 -29

No files matched your search

+39 -28
View File
@@ -4,14 +4,13 @@
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)): Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)):
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition. 1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition. The number of partitions is always a power of two, so this routing amounts to keeping the low bits of the hash.
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it. 2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage. 3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. To accumulate these counts, a provisional minimal perfect hash function is built over the distinct kmers of the partition and a temporary count file is filled through it. Both are working files: they are deleted at the end of the build unless `--keep-intermediate` is given. A per-genome kmer frequency spectrum is produced at this stage, from the counts as they are before any abundance filtering.
4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set. 4. **Quorum filtering and local assembly.** The kmers of the dereplicated super-kmers are fed one by one into a local de Bruijn graph of the partition. A kmer whose count falls outside the `--min-abundance`/`--max-abundance` range is dropped at this point, and never enters the graph; the super-kmers themselves are not rewritten. The kmers that remain are assembled into unitigs — maximal non-branching runs of the graph — such that every kmer appears exactly once, at one (unitig, offset) location.
5. **Local assembly.** The surviving kmers of each partition are assembled into unitigs — maximal non-branching runs of a local de Bruijn graph — such that every kmer appears exactly once, at one (unitig, offset) location. 5. **MPHF and evidence construction.** The final minimal perfect hash function is built over the canonical kmers that survived, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested; the counts come from the counting phase.
6. **MPHF and evidence construction.** A minimal perfect hash function is built over the canonical kmers of each partition, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested.
Phases 1–5 are independent per partition and run in parallel; phase 6 finalizes each partition once its kmer set is fixed. Within each phase, the partitions are independent of one another and are processed in parallel. The phases themselves follow one another over the whole index, and the index records in its `index.meta` file how far the build has gone (after scatter, after counting, and when fully built). A build that was interrupted can therefore be resumed: the phases already completed are skipped.
## Minimal perfect hash function (MPHF) ## Minimal perfect hash function (MPHF)
@@ -19,38 +18,50 @@ Each partition's surviving kmers are mapped to a dense range of integer slots by
## Evidence: exact vs. approximate ## Evidence: exact vs. approximate
Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`convert`](../usage/convert.md)): Three verification modes exist. The exact and approximate modes are selected at build time (`index --approx`), and any of the three can be reached afterwards with [`convert`](../usage/convert.md):
- **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup. - **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
- **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building. - **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
- **Hybrid** (`convert --hybrid-evidence`): both the fingerprint and the exact evidence are kept. A lookup uses the fingerprint, which is fast and approximate, while the exact evidence remains stored alongside it.
## On-disk layout ## On-disk layout
``` ```
<index_root>/ 📁 <index_root>/
index.meta global configuration (k, minimizer size, partition count, ├── 📄 index.meta configuration (k, minimizer size, partition count,
evidence mode, whether counts are stored) and genome list/metadata │ evidence mode, counts stored or not), genome list
scatter.done / count.done / index.done build-progress sentinels │ and metadata, build state (empty, scattered,
spectrums/<label>.json per-genome kmer frequency histogram │ counted, indexed)
partitions/ ├── 📄 .obikmer.lock lock files: commands that read the index share it,
part_00000/ ... part_NNNNN/ ├── 📄 .obikmer.gate commands that write into it hold it exclusively
index/ ├── 📁 spectrums/
meta.json number of layers in this partition │ └── 📄 <label>.json per-genome kmer frequency histogram
layer_0/ └── 📁 partitions/
unitigs.bin reconstructible kmer sequence data — always kept ├── 📁 part_00000/
unitigs.bin.idx random-access index into unitigs.bin (exact evidence only) │ └── 📁 index/
mphf.bin the minimal perfect hash function │ ├── 📁 layer_0/
evidence.bin exact evidence (exact mode only) │ │ ├── 📄 layer_meta.json number of kmers in this layer
fingerprint.bin approximate evidence (approximate mode only) │ │ ├── 📄 unitigs.bin reconstructible kmer sequence data — always kept
counts/ per-genome kmer counts (if counts were requested) │ │ ├── 📄 unitigs.bin.idx random access into unitigs.bin (exact, hybrid)
presence/ per-genome presence/absence bits │ │ ├── 📄 mphf.bin the minimal perfect hash function
layer_1/, layer_2/, ... added by later merges, same internal structure │ │ ├── 📄 evidence.bin exact evidence (exact, hybrid)
│ │ ├── 📄 fingerprint.bin approximate evidence (approximate, hybrid)
│ │ ├── 📁 counts/ per-genome kmer counts (if requested)
│ │ └── 📁 presence/ per-genome presence/absence bits
│ └── 📁 layer_1/, layer_2/, … added by later merges, same structure
├── 📁 part_00001/
│ └── …
└── 📁 part_NNNNN/
``` ```
The build state is stored in `index.meta` and not in separate marker files: it is what allows an interrupted build to resume. The number of layers of a partition is not stored anywhere either; it is found by looking for `layer_0`, `layer_1`, ... until one is missing. A partition that received no kmer at all (which happens with small inputs and many partitions) has no layer on disk.
The matrix directory of a layer, `counts/` or `presence/`, exists in three forms, which hold the same information. A freshly built index stores one file per genome, with a small metadata file (for counts, `col_000000.pciv`, `col_000001.pciv`, ... and `meta.json`). The dense packed form, produced by [`pack`](../usage/pack.md), stores the whole matrix in a single file. The sparse packed form, produced by `pack --sparse` and, by default, by [`merge`](../usage/merge.md), stores for each kmer only the list of genomes that carry it, with the lists shared between kmers deduplicated (for counts, the directory then holds files such as `dict_values.bin`, `singleton.pfiv`, `multi.pfiv` and `sparse_meta.json`). Which form a layer uses is detected from the files found in the directory.
`unitigs.bin` is the only file from which the indexed kmer content can be fully recovered; it is always retained. Every other file (MPHF, evidence, counts) is derived from it. `unitigs.bin` is the only file from which the indexed kmer content can be fully recovered; it is always retained. Every other file (MPHF, evidence, counts) is derived from it.
A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn. A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. The genomes of the merged sources are added as new columns to the count/presence data of the existing layers; only genuinely new kmer content, that is kmers absent from every existing layer of the partition, is assembled into a new layer. A kmer therefore belongs to exactly one layer of its partition. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`convert`](../usage/convert.md) one of the sources first if needed. Sources merged together must share the same kmer size, minimizer size and partition count; any mismatch is rejected rather than silently reconciled. The evidence mode, on the other hand, may differ between sources: as soon as one source is approximate or hybrid, the destination is chosen among the approximate or hybrid sources, and the merged index takes its evidence mode. A count merge additionally requires every source to store counts. Finally, an index whose matrices are already packed, which includes any merge output, can be merged in as a source but not used as the destination, so at least one source must still be unpacked.
`obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time. [`pack`](../usage/pack.md) consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time; with `--sparse`, it also switches to the sparse form described above.
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "obikmer" name = "obikmer"
version = "1.4.6" version = "1.4.7"
edition = "2024" edition = "2024"
[[bin]] [[bin]]