feat: enable index resumption and enforce directory creation

The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
This commit is contained in:
Eric Coissac
2026-08-21 05:06:38 +02:00
parent abc51c2add
commit 5c1584967f
161 changed files with 5274 additions and 845 deletions
@@ -3,9 +3,9 @@
## Code couvert
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
- `obilayeredmap/src/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
- `obilayeredmap/src/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
- `obikindex/src/layer/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
- `obikindex/src/layer/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
- `obikindex/src/reindex.rs` — KmerIndex::reindex(), conversion exact↔approx en place
- `obikmer/src/cmd/reindex.rs` — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size
- `obikmer/src/cmd/index.rs` — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp
+2 -2
View File
@@ -303,7 +303,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
## Implementation
- **`obikpartition::filter::GroupQuorumFilter`** — implements `KmerFilter`
- **`obikindex::partition::filter::GroupQuorumFilter`** — implements `KmerFilter`
using pre-computed ingroup and outgroup index vectors. The heavy logic
(predicate parsing, three-value evaluation, genome classification) happens
once before any iteration; each k-mer row evaluation is a simple index
@@ -314,7 +314,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
`UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
list.
- **`obikpartition::KmerPartition::iter_partition_kmers`** — accepts
- **`obikindex::partition::KmerPartition::iter_partition_kmers`** — accepts
`filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
the callback. `filter`, `dump`, and `unitig` all go through this single
entry point.
@@ -1,8 +1,8 @@
# obilayeredmap — layered kmer index crate
# obikindex::layer — the Layer tier
## Purpose
`obilayeredmap` implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
`obikindex::layer` (the `layer/` submodule of the `obikindex` crate — a standalone `obilayeredmap` crate until 2026-08-21, folded back in alongside `obikpartition` as part of a broader `Index { Partition { Layer } }` submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
---
@@ -1,14 +1,14 @@
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
# Coverage: implementation/obilayeredmap.md
# Coverage: implementation/layer_tier.md
## Code couvert
- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
- `obilayeredmap/src/map.rs` — LayeredMap<D>, push_layer(), query()
- `obilayeredmap/src/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
- `obilayeredmap/src/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
- `obikindex/src/layer/map.rs` — LayeredMap<D>, push_layer(), query()
- `obikindex/src/layer/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
- `obikindex/src/layer/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
## Notes
+3 -3
View File
@@ -4,9 +4,9 @@
## Code couvert
- `obikindex/src/merge.rs``KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
- `obikpartition/src/merge_layer.rs``merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
- `obikpartition/src/rebuild_layer.rs``rebuild_partition()`, paramètre `block_bits`
- `obilayeredmap/src/layer.rs``Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
- `obikindex/src/partition/merge_layer.rs``merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
- `obikindex/src/partition/rebuild_layer.rs``rebuild_partition()`, paramètre `block_bits`
- `obikindex/src/layer/typed_layer.rs``Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
- `obicompactvec/src/intmatrix.rs``append_column` pour PersistentCompactIntMatrix
- `obicompactvec/src/bitmatrix.rs``append_column` pour PersistentBitMatrix
+3 -3
View File
@@ -6,7 +6,7 @@ Kmer indexing per partition proceeds in two phases. The separation is necessary
### Phase 1 — provisional MPHF + kmer spectrum
Implemented in `obikpartition::KmerPartition::count_kmer()``count_partition()`.
Implemented in `obikindex::partition::KmerPartition::count_kmer()``count_partition()`.
1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
@@ -101,7 +101,7 @@ type Mphf = PtrHash<
### Layer structure
Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
Each layer is a self-contained unit. See [obikindex::layer](layer_tier.md) for the full on-disk layout. The MPHF-relevant files are:
```
layer_i/
@@ -148,7 +148,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
In `obikpartition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
In `obikindex::partition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
### Membership verification
+2 -2
View File
@@ -3,8 +3,8 @@
## Code couvert
- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
- `obikpartition/src/index_layer.rs``build_index_layer()` avec passage de `block_bits`
- `obikindex/src/layer/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
- `obikindex/src/partition/index_layer.rs``build_index_layer()` avec passage de `block_bits`
## Notes
+1 -1
View File
@@ -9,5 +9,5 @@
## Notes
Document stable (librairie générique, peu de risque de dérive).
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartition`
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikindex::partition`
ou s'il a été remplacé par Rayon dans certains chemins.
@@ -1,5 +1,21 @@
# Partition and layer caching (discussion)
**Superseded (2026-08-21):** `obikpartition` and `obilayeredmap` are no
longer separate workspace crates — both were folded back into `obikindex`
as submodules (`obikindex::partition`, `obikindex::layer`), alongside the
crate's original content as `obikindex::index`, purely to reduce the
crate count (no behavior change). Every mention of `obikpartition`/
`obilayeredmap` as a *crate* below, and every dependency-direction
argument phrased in terms of "which crate depends on which" (e.g. "this
crate depends only on `obilayeredmap` and below, never on `obikindex`"),
describes that now-superseded split-crate architecture and is kept as-is
for historical context — read `obikpartition::X` as `obikindex::
partition::X` and `obilayeredmap::X` as `obikindex::layer::X` throughout.
The underlying module boundary and its rationale (Layer tier / Partition
tier / Index tier, each depending only downward) are unchanged; only the
crate-vs-module packaging changed. See [obikindex::layer](layer_tier.md)
for the current module doc.
Status (2026-08-20, latest pass): (1) done — `obilayeredmap::Layer`
exists, `Mat` is gone. (1b) done — `Layer::Empty`, the first non-ready
state, added (panics on every read method). (2a) done — the
+1 -1
View File
@@ -203,7 +203,7 @@ part_XXXXX/
**Cleanup:** unless `--keep-intermediate` is set, `remove_build_artifacts` deletes `dereplicated.skmer.zst`, `mphf1.bin`, and `counts1.bin` after all partitions are indexed.
See [obilayeredmap](obilayeredmap.md) and [MPHF selection](mphf.md) for data structure details.
See [obikindex::layer](layer_tier.md) and [MPHF selection](mphf.md) for data structure details.
**Query path (exact evidence):**
+4 -4
View File
@@ -3,12 +3,12 @@
## Code couvert
- `obikpartition/src/partition.rs` — estimation des paramètres (phase 0)
- `obikindex/src/partition/partition.rs` — estimation des paramètres (phase 0)
- `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
- `obikpartition/src/filter.rs` — déduplication bucket-sort (phase 2)
- `obikpartition/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
- `obikindex/src/partition/filter.rs` — déduplication bucket-sort (phase 2)
- `obikindex/src/partition/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
- `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
- `obikpartition/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
- `obikindex/src/partition/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
- `obikindex/src/index.rs``build_layers()`, `dereplicate_and_count()`
## Notes
+1 -1
View File
@@ -5,7 +5,7 @@
- `obikindex/src/meta.rs` — IndexMeta, IndexConfig (version, config, genomes)
- `obikindex/src/index.rs` — layout sur disque : partitions/, index.meta
- `obilayeredmap/src/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
- `obikindex/src/layer/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
- `obiskio/src/unitig_index.rs` — fichiers unitigs.bin + unitigs.bin.idx
## Notes
@@ -4,7 +4,7 @@
## Code couvert
- `obiskio/src/unitig_index.rs` — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()
- `obilayeredmap/src/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
- `obikindex/src/layer/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
- `obidebruinj/src/debruijn.rs` — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK
## Notes