feat: enable index resumption and enforce directory creation
The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
This commit is contained in:
@@ -3,9 +3,9 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
|
||||
- `obilayeredmap/src/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
|
||||
- `obilayeredmap/src/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
|
||||
- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
|
||||
- `obikindex/src/layer/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
|
||||
- `obikindex/src/layer/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
|
||||
- `obikindex/src/reindex.rs` — KmerIndex::reindex(), conversion exact↔approx en place
|
||||
- `obikmer/src/cmd/reindex.rs` — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size
|
||||
- `obikmer/src/cmd/index.rs` — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp
|
||||
|
||||
@@ -303,7 +303,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
|
||||
|
||||
## Implementation
|
||||
|
||||
- **`obikpartition::filter::GroupQuorumFilter`** — implements `KmerFilter`
|
||||
- **`obikindex::partition::filter::GroupQuorumFilter`** — implements `KmerFilter`
|
||||
using pre-computed ingroup and outgroup index vectors. The heavy logic
|
||||
(predicate parsing, three-value evaluation, genome classification) happens
|
||||
once before any iteration; each k-mer row evaluation is a simple index
|
||||
@@ -314,7 +314,7 @@ This parameter has no effect on presence/absence indexes (where values are alrea
|
||||
`UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
|
||||
list.
|
||||
|
||||
- **`obikpartition::KmerPartition::iter_partition_kmers`** — accepts
|
||||
- **`obikindex::partition::KmerPartition::iter_partition_kmers`** — accepts
|
||||
`filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
|
||||
the callback. `filter`, `dump`, and `unitig` all go through this single
|
||||
entry point.
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# obilayeredmap — layered kmer index crate
|
||||
# obikindex::layer — the Layer tier
|
||||
|
||||
## Purpose
|
||||
|
||||
`obilayeredmap` implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
|
||||
`obikindex::layer` (the `layer/` submodule of the `obikindex` crate — a standalone `obilayeredmap` crate until 2026-08-21, folded back in alongside `obikpartition` as part of a broader `Index { Partition { Layer } }` submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
|
||||
|
||||
---
|
||||
|
||||
+7
-7
@@ -1,14 +1,14 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/obilayeredmap.md
|
||||
# Coverage: implementation/layer_tier.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
|
||||
- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
|
||||
- `obilayeredmap/src/map.rs` — LayeredMap<D>, push_layer(), query()
|
||||
- `obilayeredmap/src/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
|
||||
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
|
||||
- `obilayeredmap/src/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
|
||||
- `obikindex/src/layer/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
|
||||
- `obikindex/src/layer/typed_layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
|
||||
- `obikindex/src/layer/map.rs` — LayeredMap<D>, push_layer(), query()
|
||||
- `obikindex/src/layer/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
|
||||
- `obikindex/src/layer/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
|
||||
- `obikindex/src/layer/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -4,9 +4,9 @@
|
||||
## Code couvert
|
||||
|
||||
- `obikindex/src/merge.rs` — `KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
|
||||
- `obikpartition/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
|
||||
- `obikpartition/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
|
||||
- `obilayeredmap/src/layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
|
||||
- `obikindex/src/partition/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
|
||||
- `obikindex/src/partition/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
|
||||
- `obikindex/src/layer/typed_layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
|
||||
- `obicompactvec/src/intmatrix.rs` — `append_column` pour PersistentCompactIntMatrix
|
||||
- `obicompactvec/src/bitmatrix.rs` — `append_column` pour PersistentBitMatrix
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@ Kmer indexing per partition proceeds in two phases. The separation is necessary
|
||||
|
||||
### Phase 1 — provisional MPHF + kmer spectrum
|
||||
|
||||
Implemented in `obikpartition::KmerPartition::count_kmer()` → `count_partition()`.
|
||||
Implemented in `obikindex::partition::KmerPartition::count_kmer()` → `count_partition()`.
|
||||
|
||||
1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
|
||||
2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
|
||||
@@ -101,7 +101,7 @@ type Mphf = PtrHash<
|
||||
|
||||
### Layer structure
|
||||
|
||||
Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
|
||||
Each layer is a self-contained unit. See [obikindex::layer](layer_tier.md) for the full on-disk layout. The MPHF-relevant files are:
|
||||
|
||||
```
|
||||
layer_i/
|
||||
@@ -148,7 +148,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
|
||||
|
||||
There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
|
||||
|
||||
In `obikpartition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
|
||||
In `obikindex::partition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
|
||||
|
||||
### Membership verification
|
||||
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
|
||||
- `obikpartition/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
|
||||
- `obikindex/src/layer/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
|
||||
- `obikindex/src/partition/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -9,5 +9,5 @@
|
||||
## Notes
|
||||
|
||||
Document stable (librairie générique, peu de risque de dérive).
|
||||
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartition`
|
||||
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikindex::partition`
|
||||
ou s'il a été remplacé par Rayon dans certains chemins.
|
||||
|
||||
@@ -1,5 +1,21 @@
|
||||
# Partition and layer caching (discussion)
|
||||
|
||||
**Superseded (2026-08-21):** `obikpartition` and `obilayeredmap` are no
|
||||
longer separate workspace crates — both were folded back into `obikindex`
|
||||
as submodules (`obikindex::partition`, `obikindex::layer`), alongside the
|
||||
crate's original content as `obikindex::index`, purely to reduce the
|
||||
crate count (no behavior change). Every mention of `obikpartition`/
|
||||
`obilayeredmap` as a *crate* below, and every dependency-direction
|
||||
argument phrased in terms of "which crate depends on which" (e.g. "this
|
||||
crate depends only on `obilayeredmap` and below, never on `obikindex`"),
|
||||
describes that now-superseded split-crate architecture and is kept as-is
|
||||
for historical context — read `obikpartition::X` as `obikindex::
|
||||
partition::X` and `obilayeredmap::X` as `obikindex::layer::X` throughout.
|
||||
The underlying module boundary and its rationale (Layer tier / Partition
|
||||
tier / Index tier, each depending only downward) are unchanged; only the
|
||||
crate-vs-module packaging changed. See [obikindex::layer](layer_tier.md)
|
||||
for the current module doc.
|
||||
|
||||
Status (2026-08-20, latest pass): (1) done — `obilayeredmap::Layer`
|
||||
exists, `Mat` is gone. (1b) done — `Layer::Empty`, the first non-ready
|
||||
state, added (panics on every read method). (2a) done — the
|
||||
|
||||
@@ -203,7 +203,7 @@ part_XXXXX/
|
||||
|
||||
**Cleanup:** unless `--keep-intermediate` is set, `remove_build_artifacts` deletes `dereplicated.skmer.zst`, `mphf1.bin`, and `counts1.bin` after all partitions are indexed.
|
||||
|
||||
See [obilayeredmap](obilayeredmap.md) and [MPHF selection](mphf.md) for data structure details.
|
||||
See [obikindex::layer](layer_tier.md) and [MPHF selection](mphf.md) for data structure details.
|
||||
|
||||
**Query path (exact evidence):**
|
||||
|
||||
|
||||
@@ -3,12 +3,12 @@
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikpartition/src/partition.rs` — estimation des paramètres (phase 0)
|
||||
- `obikindex/src/partition/partition.rs` — estimation des paramètres (phase 0)
|
||||
- `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
|
||||
- `obikpartition/src/filter.rs` — déduplication bucket-sort (phase 2)
|
||||
- `obikpartition/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
|
||||
- `obikindex/src/partition/filter.rs` — déduplication bucket-sort (phase 2)
|
||||
- `obikindex/src/partition/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
|
||||
- `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
|
||||
- `obikpartition/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
|
||||
- `obikindex/src/partition/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
|
||||
- `obikindex/src/index.rs` — `build_layers()`, `dereplicate_and_count()`
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
|
||||
- `obikindex/src/meta.rs` — IndexMeta, IndexConfig (version, config, genomes)
|
||||
- `obikindex/src/index.rs` — layout sur disque : partitions/, index.meta
|
||||
- `obilayeredmap/src/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
|
||||
- `obikindex/src/layer/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
|
||||
- `obiskio/src/unitig_index.rs` — fichiers unitigs.bin + unitigs.bin.idx
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
## Code couvert
|
||||
|
||||
- `obiskio/src/unitig_index.rs` — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()
|
||||
- `obilayeredmap/src/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
|
||||
- `obikindex/src/layer/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
|
||||
- `obidebruinj/src/debruijn.rs` — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK
|
||||
|
||||
## Notes
|
||||
|
||||
Reference in New Issue
Block a user