OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity

feat: enable index resumption and enforce directory creation

Browse Source 
The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
This commit is contained in:
Eric Coissac
2026-08-21 05:06:38 +02:00
parent abc51c2add
commit 5c1584967f
161 changed files with 5274 additions and 845 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DevDocMD/implementation/mphf.md
+3 -3
View File
@@ -6,7 +6,7 @@ Kmer indexing per partition proceeds in two phases. The separation is necessary
### Phase 1 — provisional MPHF + kmer spectrum
Implemented in `obikpartition::KmerPartition::count_kmer()` → `count_partition()`.
Implemented in `obikindex::partition::KmerPartition::count_kmer()` → `count_partition()`.
1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
@@ -101,7 +101,7 @@ type Mphf = PtrHash<
### Layer structure
Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
Each layer is a self-contained unit. See [obikindex::layer](layer_tier.md) for the full on-disk layout. The MPHF-relevant files are:
```
layer_i/
@@ -148,7 +148,7 @@ MphfLayer::build_approx_evidence(dir, b, z)
There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
In `obikpartition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
In `obikindex::partition`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
### Membership verification