feat: enable index resumption and enforce directory creation
The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
This commit is contained in:
@@ -17,7 +17,7 @@ one-way function, not an invertible bijection with a stored inverse. Any
|
||||
method that reconstructs a kmer from a bare slot number is wrong by
|
||||
construction, regardless of the mechanism used (MPHF re-hash, or evidence
|
||||
decode + direct unitig read). See `MphfLayer::kmer_at`
|
||||
(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
|
||||
(`obikindex/src/layer/mphf_layer.rs`) — flagged for removal, currently called
|
||||
from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
|
||||
"Pending work" status below).
|
||||
|
||||
@@ -70,7 +70,7 @@ partition is unknown) and must keep going through
|
||||
## Pending work — done
|
||||
|
||||
The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
|
||||
traits over `obikindex::KmerIndex`/`obilayeredmap::Layer<D>`, replacing the
|
||||
traits over `obikindex::KmerIndex`/`obikindex::layer::Layer<D>`, replacing the
|
||||
old `obikindex::siblings` module) builds and reads the annex purely in
|
||||
iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
|
||||
batch variants, mirroring `Layer<D>`'s own `KmerIter`/`KmerBatchIter`
|
||||
@@ -652,13 +652,13 @@ BRWT-style column-correlation exploitation.
|
||||
`PersistentSparseBitMatrix` went from a validated but unused type to a
|
||||
real, selectable on-disk format:
|
||||
|
||||
- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
|
||||
- **Generic `Layer<D>`**: `obikindex::layer::Layer<D>`'s presence-only methods
|
||||
(`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
|
||||
`D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
|
||||
`PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
|
||||
`LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
|
||||
already generic over any `D: LayerData`, so they needed no change.
|
||||
Verified by `obilayeredmap`'s
|
||||
Verified by `obikindex::layer`'s
|
||||
`presence_layer_generic_over_sparse_matches_dense` test: build a dense
|
||||
presence layer, convert it to sparse via `build_from_dense`, open both
|
||||
as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
|
||||
@@ -668,7 +668,7 @@ real, selectable on-disk format:
|
||||
`AtomicUsize`s in test builds, not thread-local, so a test using a
|
||||
different `k` races every other test in the same crate binary; a k=11
|
||||
version of this test passed alone but failed under the full
|
||||
`obilayeredmap` suite for exactly that reason before being fixed.)
|
||||
`obikindex::layer` suite for exactly that reason before being fixed.)
|
||||
- **`obikphylo::siblings::cache::Mat`** gained a third variant,
|
||||
`SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
|
||||
and `Presence` — every method (`find_slot`, `index_batch`,
|
||||
@@ -698,7 +698,7 @@ real, selectable on-disk format:
|
||||
dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
|
||||
sparse format round-trips through the real build pipeline
|
||||
(`PartitionCache` sparse-detection included), not just the
|
||||
`obicompactvec`/`obilayeredmap` unit layers below it.
|
||||
`obicompactvec`/`obikindex::layer` unit layers below it.
|
||||
|
||||
Full workspace `cargo test` (all crates, unit + doc tests) green after
|
||||
this change.
|
||||
@@ -746,7 +746,7 @@ Replaces the "four independent scans" problem above and implements
|
||||
(activation: either given; defaults `1.0`/`0.5` for whichever is unset).
|
||||
|
||||
Full workspace `cargo test` green after this change (167 unit tests in
|
||||
`obicompactvec`+`obilayeredmap`+`obikphylo` alone, plus every other
|
||||
`obicompactvec`+`obikindex::layer`+`obikphylo` alone, plus every other
|
||||
crate's suite, no regressions).
|
||||
|
||||
**Still open, not part of this change** (per "Correction to the 'single
|
||||
@@ -770,7 +770,7 @@ reproducible across two runs with warm disk cache), the opposite of
|
||||
query").
|
||||
|
||||
**Root cause, read from source, not measured in isolation:**
|
||||
`KmerPartition::query_partition_with` (`obikpartition/src/query_layer.rs:155-220`)
|
||||
`KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs:155-220`)
|
||||
is architecturally column-major: stage 2 walks `for g in 0..n_cols { for
|
||||
slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly)
|
||||
as the right locality strategy for the packed/columnar formats, where
|
||||
@@ -914,7 +914,7 @@ replaces.
|
||||
`PersistentCompactIntMatrix::nonzero_iter` added the same way (counts
|
||||
not excluded, per the earlier ask) — no native low-effort case, since no
|
||||
sparse count format exists, but on the same primitive, ready for one.
|
||||
- `KmerPartition::query_partition_with` (`obikpartition/src/query_layer.rs`):
|
||||
- `KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs`):
|
||||
stage 2's column-major `for g { for slot { col_value } }` replaced by one
|
||||
`layer.nonzero_iter(&slot_list)` call per layer, format-agnostic.
|
||||
- Tests: `nonzero_iter_matches_dense`, `nonzero_iter_matches_row`, and —
|
||||
|
||||
Reference in New Issue
Block a user