OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Split index lifecycle into IndexBuilder trait for maintenance operations

Browse Source 
Extracts directory bookkeeping and construction methods into a new IndexBuilder trait, separating maintenance operations from scientific computation logic. Migrates relevant methods to the new trait, updates module imports across obikindex, and narrows the crate's public API surface. Adds an end-to-end CLI smoke test to verify multi-index merge workflows.
This commit is contained in:
Eric Coissac
2026-08-21 10:46:57 +02:00
parent da3aa5a2cb
commit 02dbdd11aa
10 changed files with 198 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DevDocMD/implementation/partition_layer_cache.md
+92 -16
View File
@@ -56,12 +56,16 @@ algorithm, extracted from `PartitionRouter` the same way `Dereplicator`
was in (4). (7) done — `LayerBuilder`, the fourth and last pipeline
algorithm; the indexing pipeline is now fully decomposed into
`obikindexer::algorithms::{partitionner, dereplicator, counter,
layer_builder}`. (8) design agreed, item 1 (`obikindexer::extensions::
IndexBuilder`) done in (9) — private extension trait, six construction-only
`KmerIndex` methods moved out; item 2 (`obikalgorithm::Algorithm`) still
not started. Note: `Layer` renamed `KmerLayer` (2026-08-21, outside this
conversation). (5) itself still not implemented, still first on the
"order of remaining work" list. Earlier mix-up, for
layer_builder}`. (8) design agreed, item 1 done in (9) —
`obikindexer::extensions::PrivateBuilder`, private, six construction-only
`KmerIndex` methods moved out. (10) done, same session — `obikindex::
IndexBuilder`, public, the four maintenance methods
(`clear_output_for_create`/`create_skeleton`/`finalize_indexed`/`state`)
shared with `merge`/`select`/`rebuild`/`reindex`. Item 2 from (8)
(`obikalgorithm::Algorithm`) still not started. Note: `Layer` renamed
`KmerLayer` (2026-08-21, outside this conversation). (5) itself still not
implemented, still first on the "order of remaining work" list. Earlier
mix-up, for
context: an earlier
version of this doc used the name `KmerPartition` (singular) for what was
actually the *collection* type (later renamed `KmerPartitions`, later
@@ -1056,14 +1060,18 @@ trait(s), of two kinds:
Both items: **design only, nothing implemented yet** — this section is
the record to resume from, not a plan already executed.
## (9) done (2026-08-21): `obikindexer::extensions::IndexBuilder` — item 1 above, implemented
## (9) done (2026-08-21): `obikindexer::extensions::PrivateBuilder` — item 1 above, implemented
Renamed from `IndexBuilder` to `PrivateBuilder` immediately after (same
session), freeing the name `IndexBuilder` for (10)'s public trait — read
`IndexBuilder` below as `PrivateBuilder` throughout this section.
Scoped down from (8)'s six-method list to the concrete set that's
genuinely movable without further ripple — checked, not assumed, before
writing anything:
```rust
pub(crate) trait IndexBuilder {
pub(crate) trait PrivateBuilder {
fn mark_scattered(&mut self) -> OKIResult<()>;
fn mark_counted(&self) -> OKIResult<()>;
fn mark_indexed(&self) -> OKIResult<()>;
@@ -1071,7 +1079,7 @@ pub(crate) trait IndexBuilder {
fn build_index_layer(&self, i: usize, min_ab: u32, max_ab: Option<u32>, with_counts: bool, mode: &IndexMode, block_bits: u8) -> Result<usize, SKError>;
fn remove_build_artifacts(&self, i: usize);
}
impl IndexBuilder for KmerIndex { ... }
impl PrivateBuilder for KmerIndex { ... }
```
All six moved bodily out of `obikindex::index::{kmer_index, index_layer}`
@@ -1105,7 +1113,7 @@ crate, not "private except to one named dependent") — the opposite of
what was wanted.
**A real design decision made while wiring callers up, not a mechanical
rename**: `IndexBuilder` being genuinely `pub(crate)` to `obikindexer`
rename**: `PrivateBuilder` being genuinely `pub(crate)` to `obikindexer`
means `obikmer::cmd::index` (a different crate) can no longer call
`mark_scattered`/`mark_counted`/`mark_indexed`/`write_spectrum` directly —
it never could have, once privacy was real rather than aspirational. Each
@@ -1141,12 +1149,80 @@ Full workspace suite green (`cargo check --workspace --all-targets` +
`cargo test --workspace`, exit code 0), plus the CLI smoke test — 870
kmers, same as (6)/(7).
Still not done: item 2 from (8) (`obikalgorithm::Algorithm`), (5), the
future cache crate, the `distance.rs` → `obikphylo` relocation (noted in
(8), explicitly deferred), and extracting
`merge`/`select`/`rebuild`/`reindex` into algorithms (which would unblock
moving `clear_output_for_create`/`create_skeleton`/`finalize_indexed`/
`state` the same way).
Still not done at the time of writing: item 2 from (8) (`obikalgorithm::
Algorithm`), (5), the future cache crate, the `distance.rs` →
`obikphylo` relocation (noted in (8), explicitly deferred), and
extracting `merge`/`select`/`rebuild`/`reindex` into algorithms.
## (10) done (2026-08-21): `obikindex::IndexBuilder` — the public counterpart, same session
Immediate correction to (9): the private trait built there was renamed
`PrivateBuilder` (freeing the name), and the four methods (9) had left
inherent on `KmerIndex` — `clear_output_for_create`/`create_skeleton`/
`finalize_indexed`/`state` — got their own trait after all: **`IndexBuilder`**,
public, defined in `obikindex` itself (not `obikindexer`):
```rust
pub trait IndexBuilder: Sized {
fn clear_output_for_create<P: AsRef<Path>>(output: P, force: bool) -> OKIResult<()>;
fn create_skeleton<P: AsRef<Path>>(output: P, meta: &IndexMeta) -> OKIResult<Self>;
fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter) -> OKIResult<Self>;
fn state(&self) -> IndexState;
}
impl IndexBuilder for KmerIndex { ... }
```
User's framing: these four are "maintenance", not "scientific computation
on an index" — a different kind of non-generic-ness than (9)'s six
(`mark_*`/`write_spectrum`/`build_index_layer`/`remove_build_artifacts`,
exclusive to the 4-stage pipeline). Maintenance is used more broadly
(`merge`/`select`/`rebuild`/`reindex`), so it gets a real, public trait —
not folded back into `KmerIndex`'s inherent surface, and not private
either.
**Where it lives, and why that's not arbitrary**: (9) needed the orphan
rule to force its trait into `obikindexer`, to achieve genuine
crate-private visibility. Here the requirement is the opposite:
`merge.rs`/`select.rs`/`rebuild.rs`/`reindex.rs` — the trait's own
heaviest users — live *inside* `obikindex`. A trait they need to reach
must be local to `obikindex` (or a crate `obikindex` itself depends on,
which doesn't exist for this). So `IndexBuilder` lives in a new
`obikindex/src/index/builder.rs`, `pub trait` (no orphan-rule tension at
all here — both trait and type are local to the same crate), re-exported
from `obikindex`'s crate root alongside `PrivateBuilder`'s sibling
`obikindexer::extensions::PrivateBuilder` staying where it is. Two
traits, two crates, two different reasons, not a contradiction.
**Blast radius, all inside `obikindex` plus one external crate**: every
internal caller of these four methods needs the trait imported now that
they're no longer inherent — `merge.rs`, `select.rs`, `rebuild.rs`,
`reindex.rs` (`use crate::index::builder::IndexBuilder;`) and, externally,
`obikmer::cmd::index::mod` (`use obikindex::IndexBuilder;`, for the three
`idx.state() < IndexState::X` resumability checks). Call syntax at every
site is unchanged (`KmerIndex::create_skeleton(...)`,
`self.state()`) — only trait-in-scope requirements are new, which is
exactly the point: same ergonomics, less surface baked into `KmerIndex`
itself.
Verification went one step further than (9): beyond
`cargo check --workspace --all-targets` + `cargo test --workspace` +
`scripts/smoke_test_index.sh` (all green, 870 kmers again), ran
`obikmer merge` end to end on two freshly built indexes (exercises
`clear_output_for_create`/`finalize_indexed` directly, the two methods
`scripts/smoke_test_index.sh` itself never touches) — exit 0, `pack`
stage completed. Test suite alone would not have caught a regression
here: no existing test builds two real indexes and merges them through
the CLI.
`KmerIndex` itself now carries only: identity/config accessors
(`root_path`/`meta`/`kmer_size`/`n_bits`/`evidence_mode`/`genomes`/...),
path resolution (`partition_dir`/`index_dir`/`layer_dir`/
`partition_meta`/`n_layers`), and a few index-maintenance operations not
yet sorted into either trait (`layer_unitigs_path`, `pack_matrices`,
`upgrade_layer_meta` — see (8)'s "tested and discarded" list; still
correctly inherent, not construction-only by the semantic criterion) —
`create`/`open`/`exists` (identity, can't be trait methods needing `Self`
before one exists) round that out.
## The problem