Extract indexing stage into LayerBuilder with NUMA-aware scheduling

Introduce a dedicated LayerBuilder struct to orchestrate parallel layer 0 construction across partitions. The fluent API supports configurable abundance thresholds and intermediate artifact retention. Parallel execution is delegated to a NUMA-aware PartitionRunner scheduler, while progress reporting and stage timing are shifted to the command layer. A new mark_indexed method cleanly separates state tracking from orchestration by generating a completion sentinel.

DevDocMD/implementation/partition_layer_cache.md

@@ -53,8 +53,11 @@ detail, session ended (budget) before implementation — see "(5) design
 agreed" below; **read it before touching `KmerPartition`/`Layer`
 signatures**, the shape is fully specified. (6) done — `Counter`, a third
 algorithm, extracted from `PartitionRouter` the same way `Dereplicator`
-was in (4); (5) itself still not implemented, still first on the "order of
-remaining work" list — see "(6) done" below. Earlier mix-up, for
+was in (4). (7) done — `LayerBuilder`, the fourth and last pipeline
+algorithm; the indexing pipeline is now fully decomposed into
+`obikindexer::algorithms::{partitionner, dereplicator, counter,
+layer_builder}`. (5) itself still not implemented, still first on the
+"order of remaining work" list — see "(7) done" below. Earlier mix-up, for
 context: an earlier
 version of this doc used the name `KmerPartition` (singular) for what was
 actually the *collection* type (later renamed `KmerPartitions`, later
@@ -851,6 +854,104 @@ Still not done: (5) (`Layer`/`KmerPartition` redesign, `PartitionRouter`'s
 `&mut`→`&`), the future cache crate, `build_layers` (still a `KmerIndex`
 inherent method, not an algorithm), and `obikalgorithm` itself.
 
+## (7) done (2026-08-21): `LayerBuilder` — fourth and last pipeline algorithm
+
+Closes out the indexing pipeline: `build_layers`/`build_index_layer`
+(the last stage still living as `KmerIndex` inherent methods, flagged as
+inconsistent since (6)) extracted into `obikindexer::algorithms::
+layer_builder::LayerBuilder`, same two-phase shape as the other three.
+
+```rust
+pub struct LayerBuilder<'a> {
+    index: &'a KmerIndex,
+    n_partitions: usize,
+    min_abundance: u32,
+    max_abundance: Option<u32>,
+    keep_intermediate: bool,
+}
+
+impl<'a> LayerBuilder<'a> {
+    pub fn new(index: &'a KmerIndex) -> Self;
+    pub fn min_abundance(mut self, v: u32) -> Self;
+    pub fn max_abundance(mut self, v: Option<u32>) -> Self;
+    pub fn keep_intermediate(mut self, v: bool) -> Self;
+    pub fn run(&self, on_progress: Option<impl FnMut(Progress) + Send>) -> SKResult<usize>;  // returns total kmers built
+}
+```
+
+**Different from all three prior extractions in one respect, deliberately
+not "fixed" to match them**: the actual per-partition construction logic
+(De Bruijn graph from dereplicated superkmers + provisional counts →
+unitigs → MPHF → matrix) stayed put as `KmerIndex::build_index_layer`/
+`remove_build_artifacts` (both already `pub`) — not moved into
+`obikindexer`. Checked first: unlike `dereplicate_partition`/
+`count_partition` (which only ever had one caller), `build_index_layer`
+depends on several `obikindex`-internal helpers (`graph_pipeline::
+{write_graph_as_unitigs, materialize_layer}`, `common::olm_to_sk`) that
+are `pub(crate)` and shared with `merge`/`select`/`rebuild`'s own
+layer-construction paths — moving `build_index_layer` out would have
+meant either exporting that internal surface just for this one algorithm
+or duplicating it. Neither was needed: `build_index_layer`/
+`remove_build_artifacts` were *already* public `KmerIndex` methods, so
+`LayerBuilder`'s job is purely the orchestration around them (scheduling,
+config, progress) — the exact same "algorithm calls already-public
+`KmerIndex` primitives" shape `PartitionRouter`/`Dereplicator`/`Counter`
+already have, just at a coarser grain for this one stage. This is the
+"is the producer's API actually deficient?" check from
+[[feedback_no_spaghetti_petits_pois]] applied and answered "no" — not
+skipped.
+
+**Two more real divergences, both forced by `PartitionRunner`, not
+arbitrary:**
+- Uses `obikindex::PartitionRunner` (NUMA-aware scheduler, already
+  `pub use`d from `obikindex`) instead of plain `rayon::into_par_iter`
+  like `Dereplicator`/`Counter` — matches what `build_layers` already used
+  before extraction; this stage is more CPU/memory-intensive per partition
+  (graph construction) than scatter/dereplicate/count.
+- Callback bound is `FnMut(Progress) + Send` — a third variant, not
+  matching either prior shape. `PartitionRunner::run`'s `on_done` is
+  invoked from its own single controller thread (never concurrently, so
+  no `Sync` needed, unlike `Dereplicator`/`Counter`'s `Fn + Sync`), but
+  that controller thread is itself `std::thread::scope`-spawned, so the
+  closure still has to be `Send` to cross into it — caught immediately by
+  the compiler (`cannot be sent between threads safely`) when `Send` was
+  first omitted, not a design guess. `obikalgorithm`'s eventual shared
+  trait now has three real callback-bound data points to reconcile
+  (`FnMut` alone for `PartitionRouter::run`'s sequential loop, `FnMut +
+  Send` here, `Fn + Sync` for `Dereplicator`/`Counter`'s `rayon`
+  `par_iter`), not two.
+
+`KmerIndex::build_layers` deleted outright (`KmerIndex` stays a pure data
+structure — no compute orchestration methods, consistent with `dereplicate`/
+`count_kmer`'s removal in (4)/(6)). New `KmerIndex::mark_indexed()` added,
+symmetric to `mark_scattered`/`mark_counted`, replacing the inline
+`touch(SENTINEL_INDEXED)` that used to live inside `build_layers`.
+`Stage::start("index")`/`rep.push(...)` and the `progress_bar`/
+`"{n} total kmers indexed"` log line both moved to `cmd/index/mod.rs`,
+same pattern as (3)/(4)/(6) — `LayerBuilder` renders nothing itself, just
+reports `Progress`.
+
+All callers updated: `cmd/index/mod.rs` (Stage 3), `obikphylo`'s test
+harness (also gained a `mark_indexed()` call it was missing — harmless
+before since nothing checked `IndexState::Indexed` in that test, but now
+correct).
+
+Full workspace suite green (`cargo check --workspace --all-targets` +
+`cargo test --workspace`, exit code 0), plus the end-to-end CLI smoke test
+(`scripts/smoke_test_index.sh`, built earlier specifically so this
+verification step is a one-liner from now on) — 870 kmers indexed, query
+round-trip confirmed, same numbers as (6).
+
+**The indexing pipeline is now fully decomposed**: `obikindexer::
+algorithms::{partitionner, dereplicator, counter, layer_builder}`, each a
+`new`/(setters)/`run` algorithm operating on a `&KmerIndex` (or `&mut` for
+`PartitionRouter`, not yet fixed — see (5)), `KmerIndex` itself holding no
+pipeline-orchestration logic anymore. Still not done: (5), the future
+cache crate, `obikalgorithm` (now unblocked — three real callback-bound
+variants observed, worth revisiting whether a single trait can express
+all three or whether that's itself the answer: it can't, and the trait
+should not force it).
+
 ## The problem
 
 Reading a layer's data (MPHF + matrix) is not free: `MphfLayer::open` mmaps