feat: enable index resumption and enforce directory creation

The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
2026-08-21 05:06:38 +02:00
parent abc51c2add
commit 5c1584967f
161 changed files with 5274 additions and 845 deletions
@@ -0,0 +1,449 @@
+# obikindex::layer — the Layer tier
+
+## Purpose
+
+`obikindex::layer` (the `layer/` submodule of the `obikindex` crate — a standalone `obilayeredmap` crate until 2026-08-21, folded back in alongside `obikpartition` as part of a broader `Index { Partition { Layer } }` submodule regrouping) implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
+
+---
+
+## Three usage modes
+
+The MPHF + evidence infrastructure is the same for all modes. The **payload** varies.
+
+| Mode | Description | Payload type | Storage |
+|---|---|---|---|
+| 1. Set | membership test only | `()` | — |
+| 2. Count | occurrences per kmer per sample | `PersistentCompactIntMatrix` | `counts/` directory |
+| 3. Presence/absence | which genomes contain each kmer | `PersistentBitMatrix` | `presence/` directory |
+
+Both `PersistentCompactIntMatrix` and `PersistentBitMatrix` come from the `obicompactvec` crate.
+
+---
+
+## Index mode (homogeneity invariant)
+
+A partitioned index is homogeneous: every layer within a partition shares the same mode. The mode is determined once at `LayeredMap::open()` from `PartitionMeta.mode` and passed to each `Layer::open()` — no per-layer file is read.
+
+```rust
+#[derive(Serialize, Deserialize, Default)]
+#[serde(tag = "type", rename_all = "snake_case")]
+pub enum IndexMode {
+    #[default]
+    Exact,
+    Approx { b: u8, z: u8 },
+    Hybrid { b: u8, z: u8 },
+}
+```
+
+`IndexMode` is stored once in `PartitionMeta` (`meta.json` at partition root). There is no `layer_meta.json`.
+
+- **Exact**: writes `evidence.bin` + `unitigs.bin.idx`. Zero false positives.
+- **Approx**: writes `fingerprint.bin` only. FP rate per kmer = 1/2^b; with Findere z-parameter, z consecutive kmers must all match → effective window FP ≈ 1/2^(b·z). No `.idx` written or required.
+- **Hybrid**: writes both `fingerprint.bin` and `evidence.bin` + `.idx`. `find()` uses the fingerprint (fast, O(1)); `find_strict()` uses exact evidence.
+
+---
+
+## MphfLayer — autonomous kmer → slot mapping
+
+`MphfLayer` encapsulates the MPHF and evidence store for one layer. It is independent of any payload.
+
+```rust
+pub struct MphfLayer {
+    mphf: Mphf,
+    ev:   LayerEvidence,   // loaded at open() time
+    n:    usize,
+}
+```
+
+`LayerEvidence` is an internal enum, not public:
+
+```rust
+enum LayerEvidence {
+    Exact  { evidence: Evidence, unitigs: UnitigFileReader },
+    Approx { fingerprint: FingerprintVec, unitigs_path: PathBuf },
+    Hybrid { evidence: Evidence, unitigs: UnitigFileReader, fingerprint: FingerprintVec },
+}
+```
+
+`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read.
+
+### Query API
+
+Two public query methods, both returning `Option<usize>` (slot index):
+
+```rust
+pub fn find(&self, kmer: CanonicalKmer) -> Option<usize>
+pub fn find_strict(&self, kmer: CanonicalKmer) -> Option<usize>
+```
+
+- `find`: O(1) auto-dispatch. Exact/Hybrid → exact evidence check. Approx/Hybrid → fingerprint comparison.
+- `find_strict`: always exact. Exact/Hybrid → O(1) evidence check. Approx → O(n) sequential scan (no `.idx`).
+
+There are no `find_exact`/`find_approx` methods; panicking dispatch is eliminated.
+
+### Build surface
+
+```rust
+// Full MPHF + evidence build (two-pass)
+pub(crate) fn build(dir, block_bits, mode: &IndexMode, fill_slot) -> OLMResult<usize>
+
+// Evidence-only post-hoc builds (MPHF already present)
+pub fn build_exact_evidence(dir, block_bits) -> OLMResult<usize>
+pub fn build_approx_evidence(dir, b, z)      -> OLMResult<usize>
+```
+
+`MphfLayer::build` runs two passes over `unitigs.bin`:
+
+1. **Pass 1** (parallel via rayon): a `CanonicalKmerIter` (clonable, `Arc<Mmap>`, no file reopening) is passed to `new_from_par_iter` via `par_bridge()`. Produces `mphf.bin`. No `.idx` is read or created at this stage.
+2. **Pass 2** (sequential): fill evidence files; call `fill_slot(slot, kmer)` per kmer. `.idx` is written last for Exact/Hybrid modes (query-time only).
+
+There is no `build_evidence` dispatch wrapper — callers invoke `build_exact_evidence` or `build_approx_evidence` directly.
+
+For empty layers (n = 0), all build variants return `Ok(0)` immediately after creating empty output files.
+
+---
+
+## Layer\<D: LayerData\> — MPHF + payload
+
+`Layer<D>` pairs an `MphfLayer` with one payload store.
+
+```rust
+pub trait LayerData: Sized {
+    type Item;
+    fn open(layer_dir: &Path) -> OLMResult<Self>;
+    fn read(&self, slot: usize) -> Self::Item;
+}
+
+pub struct Layer<D: LayerData = ()> {
+    mphf: MphfLayer,
+    data: D,
+}
+
+pub struct Hit<T = ()> {
+    pub slot: usize,
+    pub data: T,
+}
+```
+
+`LayerData` covers the **read path only** (`open` + `read`). Build signatures differ between modes and are not part of the trait.
+
+| Type | `Item` | Description |
+|---|---|---|
+| `()` | `()` | mode 1 — membership only |
+| `PersistentCompactIntMatrix` | `Box<[u32]>` | mode 2 — count matrix (one u32 per column per slot) |
+| `PersistentBitMatrix` | `Box<[bool]>` | mode 3 — presence matrix (one bit per genome per slot) |
+
+### Build signatures
+
+```rust
+// mode 1
+impl Layer<()> {
+    pub fn build(out_dir: &Path, block_bits: u8, mode: &IndexMode) -> OLMResult<usize>
+}
+
+// mode 2
+impl Layer<PersistentCompactIntMatrix> {
+    pub fn build(out_dir: &Path, block_bits: u8, mode: &IndexMode,
+                 count_of: impl Fn(CanonicalKmer) -> u32) -> OLMResult<usize>
+    pub fn build_from_map(out_dir: &Path, block_bits: u8, mode: &IndexMode,
+                          counts: &HashMap<CanonicalKmer, u32>) -> OLMResult<usize>
+}
+
+// mode 3
+impl Layer<PersistentBitMatrix> {
+    pub fn build_presence(out_dir: &Path, block_bits: u8, mode: &IndexMode,
+                          n_genomes: usize,
+                          present_in: impl Fn(CanonicalKmer, usize) -> bool) -> OLMResult<usize>
+}
+```
+
+All build impls delegate to `MphfLayer::build` via a mode-specific `fill_slot` callback. The `mode` parameter is forwarded directly — no `LayerMeta` is written.
+
+Evidence-only post-hoc builds are accessible directly on `Layer<D>`:
+
+```rust
+impl<D: LayerData> Layer<D> {
+    pub fn build_exact_evidence(layer_dir: &Path, block_bits: u8) -> OLMResult<usize>
+    pub fn build_approx_evidence(layer_dir: &Path, b: u8, z: u8)  -> OLMResult<usize>
+}
+```
+
+There is no `build_evidence` dispatch wrapper.
+
+---
+
+## FingerprintVec and FingerprintVecWriter
+
+Approximate evidence is stored as a packed b-bit array, one fingerprint per MPHF slot.
+
+```
+fingerprint.bin format:
+  magic:   b"FPVF"  (4 bytes)
+  b:       u8       (bits per fingerprint, 1..=64)
+  padding: [0u8; 3]
+  n:       u64 LE   (number of slots)
+  data:    packed bits, ceil(n*b/8) bytes, Lsb0 order
+```
+
+```rust
+impl FingerprintVec {
+    pub fn open(path: &Path) -> OLMResult<Self>
+    pub fn get(&self, slot: usize) -> u64
+    pub fn matches(&self, slot: usize, fingerprint: u64) -> bool
+    pub fn n(&self) -> usize
+    pub fn b(&self) -> u8
+}
+```
+
+`matches(slot, hash)` extracts the b-bit fingerprint stored at `slot` and compares it to the low b bits of `hash`. It is the core operation of `find_approx`.
+
+---
+
+## LayeredMap\<D\> — collection of layers
+
+`LayeredMap<D>` wraps `Vec<Layer<D>>` for a single partition directory.
+
+```rust
+pub struct LayeredMap<D: LayerData = ()> {
+    root:   PathBuf,
+    meta:   PartitionMeta,
+    layers: Vec<Layer<D>>,
+}
+```
+
+`PartitionMeta` (`meta.json` at the partition root) stores `n_layers`.
+
+### Common methods
+
+```rust
+pub fn open(root: &Path)              -> OLMResult<Self>
+pub fn create(root: &Path, mode: IndexMode) -> OLMResult<Self>
+pub fn n_layers(&self)                -> usize
+pub fn layer(&self, i: usize)         -> &Layer<D>
+pub fn mode(&self)                    -> &IndexMode
+pub fn query(&self, kmer: CanonicalKmer) -> Option<(usize, Hit<D::Item>)>
+pub fn next_layer_writer(&self)       -> OLMResult<UnitigFileWriter>
+```
+
+`open` reads `PartitionMeta` once, extracts `mode`, and passes it to every `Layer::open` — no per-layer file is read. `create` stores the given mode in `PartitionMeta`.
+
+`query` probes layers in order and returns `(layer_index, Hit)` on the first match. Expected probe depth: 1 for kmers in layer 0.
+
+### push_layer
+
+`push_layer` builds the next layer from a `unitigs.bin` already written via `next_layer_writer`, using `DEFAULT_BLOCK_BITS`:
+
+```rust
+// mode 1
+impl LayeredMap<()> {
+    pub fn push_layer(&mut self) -> OLMResult<usize>
+}
+
+// mode 2
+impl LayeredMap<PersistentCompactIntMatrix> {
+    pub fn push_layer(&mut self, count_of: impl Fn(CanonicalKmer) -> u32) -> OLMResult<usize>
+    pub fn push_layer_from_map(&mut self, counts: &HashMap<CanonicalKmer, u32>) -> OLMResult<usize>
+}
+```
+
+Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers build directly via `Layer<PersistentBitMatrix>::build_presence`.
+
+---
+
+## Layer\<D\> — raw mapping, iteration, and batch access
+
+Beyond `query`/`find` (membership-checked), `Layer<D>` exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.
+
+### Raw kmer → slot mapping
+
+```rust
+pub fn index(&self, kmer: CanonicalKmer) -> usize
+pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize>
+```
+
+Pure MPHF mapping, no evidence/fingerprint check — equivalent to `MphfOnly::index`. Only meaningful when the caller already knows `kmer` belongs to the layer; on an absent kmer the MPHF still returns *some* slot (undefined, not `None`).
+
+### Kmer iteration
+
+Four iterators, all built from `unitigs.bin` (physical layout order, **not** correlated with MPHF slot numbers):
+
+```rust
+pub fn iter_kmers(&self) -> KmerIter<'_>
+pub fn enumerate_kmers(&self) -> Enumerate<KmerIter<'_>>              // (order_index, kmer)
+pub fn iter_kmers_batch(&self, n: usize) -> KmerBatchIter<'_>          // Vec<CanonicalKmer> of size ≤ n
+pub fn enumerate_kmers_batch(&self, n: usize) -> impl Iterator<Item = (usize, Vec<CanonicalKmer>)> + Send + 'static
+```
+
+`KmerIter`/`KmerBatchIter` own a clone of the underlying `Arc<UnitigFileReader>` rather than borrowing `self` — `Send + 'static`, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. `enumerate_kmers_batch`'s index is the batch's starting offset in iteration order (a multiple of `n` except for the final, possibly shorter, batch).
+
+### Batch lookup on payload vectors/views
+
+`PersistentCompactIntVec`, `PersistentBitVec`, `IntSliceView`, `BitSliceView` all expose:
+
+```rust
+fn get_batch(&self, slots: &[usize]) -> Vec<T>
+fn fill_batch(&self, slots: &[usize], out: &mut [T])
+```
+
+Both sort `slots` internally for sequential mmap access, then reorder results back to the caller's original order. `fill_batch` fills a caller-provided buffer, avoiding the `Vec` allocation.
+
+### sub_matrix / fill_sub_matrix
+
+```rust
+// Layer<PersistentCompactIntMatrix>
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<u32>>            // column-first
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<u32>])
+
+// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)
+pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
+pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
+```
+
+Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
+
+---
+
+## LayeredStore\<S\> and aggregation traits
+
+`LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls:
+
+```rust
+pub struct LayeredStore<S>(pub Vec<S>);
+
+impl<S: ColumnWeights> ColumnWeights for LayeredStore<S> { … }  // Σ col_weights across inner stores
+impl<S: CountPartials> CountPartials  for LayeredStore<S> { … }  // element-wise Σ partials
+impl<S: BitPartials>   BitPartials    for LayeredStore<S> { … }  // element-wise Σ partials
+```
+
+Because blanket impls compose, `LayeredStore<LayeredStore<S>>` automatically inherits all three traits when `S` does — providing the partitioned level without a separate type.
+
+**Leaf implementors** (in `obicompactvec`):
+
+| Type | Traits |
+|---|---|
+| `PersistentCompactIntMatrix` | `ColumnWeights` (via `sum()`) + `CountPartials` |
+| `PersistentBitMatrix` | `ColumnWeights` (via `count_ones()`) + `BitPartials` |
+
+See [Kmer index architecture](../architecture/index_architecture.md) for the full trait API and the two-pass normalised-metric pattern.
+
+---
+
+## On-disk structure
+
+```
+partition_root/                    ← LayeredMap (one partition)
+  meta.json                        — {"n_layers": N, "mode": {"type": "exact"|"approx"|"hybrid", ...}}
+  layer_0/                         ← Layer
+    mphf.bin                       — ptr_hash MPHF (epserde format)
+    unitigs.bin                    — packed 2-bit nucleotide sequences
+    unitigs.bin.idx                — UIDX index (Exact/Hybrid only; query-time, never built during MPHF construction)
+    evidence.bin                   — [u32; n], LE  (Exact/Hybrid only)
+    fingerprint.bin                — packed b-bit array  (Approx/Hybrid only)
+    counts/                        [mode 2] PersistentCompactIntMatrix
+      meta.json
+      col_000000.pciv
+    presence/                      [mode 3] PersistentBitMatrix
+      meta.json
+      col_000000.pbiv  …
+  layer_1/
+    …
+```
+
+There is no `layer_meta.json`. The mode is stored once in `PartitionMeta` and is valid for all layers. `unitigs.bin.idx` is built at the end of `build_exact_evidence` — never during MPHF construction — and is consumed at query time only.
+
+---
+
+## Evidence encoding (exact)
+
+`evidence.bin` is a flat `[u32; n]` array with no header. Each u32 encodes one slot:
+
+```
+bits [31:7] = chunk_id (25 bits) — index of the unitig chunk
+bits [6:0]  = rank     (7 bits)  — kmer index within the chunk (0-based)
+```
+
+`chunk_id = raw >> 7`, `rank = raw & 0x7F`. Reconstructing the kmer: read k nucleotides at position `rank` within unitig `chunk_id` (requires `unitigs.bin.idx` for random access).
+
+For k=31, m=11, the observed maximum is ~46 kmers per chunk — well within the 127-kmer u7 capacity.
+
+---
+
+## ptr_hash configuration
+
+```rust
+type Mphf = PtrHash<
+    u64,                              // key type: canonical kmer raw encoding
+    CubicEps,                         // bucket fn: 2.4 bits/key, λ=3.5, α=0.99
+    CachelineEfVec<Vec<CachelineEf>>, // remap: Elias-Fano
+    Xx64,                             // hasher: XXH3-64 with seed
+    Vec<u8>,                          // pilots
+>;
+```
+
+`Xx64` is chosen over `FxHash` because canonical kmer raw values are left-aligned u64 with structural zeros in the low bits (42 zeros for k=11, 2 zeros for k=31), which single-multiply hashes distribute poorly.
+
+`CubicEps` with `PtrHashParams::<CubicEps>::default()` (λ=3.5): 2× slower construction than `Linear/λ=3.0`, ~20% less space.
+
+---
+
+## Column append and merge support
+
+These methods extend existing layers with new genome columns without touching the MPHF.
+
+### Layer-level genome column append
+
+```rust
+impl Layer<PersistentBitMatrix> {
+    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> bool) -> OLMResult<()>
+}
+
+impl Layer<PersistentCompactIntMatrix> {
+    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> u32) -> OLMResult<()>
+}
+```
+
+Both delegate to the corresponding `PersistentBitMatrix::append_column` / `PersistentCompactIntMatrix::append_column`. They write a new column file (`col_NNNNNN.pbiv` / `col_NNNNNN.pciv`) and update `meta.json` to increment `n_cols`. `value_of` is called once per slot (0..n).
+
+### Presence matrix initialisation
+
+```rust
+impl Layer<()> {
+    pub fn init_presence_matrix(layer_dir: &Path, n_kmers: usize) -> OLMResult<()>
+}
+```
+
+Called on the first merge of a Presence-mode index. Creates `presence/` with `meta.json {"n": n_kmers, "n_cols": 1}` and `col_000000.pbiv` set entirely to `true`. This retroactively records genome 0 (the original source) as present in every slot, satisfying the column-count invariant before any new-source column is appended.
+
+### Why the MPHF is never rebuilt
+
+The MPHF, evidence, and unitigs are built once from the kmer set of a layer and are immutable for the lifetime of that layer. Adding a genome column does not change the kmer set — it only appends a new data column indexed by the same slot numbers. The only disk writes are one new `.pciv`/`.pbiv` file and a single `meta.json` update.
+
+---
+
+## Add-layer algorithm
+
+When adding dataset B to an existing index:
+
+1. For each partition, probe existing layers for kmers of B routed to that partition.
+2. Collect kmers absent from all layers → `B \ index`.
+3. Write `B \ index` to a new `unitigs.bin` via `next_layer_writer()`.
+4. Call `Layer<D>::build` (or `build_presence`) on the new layer directory.
+5. Call `push_layer` (or `append_layer`) to register the new layer in `meta.json`.
+
+Each partition's new layer is built independently; the operation is fully parallel across partitions.
+
+---
+
+## Dependencies
+
+| crate | role |
+|---|---|
+| `ptr_hash 1.1` | MPHF per layer |
+| `cacheline-ef 1.1` | compact remap inside ptr_hash |
+| `epserde 0.8` | zero-copy MPHF serialisation |
+| `memmap2 0.9` | mmap of evidence and fingerprint files |
+| `bitvec` | packed b-bit fingerprint storage |
+| `obiskio` | unitig file writer/reader + `.idx` build |
+| `obicompactvec` | payload types + aggregation traits |
+| `rayon 1` | parallel MPHF construction pass |
+| `serde / serde_json` | `PartitionMeta` serialisation |