refine k-mer index architecture documentation and remove obsolete spec

Introduces raw mapping and iteration APIs that bypass membership checks, clarifies variant-specific storage layouts and auto-detection logic, and documents optimized batch access patterns with caller-provided buffers. Removes the outdated obicompactvector_reflexion.md specification to consolidate architectural details into current implementation docs.
This commit is contained in:
Eric Coissac
2026-08-20 13:20:29 +02:00
parent 3b65319529
commit 0da725ffe9
4 changed files with 62 additions and 107 deletions
+53
View File
@@ -250,6 +250,59 @@ Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers buil
---
## Layer\<D\> — raw mapping, iteration, and batch access
Beyond `query`/`find` (membership-checked), `Layer<D>` exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.
### Raw kmer → slot mapping
```rust
pub fn index(&self, kmer: CanonicalKmer) -> usize
pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize>
```
Pure MPHF mapping, no evidence/fingerprint check — equivalent to `MphfOnly::index`. Only meaningful when the caller already knows `kmer` belongs to the layer; on an absent kmer the MPHF still returns *some* slot (undefined, not `None`).
### Kmer iteration
Four iterators, all built from `unitigs.bin` (physical layout order, **not** correlated with MPHF slot numbers):
```rust
pub fn iter_kmers(&self) -> KmerIter<'_>
pub fn enumerate_kmers(&self) -> Enumerate<KmerIter<'_>> // (order_index, kmer)
pub fn iter_kmers_batch(&self, n: usize) -> KmerBatchIter<'_> // Vec<CanonicalKmer> of size ≤ n
pub fn enumerate_kmers_batch(&self, n: usize) -> impl Iterator<Item = (usize, Vec<CanonicalKmer>)> + Send + 'static
```
`KmerIter`/`KmerBatchIter` own a clone of the underlying `Arc<UnitigFileReader>` rather than borrowing `self``Send + 'static`, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. `enumerate_kmers_batch`'s index is the batch's starting offset in iteration order (a multiple of `n` except for the final, possibly shorter, batch).
### Batch lookup on payload vectors/views
`PersistentCompactIntVec`, `PersistentBitVec`, `IntSliceView`, `BitSliceView` all expose:
```rust
fn get_batch(&self, slots: &[usize]) -> Vec<T>
fn fill_batch(&self, slots: &[usize], out: &mut [T])
```
Both sort `slots` internally for sequential mmap access, then reorder results back to the caller's original order. `fill_batch` fills a caller-provided buffer, avoiding the `Vec` allocation.
### sub_matrix / fill_sub_matrix
```rust
// Layer<PersistentCompactIntMatrix>
pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<u32>> // column-first
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<u32>])
// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)
pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
```
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
---
## LayeredStore\<S\> and aggregation traits
`LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls: