refine k-mer index architecture documentation and remove obsolete spec
Introduces raw mapping and iteration APIs that bypass membership checks, clarifies variant-specific storage layouts and auto-detection logic, and documents optimized batch access patterns with caller-provided buffers. Removes the outdated obicompactvector_reflexion.md specification to consolidate architectural details into current implementation docs.
This commit is contained in:
@@ -250,6 +250,59 @@ Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers buil
|
||||
|
||||
---
|
||||
|
||||
## Layer\<D\> — raw mapping, iteration, and batch access
|
||||
|
||||
Beyond `query`/`find` (membership-checked), `Layer<D>` exposes lower-level access used by consumers that already know a kmer is in the layer (e.g. cross-partition sibling resolution) or that need to sweep every kmer/slot without paying for a membership check each time.
|
||||
|
||||
### Raw kmer → slot mapping
|
||||
|
||||
```rust
|
||||
pub fn index(&self, kmer: CanonicalKmer) -> usize
|
||||
pub fn index_batch(&self, kmers: &[CanonicalKmer]) -> Vec<usize>
|
||||
```
|
||||
|
||||
Pure MPHF mapping, no evidence/fingerprint check — equivalent to `MphfOnly::index`. Only meaningful when the caller already knows `kmer` belongs to the layer; on an absent kmer the MPHF still returns *some* slot (undefined, not `None`).
|
||||
|
||||
### Kmer iteration
|
||||
|
||||
Four iterators, all built from `unitigs.bin` (physical layout order, **not** correlated with MPHF slot numbers):
|
||||
|
||||
```rust
|
||||
pub fn iter_kmers(&self) -> KmerIter<'_>
|
||||
pub fn enumerate_kmers(&self) -> Enumerate<KmerIter<'_>> // (order_index, kmer)
|
||||
pub fn iter_kmers_batch(&self, n: usize) -> KmerBatchIter<'_> // Vec<CanonicalKmer> of size ≤ n
|
||||
pub fn enumerate_kmers_batch(&self, n: usize) -> impl Iterator<Item = (usize, Vec<CanonicalKmer>)> + Send + 'static
|
||||
```
|
||||
|
||||
`KmerIter`/`KmerBatchIter` own a clone of the underlying `Arc<UnitigFileReader>` rather than borrowing `self` — `Send + 'static`, streamed from disk one kmer at a time, never materialised as a whole. Multiple instances can coexist concurrently, each with its own cursor. `enumerate_kmers_batch`'s index is the batch's starting offset in iteration order (a multiple of `n` except for the final, possibly shorter, batch).
|
||||
|
||||
### Batch lookup on payload vectors/views
|
||||
|
||||
`PersistentCompactIntVec`, `PersistentBitVec`, `IntSliceView`, `BitSliceView` all expose:
|
||||
|
||||
```rust
|
||||
fn get_batch(&self, slots: &[usize]) -> Vec<T>
|
||||
fn fill_batch(&self, slots: &[usize], out: &mut [T])
|
||||
```
|
||||
|
||||
Both sort `slots` internally for sequential mmap access, then reorder results back to the caller's original order. `fill_batch` fills a caller-provided buffer, avoiding the `Vec` allocation.
|
||||
|
||||
### sub_matrix / fill_sub_matrix
|
||||
|
||||
```rust
|
||||
// Layer<PersistentCompactIntMatrix>
|
||||
pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<u32>> // column-first
|
||||
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<u32>])
|
||||
|
||||
// Layer<PersistentBitMatrix> (and any D: BinaryMatrix, e.g. PersistentSparseBitMatrix)
|
||||
pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>>
|
||||
pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
|
||||
```
|
||||
|
||||
Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md).
|
||||
|
||||
---
|
||||
|
||||
## LayeredStore\<S\> and aggregation traits
|
||||
|
||||
`LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls:
|
||||
|
||||
Reference in New Issue
Block a user