Eric Coissac
b1f54b7d2f
Introduces batch retrieval and sub-matrix extraction methods across vector, view, reader, and matrix types. These implementations optimize cache locality by sorting requested indices for sequential memory access before applying an inverse permutation to restore original order. Includes allocation-free variants that populate caller-provided buffers. Updates architecture documentation to define sibling annex persistence in iteration order and clarify pipeline separation.
78 lines
4.0 KiB
Markdown
78 lines
4.0 KiB
Markdown
# Sibling annex — architecture (discussion)
|
|
|
|
Status: architecture decided (2026-08-14). Implementation not yet mandated.
|
|
|
|
## Two index spaces, uncorrelated
|
|
|
|
Every kmer stored in a `Layer` lives in two independent index spaces:
|
|
|
|
- **Iteration order**: its position when enumerating `unitigs.bin` (the
|
|
superkmer file), deterministic but arbitrary with respect to slot.
|
|
- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns.
|
|
|
|
The two are not correlated by any formula. Converting from one to the other
|
|
requires either recomputing the MPHF (kmer → slot) or scanning the iteration
|
|
stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a
|
|
one-way function, not an invertible bijection with a stored inverse. Any
|
|
method that reconstructs a kmer from a bare slot number is wrong by
|
|
construction, regardless of the mechanism used (MPHF re-hash, or evidence
|
|
decode + direct unitig read). See `MphfLayer::kmer_at`
|
|
(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
|
|
from `obikindex/siblings/build.rs:122` and `family_scan.rs:173`.
|
|
|
|
## Two pipelines, never mixed
|
|
|
|
| | origin of the kmer | membership known? | correct mapping |
|
|
|---|---|---|---|
|
|
| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check |
|
|
| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence |
|
|
|
|
Evidence exists solely to answer "is this external kmer a member of the
|
|
layer" for the query pipeline. Using it (or the MPHF) to go the other way —
|
|
recover a kmer from a slot, or re-verify a kmer that was just produced by
|
|
iterating the layer — is a conceptual error: evidence can be probabilistic
|
|
(`Approx` mode), so any slot→kmer attempt is unsound in general, and
|
|
pointless even in `Exact`/`Hybrid` mode since the kmer was already known.
|
|
|
|
## Sibling annex: an iteration-pipeline artifact only
|
|
|
|
The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`,
|
|
`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a
|
|
family minorant and which family members are present in the index. Its only
|
|
consumers (`obikindex/siblings/stats.rs`, `family_scan.rs`) enumerate it
|
|
exhaustively (`0..annex.len()`); no query-pipeline code path touches it.
|
|
|
|
**Decision**: the annex must be persisted in iteration order, not slot
|
|
order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex
|
|
file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
|
|
no `kmer_at`. It also enables specialized iterators building on this zip:
|
|
minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.
|
|
|
|
Today the annex is built and stored in **slot** order
|
|
(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated
|
|
via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is
|
|
threaded through the whole cross-partition reconciliation pipeline (variant
|
|
generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This
|
|
must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading
|
|
the **iteration index** instead of the slot end to end — eliminating
|
|
`kmer_at` from the build path entirely, not just the read path. No
|
|
slot-indexed intermediate is needed even during construction; the
|
|
iteration-order id is sufficient throughout.
|
|
|
|
The cross-partition side of the same pipeline is unaffected: checking
|
|
whether a generated family-variant kmer exists in another partition is a
|
|
genuine query-pipeline operation (the variant's membership in the *target*
|
|
partition is unknown) and must keep going through
|
|
`KmerPartition::query_partition_with` (MPHF + evidence), never a raw
|
|
`index()`.
|
|
|
|
## Pending work
|
|
|
|
- Remove `MphfLayer::kmer_at`.
|
|
- `siblings/build.rs`: build `slot_kmer`-equivalent via iteration, not
|
|
`kmer_at`; thread iteration index instead of slot through the
|
|
variant/reconciliation pipeline; persist the annex in iteration order.
|
|
- `siblings/family_scan.rs`, `stats.rs`: read the annex via zipped
|
|
iteration (`iter_kmers().zip(annex_iter)`) instead of `0..annex.len()` +
|
|
`kmer_at`.
|