78 lines
4.0 KiB
Markdown
78 lines
4.0 KiB
Markdown
|
# Sibling annex — architecture (discussion)
|
||
|
|
|
||
|
|
Status: architecture decided (2026-08-14). Implementation not yet mandated.
|
||
|
|
|
||
|
|
## Two index spaces, uncorrelated
|
||
|
|
|
||
|
|
Every kmer stored in a `Layer` lives in two independent index spaces:
|
||
|
|
|
||
|
|
- **Iteration order**: its position when enumerating `unitigs.bin` (the
|
||
|
|
superkmer file), deterministic but arbitrary with respect to slot.
|
||
|
|
- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns.
|
||
|
|
|
||
|
|
The two are not correlated by any formula. Converting from one to the other
|
||
|
|
requires either recomputing the MPHF (kmer → slot) or scanning the iteration
|
||
|
|
stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a
|
||
|
|
one-way function, not an invertible bijection with a stored inverse. Any
|
||
|
|
method that reconstructs a kmer from a bare slot number is wrong by
|
||
|
|
construction, regardless of the mechanism used (MPHF re-hash, or evidence
|
||
|
|
decode + direct unitig read). See `MphfLayer::kmer_at`
|
||
|
|
(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
|
||
|
|
from `obikindex/siblings/build.rs:122` and `family_scan.rs:173`.
|
||
|
|
|
||
|
|
## Two pipelines, never mixed
|
||
|
|
|
||
|
|
| | origin of the kmer | membership known? | correct mapping |
|
||
|
|
|---|---|---|---|
|
||
|
|
| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check |
|
||
|
|
| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence |
|
||
|
|
|
||
|
|
Evidence exists solely to answer "is this external kmer a member of the
|
||
|
|
layer" for the query pipeline. Using it (or the MPHF) to go the other way —
|
||
|
|
recover a kmer from a slot, or re-verify a kmer that was just produced by
|
||
|
|
iterating the layer — is a conceptual error: evidence can be probabilistic
|
||
|
|
(`Approx` mode), so any slot→kmer attempt is unsound in general, and
|
||
|
|
pointless even in `Exact`/`Hybrid` mode since the kmer was already known.
|
||
|
|
|
||
|
|
## Sibling annex: an iteration-pipeline artifact only
|
||
|
|
|
||
|
|
The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`,
|
||
|
|
`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a
|
||
|
|
family minorant and which family members are present in the index. Its only
|
||
|
|
consumers (`obikindex/siblings/stats.rs`, `family_scan.rs`) enumerate it
|
||
|
|
exhaustively (`0..annex.len()`); no query-pipeline code path touches it.
|
||
|
|
|
||
|
|
**Decision**: the annex must be persisted in iteration order, not slot
|
||
|
|
order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex
|
||
|
|
file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
|
||
|
|
no `kmer_at`. It also enables specialized iterators building on this zip:
|
||
|
|
minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.
|
||
|
|
|
||
|
|
Today the annex is built and stored in **slot** order
|
||
|
|
(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated
|
||
|
|
via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is
|
||
|
|
threaded through the whole cross-partition reconciliation pipeline (variant
|
||
|
|
generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This
|
||
|
|
must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading
|
||
|
|
the **iteration index** instead of the slot end to end — eliminating
|
||
|
|
`kmer_at` from the build path entirely, not just the read path. No
|
||
|
|
slot-indexed intermediate is needed even during construction; the
|
||
|
|
iteration-order id is sufficient throughout.
|
||
|
|
|
||
|
|
The cross-partition side of the same pipeline is unaffected: checking
|
||
|
|
whether a generated family-variant kmer exists in another partition is a
|
||
|
|
genuine query-pipeline operation (the variant's membership in the *target*
|
||
|
|
partition is unknown) and must keep going through
|
||
|
|
`KmerPartition::query_partition_with` (MPHF + evidence), never a raw
|
||
|
|
`index()`.
|
||
|
|
|
||
|
|
## Pending work
|
||
|
|
|
||
|
|
- Remove `MphfLayer::kmer_at`.
|
||
|
|
- `siblings/build.rs`: build `slot_kmer`-equivalent via iteration, not
|
||
|
|
`kmer_at`; thread iteration index instead of slot through the
|
||
|
|
variant/reconciliation pipeline; persist the annex in iteration order.
|
||
|
|
- `siblings/family_scan.rs`, `stats.rs`: read the annex via zipped
|
||
|
|
iteration (`iter_kmers().zip(annex_iter)`) instead of `0..annex.len()` +
|
||
|
|
`kmer_at`.
|