obikmer/docmd/architecture/siblings.md

Sibling annex — architecture (discussion)

Status: architecture decided (2026-08-14). Implementation not yet mandated.

Two index spaces, uncorrelated

Every kmer stored in a Layer lives in two independent index spaces:

• Iteration order: its position when enumerating unitigs.bin (the superkmer file), deterministic but arbitrary with respect to slot.
• MPHF slot: MphfLayer::index(kmer), the number the MPHF assigns.

The two are not correlated by any formula. Converting from one to the other requires either recomputing the MPHF (kmer → slot) or scanning the iteration stream (kmer → order). There is no slot → kmer operation: the MPHF is a one-way function, not an invertible bijection with a stored inverse. Any method that reconstructs a kmer from a bare slot number is wrong by construction, regardless of the mechanism used (MPHF re-hash, or evidence decode + direct unitig read). See MphfLayer::kmer_at (obilayeredmap/src/mphf_layer.rs) — flagged for removal, currently called from obikindex/siblings/build.rs:122 and family_scan.rs:173.

Two pipelines, never mixed

	origin of the kmer	membership known?	correct mapping
query pipeline	external (caller-supplied)	no	query/find/find_strict — MPHF + evidence check
iteration pipeline	enumerated from this layer's own unitigs.bin	yes, by construction	index/index_batch — MPHF only, no evidence

Evidence exists solely to answer "is this external kmer a member of the layer" for the query pipeline. Using it (or the MPHF) to go the other way — recover a kmer from a slot, or re-verify a kmer that was just produced by iterating the layer — is a conceptual error: evidence can be probabilistic (Approx mode), so any slot→kmer attempt is unsound in general, and pointless even in Exact/Hybrid mode since the kmer was already known.

Sibling annex: an iteration-pipeline artifact only

The sibling annex (FamilyMask/SiblingAnnex, .psib, obicompactvec/src/siblingannex.rs) records, per kmer, whether it is a family minorant and which family members are present in the index. Its only consumers (obikindex/siblings/stats.rs, family_scan.rs) enumerate it exhaustively (0..annex.len()); no query-pipeline code path touches it.

Decision: the annex must be persisted in iteration order, not slot order. This lets readers zip-iterate Layer::iter_kmers() and the annex file directly — one linear, cache-friendly pass, no MPHF/slot indirection, no kmer_at. It also enables specialized iterators building on this zip: minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.

Today the annex is built and stored in slot order (build_layer_sibling_annex, siblings/build.rs): slot_kmer is populated via (0..n_slots).map(|slot| mphf.kmer_at(slot)), and the origin slot is threaded through the whole cross-partition reconciliation pipeline (variant generation, query_partition_with, final mask[slot].fetch_or(...)). This must change to iterating iter_kmers()/enumerate_kmers() and threading the iteration index instead of the slot end to end — eliminating kmer_at from the build path entirely, not just the read path. No slot-indexed intermediate is needed even during construction; the iteration-order id is sufficient throughout.

The cross-partition side of the same pipeline is unaffected: checking whether a generated family-variant kmer exists in another partition is a genuine query-pipeline operation (the variant's membership in the target partition is unknown) and must keep going through KmerPartition::query_partition_with (MPHF + evidence), never a raw index().

Pending work

• Remove MphfLayer::kmer_at.
• siblings/build.rs: build slot_kmer-equivalent via iteration, not kmer_at; thread iteration index instead of slot through the variant/reconciliation pipeline; persist the annex in iteration order.
• siblings/family_scan.rs, stats.rs: read the annex via zipped iteration (iter_kmers().zip(annex_iter)) instead of 0..annex.len() + kmer_at.