obikmer/docmd/architecture/siblings.md

Sibling annex — architecture (discussion)

Status: architecture decided (2026-08-14). Implementation not yet mandated.

Two index spaces, uncorrelated

Every kmer stored in a Layer lives in two independent index spaces:

• Iteration order: its position when enumerating unitigs.bin (the superkmer file), deterministic but arbitrary with respect to slot.
• MPHF slot: MphfLayer::index(kmer), the number the MPHF assigns.

The two are not correlated by any formula. Converting from one to the other requires either recomputing the MPHF (kmer → slot) or scanning the iteration stream (kmer → order). There is no slot → kmer operation: the MPHF is a one-way function, not an invertible bijection with a stored inverse. Any method that reconstructs a kmer from a bare slot number is wrong by construction, regardless of the mechanism used (MPHF re-hash, or evidence decode + direct unitig read). See MphfLayer::kmer_at (obilayeredmap/src/mphf_layer.rs) — flagged for removal, currently called from obikphylo/siblings/build.rs and family_scan.rs (since removed — see "Pending work" status below).

Two pipelines, never mixed

	origin of the kmer	membership known?	correct mapping
query pipeline	external (caller-supplied)	no	query/find/find_strict — MPHF + evidence check
iteration pipeline	enumerated from this layer's own unitigs.bin	yes, by construction	index/index_batch — MPHF only, no evidence

Evidence exists solely to answer "is this external kmer a member of the layer" for the query pipeline. Using it (or the MPHF) to go the other way — recover a kmer from a slot, or re-verify a kmer that was just produced by iterating the layer — is a conceptual error: evidence can be probabilistic (Approx mode), so any slot→kmer attempt is unsound in general, and pointless even in Exact/Hybrid mode since the kmer was already known.

Sibling annex: an iteration-pipeline artifact only

The sibling annex (FamilyMask/SiblingAnnex, .psib, obicompactvec/src/siblingannex.rs) records, per kmer, whether it is a family minorant and which family members are present in the index. Its only consumers (obikphylo/siblings/stats.rs, family_scan.rs) enumerate it exhaustively (0..annex.len()); no query-pipeline code path touches it.

Decision: the annex must be persisted in iteration order, not slot order. This lets readers zip-iterate Layer::iter_kmers() and the annex file directly — one linear, cache-friendly pass, no MPHF/slot indirection, no kmer_at. It also enables specialized iterators building on this zip: minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.

Today the annex is built and stored in slot order (build_layer_sibling_annex, siblings/build.rs): slot_kmer is populated via (0..n_slots).map(|slot| mphf.kmer_at(slot)), and the origin slot is threaded through the whole cross-partition reconciliation pipeline (variant generation, query_partition_with, final mask[slot].fetch_or(...)). This must change to iterating iter_kmers()/enumerate_kmers() and threading the iteration index instead of the slot end to end — eliminating kmer_at from the build path entirely, not just the read path. No slot-indexed intermediate is needed even during construction; the iteration-order id is sufficient throughout.

The cross-partition side of the same pipeline is unaffected: checking whether a generated family-variant kmer exists in another partition is a genuine query-pipeline operation (the variant's membership in the target partition is unknown) and must keep going through KmerPartition::query_partition_with (MPHF + evidence), never a raw index().

Pending work — done

The plan above shipped: obikphylo (a new crate — phylo-domain extension traits over obikindex::KmerIndex/obilayeredmap::Layer<D>, replacing the old obikindex::siblings module) builds and reads the annex purely in iteration order (SiblingLayerExt::iter_siblings/iter_minorants, both with batch variants, mirroring Layer<D>'s own KmerIter/KmerBatchIter shape). MphfLayer::kmer_at has no remaining callers.

A separate, unrelated bug surfaced during this work and was fixed (2026-08-14): MphfLayer::enumerate_kmers_batch computed its batch_start_index via the stdlib .enumerate() adapter, which counts batches (0, 1, 2…), not the cumulative k-mer offset the annex is actually keyed on — every batch past the first wrote its mask/annex entries at the wrong iteration-order position. Fixed by tracking a running offset instead; regression tests added (sibling_annex_no_empty_masks_after_build, sibling_histogram_does_not_panic_on_partial_last_batch).

Performance: build_sibling_annex parallelism (2026-08-14)

Investigated on a real multi-genome run (phyloskims_sal_vac, k=31/m=11). Baseline: mostly one active core, with short multi-core bursts — average ~3 cores.

Fixes that helped, kept:

• CanonicalKmerOf::minimizer() (obikseq/src/kmer.rs) — a direct O(k) bit-arithmetic minimiser for a single isolated k-mer, replacing a RollingStat instance fed byte-by-byte through an ASCII round-trip (used by helpers::partition_of, called for every generated family variant). ~3x wall-clock improvement on its own, confirmed by sampling (obiskbuilder::rolling_stat/obikentropy frames disappeared from the hot path). CanonicalKmerOf::partition() added alongside it (wraps minimizer().seq_hash() & mask, the same routing rule KmerPartition/RoutableSuperKmer use).
• Cross-partition resolution (outgoing.par_iter() in build_layer_sibling_annex) parallelised at the partition level — one Rayon task per non-empty outgoing[dest] bucket. For k=31/m=11, a central-base substitution changes the winning minimiser (and thus the destination partition) only when that window overlaps the central base: ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants route right back to the partition already being built. That self bucket ends up far larger than any other, so the per-partition split pinned one thread to it alone while the rest of the pool finished instantly — confirmed by sampling: one thread solid in MphfLayer::find, everyone else idle. Fixed by splitting each non-empty bucket into total_queries / n_workers (capped 4096) chunks before par_iter(), preserving per-partition mmap locality (each chunk stays contiguous within one partition) while letting Rayon spread an oversized bucket across several threads. Net effect of both fixes together: ~3 cores average → ~10-13 cores average on the same run, and a projected total build time of ~1h15 down to ~30min on the real phyloskims_sal_vac run this was measured against.
• TracedBar's ETA (obisys/src/progress.rs) was silently starved: the custom progress message and the self-computed ETA text used to share one pb.set_message() slot, with the ETA holding off for 2s after any custom message — fine when custom messages are rare, broken once build_sibling_annex's per-partition callback fires more often than that. Fixed by keeping the two texts in separate fields, composed together on every render instead of one overwriting the other.

Tried and reverted — do not repeat blindly:

• Parallelising the outer partition loop in build_sibling_annex with obikindex::PartitionRunner (already used by merge/build_layers), splitting a fixed core budget between outer (partition) and inner (pipeline + resolution) concurrency so their product wouldn't exceed the budget. Measured worse: throughput dropped over time (26 partitions/5min → 38/11-12min) and peak resolution concurrency fell from ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which scales very well on its own — to make room for outer concurrency, and running several partitions' resolution at once scatters access across multiple partitions' mmap regions at once, working against the locality outgoing's per-partition grouping exists for. PartitionRunner stayed exported from obikindex (new_capped too) since it's general-purpose, but nothing in obikphylo calls it.
• Splitting resolution chunks even finer (/(n_workers*8), cap 1024, instead of /n_workers, cap 4096) to smooth the residual sawtooth. Measured ~10% slower, wider dips, not narrower. Reverted to the original chunk sizing.

Known remaining limitation, not yet worth fixing: within one layer, the four stages (sequential unitigs.bin read → parallel generation → parallel resolution → sequential annex write) never overlap — confirmed by 1s-interval sampling: generation alone occupies ~17 threads evenly, but the next layer's read/generation never starts until the current layer's resolution and write are both done. This produces a real, periodic (~layer duration) alternation between "many cores" and "few cores" that neither of the fixes above touches, since both operate within one layer's resolution step. The only remaining lever is overlapping consecutive layers (e.g. a depth-2 pipeline: start layer N+1's read/generation while layer N's resolution/write is still running) — a real restructuring, not a parameter tweak, and explicitly not to be combined with the reverted budget-capping idea above (let each phase use however many cores it naturally wants; only the scheduling needs to overlap). Deferred, not started.