obikmer/docmd/architecture/siblings.md

# Sibling annex — architecture (discussion)

Status: architecture decided (2026-08-14). Implementation not yet mandated.

## Two index spaces, uncorrelated

Every kmer stored in a `Layer` lives in two independent index spaces:

- **Iteration order**: its position when enumerating `unitigs.bin` (the
  superkmer file), deterministic but arbitrary with respect to slot.
- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns.

The two are not correlated by any formula. Converting from one to the other
requires either recomputing the MPHF (kmer → slot) or scanning the iteration
stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a
one-way function, not an invertible bijection with a stored inverse. Any
method that reconstructs a kmer from a bare slot number is wrong by
construction, regardless of the mechanism used (MPHF re-hash, or evidence
decode + direct unitig read). See `MphfLayer::kmer_at`
(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
"Pending work" status below).

## Two pipelines, never mixed

| | origin of the kmer | membership known? | correct mapping |
|---|---|---|---|
| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check |
| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence |

Evidence exists solely to answer "is this external kmer a member of the
layer" for the query pipeline. Using it (or the MPHF) to go the other way —
recover a kmer from a slot, or re-verify a kmer that was just produced by
iterating the layer — is a conceptual error: evidence can be probabilistic
(`Approx` mode), so any slot→kmer attempt is unsound in general, and
pointless even in `Exact`/`Hybrid` mode since the kmer was already known.

## Sibling annex: an iteration-pipeline artifact only

The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`,
`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a
family minorant and which family members are present in the index. Its only
consumers (`obikphylo/siblings/stats.rs`, `family_scan.rs`) enumerate it
exhaustively (`0..annex.len()`); no query-pipeline code path touches it.

**Decision**: the annex must be persisted in iteration order, not slot
order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex
file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
no `kmer_at`. It also enables specialized iterators building on this zip:
minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.

Today the annex is built and stored in **slot** order
(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated
via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is
threaded through the whole cross-partition reconciliation pipeline (variant
generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This
must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading
the **iteration index** instead of the slot end to end — eliminating
`kmer_at` from the build path entirely, not just the read path. No
slot-indexed intermediate is needed even during construction; the
iteration-order id is sufficient throughout.

The cross-partition side of the same pipeline is unaffected: checking
whether a generated family-variant kmer exists in another partition is a
genuine query-pipeline operation (the variant's membership in the *target*
partition is unknown) and must keep going through
`KmerPartition::query_partition_with` (MPHF + evidence), never a raw
`index()`.

## Pending work — done

The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
traits over `obikindex::KmerIndex`/`obilayeredmap::Layer<D>`, replacing the
old `obikindex::siblings` module) builds and reads the annex purely in
iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
batch variants, mirroring `Layer<D>`'s own `KmerIter`/`KmerBatchIter`
shape). `MphfLayer::kmer_at` has no remaining callers.

A separate, unrelated bug surfaced during this work and was fixed
(2026-08-14): `MphfLayer::enumerate_kmers_batch` computed its
`batch_start_index` via the stdlib `.enumerate()` adapter, which counts
*batches* (0, 1, 2…), not the cumulative k-mer offset the annex is actually
keyed on — every batch past the first wrote its mask/annex entries at the
wrong iteration-order position. Fixed by tracking a running offset instead;
regression tests added (`sibling_annex_no_empty_masks_after_build`,
`sibling_histogram_does_not_panic_on_partial_last_batch`).

## Performance: `build_sibling_annex` parallelism (2026-08-14)

Investigated on a real multi-genome run (`phyloskims_sal_vac`, k=31/m=11).
Baseline: mostly one active core, with short multi-core bursts — average
~3 cores.

**Fixes that helped, kept:**

- `CanonicalKmerOf::minimizer()` (`obikseq/src/kmer.rs`) — a direct O(k)
  bit-arithmetic minimiser for a single isolated k-mer, replacing a
  `RollingStat` instance fed byte-by-byte through an ASCII round-trip (used
  by `helpers::partition_of`, called for every generated family variant).
  ~3x wall-clock improvement on its own, confirmed by sampling
  (`obiskbuilder::rolling_stat`/`obikentropy` frames disappeared from the
  hot path). `CanonicalKmerOf::partition()` added alongside it (wraps
  `minimizer().seq_hash() & mask`, the same routing rule
  `KmerPartition`/`RoutableSuperKmer` use).
- Cross-partition resolution (`outgoing.par_iter()` in
  `build_layer_sibling_annex`) parallelised at the *partition* level — one
  Rayon task per non-empty `outgoing[dest]` bucket. For k=31/m=11, a
  central-base substitution changes the winning minimiser (and thus the
  destination partition) only when that window overlaps the central base:
  ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants
  route right back to the partition already being built. That self bucket
  ends up far larger than any other, so the per-partition split pinned one
  thread to it alone while the rest of the pool finished instantly —
  confirmed by sampling: one thread solid in `MphfLayer::find`, everyone
  else idle. Fixed by splitting each non-empty bucket into
  `total_queries / n_workers` (capped 4096) chunks *before* `par_iter()`,
  preserving per-partition mmap locality (each chunk stays contiguous
  within one partition) while letting Rayon spread an oversized bucket
  across several threads. Net effect of both fixes together: ~3 cores
  average → ~10-13 cores average on the same run, and a projected total
  build time of ~1h15 down to ~30min on the real `phyloskims_sal_vac` run
  this was measured against.
- `TracedBar`'s ETA (`obisys/src/progress.rs`) was silently starved: the
  custom progress message and the self-computed ETA text used to share one
  `pb.set_message()` slot, with the ETA holding off for 2s after any custom
  message — fine when custom messages are rare, broken once
  `build_sibling_annex`'s per-partition callback fires more often than
  that. Fixed by keeping the two texts in separate fields, composed
  together on every render instead of one overwriting the other.

**Tried and reverted — do not repeat blindly:**

- Parallelising the *outer* partition loop in `build_sibling_annex` with
  `obikindex::PartitionRunner` (already used by `merge`/`build_layers`),
  splitting a fixed core budget between outer (partition) and inner
  (pipeline + resolution) concurrency so their product wouldn't exceed the
  budget. Measured *worse*: throughput dropped over time (26
  partitions/5min → 38/11-12min) and peak resolution concurrency fell from
  ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which
  scales very well on its own — to make room for outer concurrency, and
  running several partitions' resolution at once scatters access across
  multiple partitions' mmap regions at once, working against the
  locality `outgoing`'s per-partition grouping exists for. `PartitionRunner`
  stayed exported from `obikindex` (`new_capped` too) since it's
  general-purpose, but nothing in `obikphylo` calls it.
- Splitting resolution chunks even finer (`/(n_workers*8)`, cap 1024,
  instead of `/n_workers`, cap 4096) to smooth the residual sawtooth.
  Measured ~10% *slower*, wider dips, not narrower. Reverted to the
  original chunk sizing.

**Known remaining limitation, not yet worth fixing:** within one layer, the
four stages (sequential `unitigs.bin` read → parallel generation →
parallel resolution → sequential annex write) never overlap — confirmed by
1s-interval sampling: generation alone occupies ~17 threads evenly, but the
next layer's read/generation never starts until the current layer's
resolution and write are both done. This produces a real, periodic (~layer
duration) alternation between "many cores" and "few cores" that neither of
the fixes above touches, since both operate *within* one layer's resolution
step. The only remaining lever is overlapping consecutive layers (e.g. a
depth-2 pipeline: start layer N+1's read/generation while layer N's
resolution/write is still running) — a real restructuring, not a parameter
tweak, and explicitly *not* to be combined with the reverted
budget-capping idea above (let each phase use however many cores it
naturally wants; only the *scheduling* needs to overlap). Deferred, not
started.

## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)

`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`,
`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already
implements most of a dispatch/accumulate/resolve pipeline: generation
(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and
buckets cross-partition queries) runs on an `obipipeline::throttle` +
`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter`
*across partitions*, one generated batch resolved at a time, never several
concurrently — this ordering is deliberate, see the module's own docs on a
reverted concurrent-batch-resolution attempt that scattered mmap access).
The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`)
already exists: `n_layers <= 7` (checked once from the first non-empty
partition's `PartitionMeta::n_layers`, documented as identical across every
partition of an index — a structural, build-time property, never a
per-partition state) decides whether `FamilyMask`'s recorded `layer_value`
can be trusted to skip straight to the right layer
(`find_presence_batch_fast`) or must fall back to scanning every layer of
the destination partition (`find_presence_batch`).

**Real gap, confirmed not implemented**: resolution is triggered by the
*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the
scanned layer), not by an *output* accumulator filling up. Since most
central-base variants of a family route back to the same partition being
scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s
`outgoing[dest]` is large for the local/self partition and thin for the
other ~255 (or however many) destination partitions — each of those gets
resolved at low query density every batch instead of being accumulated
across several source batches until resolving it is worthwhile. This is
distinct from, and not fixed by, the fast/slow layer gate above.

Redesign sketched (not built): per-destination accumulators decoupled from
`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch
completion — a "hot" accumulator for the partition being scanned (sharded
one-per-generation-worker, no lock, since all `n_workers` pipeline workers
write to it concurrently — this differs from an earlier, simpler mental
model of "one thread owns one layer's local collector," which doesn't hold
here since `n_workers` threads cooperate on scanning *one* layer at a time,
not one thread per layer) and "cold" mutex-per-partition accumulators for
the rest, low contention expected since traffic to any single cold
destination is a small fraction of total.

This breaks the current strict-iteration-order delivery of `on_family`
(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH`
resolves atomically). With cross-batch accumulation, a family only becomes
complete once *every* accumulator holding one of its outgoing queries has
flushed, at unpredictable, independent times — no longer streamable
strictly in order without a large, unbounded pending buffer. Resolution
sketched: replace order-dependent consumers with coordinate-addressed
writes instead of order-dependent appends (see `PseudoAlignment` idea
below) wherever possible, since `sibling_annex_stats`'s reduction (plain
counts) is already order-independent and needs nothing here.

**Superseded 2026-08-15** by the `--subsample`/`--shannon` design below,
which sidesteps the accumulator redesign for now: bounding the number of
families actually resolved per layer (via sampling) keeps per-layer
resolution volume small enough that the batch-density problem above stops
mattering in practice for these two consumers. The accumulator redesign
remains relevant for a future *unsampled, full-index* run, but is not
required to ship `--subsample`/`--shannon`.

## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15)

The reference run (`phyloskims_sal_vac`-scale bacterial test set,
`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites
(4.8 MB) — trivially small. The in-progress plant index is expected to
carry on the order of 9 billion minorant families; a dense byte-per-cell
alignment at that column count is unbuildable regardless of genome count
(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000–
8,000 genomes on a large machine, which makes the per-cell cost dominant in
the other dimension too. Pruning the retained family set before
materializing anything is mandatory, not an optimization.

**Already free**: `family_size() < 2` (no sibling variant registered at
all) is a zero-cost structural filter, read directly off `FamilyMask` bits,
already applied in `snp_pseudo_alignment`. Insufficient alone — per the
~80-85% mono-family estimate from earlier discussion, this only brings 9
billion down to roughly 1.3-1.8 billion, still unusable.

**Entropy definition — settled 2026-08-15, correcting an earlier wrong
turn.** The project does **not** encode families as IUPAC ambiguity codes
interpreted the classical way (Fitch-parsimony subset-compatibility, or
ML's "one true state, uncertain which"); see
`docmd/theory/evolutionary_distances.md` ("Why the IUPAC/DNA encoding used
for the first `--snp` test was wrong") and the Sankoff resolution that
followed it. The real model is a genuine 16-state alphabet (the powerset of
`{A,C,G,T}`, `∅` included as a real state) scored with a *calibrated
pairwise cost matrix* (`obikphylo::cardcomp::pairwise_cost_matrix`,
`cmd/phylo/sankoff.rs`), not a compatibility/subset relation between
states. Under that model, each of the 16 states — including multi-bit ones
like `AC` — is a first-class, independently-costed state, not an
uncertainty encoding of a single true base. So: **entropy over the 15
non-empty states (`∅` excluded, matching the earlier decision to exclude
genomes where the family is absent) is the correct informativeness
measure** for this project — not a 4-symbol reduction, which would discard
exactly the cardinality/composition information the calibrated cost matrix
is built to exploit.

## `--subsample` / `--shannon` — sampling strategy (decided 2026-08-15)

Goal: make both the pseudo-alignment (`--snp`) and a Shannon-entropy
diagnostic usable at any index scale, from the 13-genome bacterial
reference run up to the 9-billion-family plant index, without requiring the
batched-accumulator redesign above.

**`--subsample N`** (integer, families to retain): bounds the pseudo-alignment
to `N` minorant families, sampled **proportionally per layer** among
non-monomorphic minorants (`family_size >= 2`) — this sidesteps the need
for a true global reservoir merge across layers while still approximating a
uniform sample over the whole index, and directly answers the earlier open
question of global-vs-per-layer selection scope.

Three passes, in order:

1. **Global count** (cheap, structural, parallel across layers — same shape
   as the existing `sibling_family_size_histogram`, extended to report a
   **per-layer** breakdown rather than one index-wide aggregate): for each
   layer, `count_layer` = number of non-monomorphic minorants. Gives
   `total_count = Σ count_layer`.
2. **Per-layer proportional reservoir sampling** (cheap, structural, one
   pass per layer, no cross-partition resolution): `N_layer = round(N ×
   count_layer / total_count)`. Since `N_layer` is a proportion of
   `count_layer`, it can never exceed it as long as `N <= total_count` — the
   one edge case is `total_count <= N`, in which case sampling is skipped
   entirely and *every* non-monomorphic minorant of every layer is kept
   (no reservoir needed, `N` was never a real constraint). Otherwise:
   Algorithm-R reservoir sampling over the layer's non-monomorphic minorant
   indices, producing `N_layer` iteration-order indices directly, no
   intermediate full list ever materialized.
3. **Filtered resolution** (the expensive step, the existing
   `scan_layer_families` engine, unchanged): re-scan the layer, generating
   and resolving cross-partition queries **only** for the indices selected
   in step 2 (cheap membership test against a small per-layer index set) —
   this is what keeps `--subsample` cheap even on an unsampled-scale index,
   since the cross-partition resolution volume is bounded by `N`, not by
   the layer's true size.

Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir
sampling would waste step-3's expensive resolution work on candidates later
evicted by the reservoir. Step 1 must fully complete (every layer) before
step 2 can start for any layer, since `total_count` is a global quantity.

**`--shannon`** (no argument): emits a CSV of per-family Shannon entropy
(15 non-empty states, `∅`/absent genomes excluded from the denominator, per
the settled definition above). Independent of `--subsample` — entropy is
computed and written per family as soon as its `genome_mask` resolves,
O(1) memory per family, so it streams fine even unsampled at full index
scale (a time cost, not a memory one). Combined with `--subsample N`, it
delivers the original exploratory diagnostic (e.g. `--subsample 1000000
--shannon`) directly from this general machinery, rather than a
purpose-built one-off script.

Validated end-to-end (2026-08-15) against real data: `--sibling-hist` on
`phyloskims_sal_vac` (91 real genomes, k=31/m=11, 256 partitions × 2
layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238
total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier
guess, corrected here). `--subsample`/`--shannon` on the smaller 20-genome
bacterial reference (`benchmark/global_index_presence`) produced a sample
size within rounding of the request (99,742/100,000) and a
[0.8,1.2)-bucket share (40.7%) matching the full unsampled population
(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a
real bug in `reservoir_sample_layer` (see next section), now fixed.

**Bug found and fixed (2026-08-15): `family_idx` numbering mismatch.**
`scan_layer_families`'s `family_idx` counts *every minorant* of a layer
(monomorphic ones included, since `iter_minorants_batch` filters only on
`is_minorant()`), not the raw annex slot (`SiblingAnnex::get(slot)` spans
every k-mer, minorant or not) and not a counter over non-monomorphic
minorants alone. `subsample.rs`'s `reservoir_sample_layer` originally
stored raw slot numbers in its `HashSet<usize>` selection, which drifts
away from `family_idx` as soon as *any* monomorphic minorant is seen —
i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by
tracking two separate counters: `family_idx` (every minorant, matching
`scan_layer_families`) and `seen` (non-monomorphic minorants only, what
Algorithm R actually samples over) — only `family_idx` values are ever
stored in the selection set. The existing unit test never exercised this
(its fixture has exactly one non-monomorphic family, always hitting the
"keep everything" shortcut) — a stronger fixture with several interleaved
monomorphic/non-monomorphic families would be needed to catch a regression
here automatically; not yet written.

## Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15)

Motivation: on real data (bacterial reference, full unsampled run), only
~5‰ of non-monomorphic minorants fall in the `[0.5, 1.5]` bit band judged
phylogenetically interesting (entropy too low = uninformative near-invariant
site; too high = saturated/noisy, see `family_entropy`'s 15-state
discussion) — roughly 1 in 10,000 minorants overall. Computing exact
entropy for every candidate just to discard 99.99% of them is wasteful at
the full 9-billion scale.

**The idea**: `PersistentBitMatrix::col_view(c)` gives a genome's whole
presence column as a `BitSliceView` (sequential, no MPHF, no cross-partition
routing — purely local to one layer's own matrix). Accumulating
`TempCompactIntVecBuilder::inc_present(col)` (already exists in
`obicompactvec/src/builder.rs:121`, along with `add`/`min`/`max`/`diff` on
`IntSliceView` — no new low-level API needed) over every column of a layer
produces `coverage[slot]`: how many genomes carry each exact k-mer, in one
sequential per-layer pass, entirely decoupled from family/sibling
structure. Persisted once per layer, a family's members' coverage could
then be looked up via a plain local MPHF `index()` (cheap) instead of a
full cross-partition presence resolution (`find_presence_batch`) — the
expensive part today is specifically the cross-partition/cross-layer
routing to a sibling's own matrix, not the bit-reading itself, and
`coverage[slot]` sidesteps that routing entirely by moving the cost into a
one-time, purely local, embarrassingly-parallel build step.

**Why not implemented**: `coverage[slot]` is a per-member marginal —
summing members' coverages to approximate a family's entropy silently
assumes no genome carries more than one member at once. It cannot
represent or detect joint co-occurrence (a genome carrying both `A` and
`C` at once, i.e. a combined 15-symbol state) at all, which is exactly the
phenomenon `family_entropy`'s 15-state definition exists to capture (see
`CardinalityTally`/`cardinality_transition_probs`, the project's own
existing machinery for this same co-occurrence structure, built for the
Sankoff matrix calibration). A family that is in truth uniformly `AC`
across every carrying genome would look like a well-balanced 2-state split
under the marginal approximation (entropy ≈ 1) while its true 15-state
entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the
"saturated, uninformative" tail this pre-filter would need to catch,
undermining the point. It stays plausible as a coarse filter for the
*low* tail only (a dominant single member's marginal share reliably
predicts low true entropy too), but not as a stand-in for the high tail —
not pursued further for now.

## Two entropy definitions kept side by side, for comparison (2026-08-15)

`--shannon`'s CSV carries both `entropy15` (`family_entropy` — the settled
15-non-empty-state definition, see above) and `entropy4` (`family_entropy_4`
— plain nucleotide reduction), computed from the same already-resolved
`genome_mask`, not from the marginal approximation above. A genome carrying
several bases at once contributes to *each* base's count (counted once per
base present, not fractionally split, not folded into one combined state)
— a genome polymorphic for the family is present at more than one base by
construction, so it is expected to count more than once; the denominator is
the total base-occurrence count, not the genome count (the two coincide
only when no genome carries more than one base). Kept side by side
specifically to measure, on real data, how much the two diverge — not yet
analyzed.