Status: architecture decided (2026-08-14). Implementation not yet mandated.
-Every kmer stored in a Layer lives in two independent index spaces:
Current state of obikphylo::siblings (crate obikphylo, module siblings).
+Historical rationale, dated findings, and superseded designs live in jj
+history, not here.
A family is the up-to-4 k-mers sharing all flanks and differing only at
+the central base (well-defined for odd k). FamilyMask (siblingannex.rs)
+is a 4-bit presence mask (bit b set iff the family member whose own
+canonical central base is b is observed anywhere in the index) plus a
+minorant flag; family_size()/siblings() derive member count from it.
+SiblingAnnex/SiblingAnnexBuilder persist one FamilyMask per k-mer,
+keyed by iteration order (the physical layout of unitigs.bin), never by
+MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream
+against the annex file directly, no MPHF/slot indirection.
algorithms::annex::build_layer_sibling_annex builds one layer's annex:
+seeds each k-mer's own central base, generates its up-to-3 central-base
+variants, resolves each cross-partition (IndexCache::find_in_partition,
+grouped and chunked per destination partition for mmap locality), writes the
+final mask (with the minorant flag) in a second streamed pass.
+SiblingExt::build_sibling_annex drives this for every layer of every
+partition, then invalidates and rebuilds the entropy and family-overlap
+annexes (their family_idx numbering depends on the sibling annex).
algorithms::family_scan::scan_layer_families is the only place that
+resolves a family's full per-genome presence (genome_mask: &[u8],
+cross-partition) — every other consumer in this module is built on it, never
+reimplements the resolution. Two decoupled concurrency mechanisms: batch
+generation (CPU-only) runs on an obipipeline throttle/pipe; per-batch
+resolution (I/O-bound, mmap page faults) runs rayon-parallel across
+destination partitions, one generated batch at a time (never several
+concurrently, to keep one partition's pages hot instead of scattering
+access). fast_mode (is_fast_mode, <= 7 layers per partition) picks
+between trusting a family member's recorded layer index outright or probing
+every layer of the destination partition.
Selection<'a> bounds which families of a layer are resolved:
unitigs.bin (the
- superkmer file), deterministic but arbitrary with respect to slot.MphfLayer::index(kmer), the number the MPHF assigns.All — every minorant, monomorphic included (SiblingExt::shannon_entropy_csv,
+ sibling_annex_stats).Some(&HashSet<usize>) — a caller-picked subset, skipping the expensive
+ cross-partition resolution entirely for excluded families
+ (subsample::sample_layer's accepted-candidates batch;
+ entropy::ensure_layer_entropy_annex's non-monomorphic-only restriction).The two are not correlated by any formula. Converting from one to the other
-requires either recomputing the MPHF (kmer → slot) or scanning the iteration
-stream (kmer → order). There is no slot → kmer operation: the MPHF is a
-one-way function, not an invertible bijection with a stored inverse. Any
-method that reconstructs a kmer from a bare slot number is wrong by
-construction, regardless of the mechanism used (MPHF re-hash, or evidence
-decode + direct unitig read). See MphfLayer::kmer_at
-(obikindex/src/layer/mphf_layer.rs) — flagged for removal, currently called
-from obikphylo/siblings/build.rs and family_scan.rs (since removed — see
-"Pending work" status below).
| - | origin of the kmer | -membership known? | -correct mapping | -
|---|---|---|---|
| query pipeline | -external (caller-supplied) | -no | -query/find/find_strict — MPHF + evidence check |
-
| iteration pipeline | -enumerated from this layer's own unitigs.bin |
-yes, by construction | -index/index_batch — MPHF only, no evidence |
-
Evidence exists solely to answer "is this external kmer a member of the
-layer" for the query pipeline. Using it (or the MPHF) to go the other way —
-recover a kmer from a slot, or re-verify a kmer that was just produced by
-iterating the layer — is a conceptual error: evidence can be probabilistic
-(Approx mode), so any slot→kmer attempt is unsound in general, and
-pointless even in Exact/Hybrid mode since the kmer was already known.
The sibling annex (FamilyMask/SiblingAnnex, .psib,
-obicompactvec/src/siblingannex.rs) records, per kmer, whether it is a
-family minorant and which family members are present in the index. Its only
-consumers (obikphylo/siblings/stats.rs, family_scan.rs) enumerate it
-exhaustively (0..annex.len()); no query-pipeline code path touches it.
Decision: the annex must be persisted in iteration order, not slot
-order. This lets readers zip-iterate Layer::iter_kmers() and the annex
-file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
-no kmer_at. It also enables specialized iterators building on this zip:
-minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.
Today the annex is built and stored in slot order
-(build_layer_sibling_annex, siblings/build.rs): slot_kmer is populated
-via (0..n_slots).map(|slot| mphf.kmer_at(slot)), and the origin slot is
-threaded through the whole cross-partition reconciliation pipeline (variant
-generation, query_partition_with, final mask[slot].fetch_or(...)). This
-must change to iterating iter_kmers()/enumerate_kmers() and threading
-the iteration index instead of the slot end to end — eliminating
-kmer_at from the build path entirely, not just the read path. No
-slot-indexed intermediate is needed even during construction; the
-iteration-order id is sufficient throughout.
The cross-partition side of the same pipeline is unaffected: checking
-whether a generated family-variant kmer exists in another partition is a
-genuine query-pipeline operation (the variant's membership in the target
-partition is unknown) and must keep going through
-KmerPartition::query_partition_with (MPHF + evidence), never a raw
-index().
The plan above shipped: obikphylo (a new crate — phylo-domain extension
-traits over obikindex::KmerIndex/obikindex::layer::Layer<D>, replacing the
-old obikindex::siblings module) builds and reads the annex purely in
-iteration order (SiblingLayerExt::iter_siblings/iter_minorants, both with
-batch variants, mirroring Layer<D>'s own KmerIter/KmerBatchIter
-shape). MphfLayer::kmer_at has no remaining callers.
A separate, unrelated bug surfaced during this work and was fixed
-(2026-08-14): MphfLayer::enumerate_kmers_batch computed its
-batch_start_index via the stdlib .enumerate() adapter, which counts
-batches (0, 1, 2…), not the cumulative k-mer offset the annex is actually
-keyed on — every batch past the first wrote its mask/annex entries at the
-wrong iteration-order position. Fixed by tracking a running offset instead;
-regression tests added (sibling_annex_no_empty_masks_after_build,
-sibling_histogram_does_not_panic_on_partial_last_batch).
build_sibling_annex parallelism (2026-08-14)Investigated on a real multi-genome run (phyloskims_sal_vac, k=31/m=11).
-Baseline: mostly one active core, with short multi-core bursts — average
-~3 cores.
Fixes that helped, kept:
+minorant_selection::eligible_bitset_layer/non_monomorphic_reindex_layer
+provide the cheap, annex-bits-only (no cross-partition resolution) filters
+these restricted scans are built from: monomorphism (family_size() < 2) is
+knowable straight from the annex, so nothing pays scan_layer_families's
+resolution cost just to discard a monomorphic result.
algorithms::masking::masked_state(m, free_loss, no_ambiguity) is the single
+function that turns a raw per-genome 4-bit state into the masked view every
+consumer needs (None for absence under --free-loss or ambiguity under
+--no-ambiguity). It is called exactly once per (family, genome), in
+subsample::sample_layer, immediately after scan_layer_families resolves
+the raw state — never recomputed downstream. survives_masking (does the
+family still carry ≥2 distinct masked states among included genomes?) and
+SurvivingFamily::masked: Vec<Option<u8>> both consume this same
+precomputed array, which is also what AlignmentReducer/PairwiseReducer
+read (see below) — this is what guarantees the alignment and the pairwise
+tally can never disagree about a genome's state at a given site.
algorithms::entropy::ensure_layer_entropy_annex builds EntropyAnnex
+(entropy_annex.rs, one f32 Shannon-entropy value per non-monomorphic
+minorant, mmap-backed) and/or feeds FamilyOverlapAccumulator
+(family_overlap.rs) from one shared Selection::Some scan — both need the
+same genome_mask resolution, so a missing one never pays for it twice.
+family_entropy (15 non-empty states of {A,C,G,T}'s powerset, ∅
+excluded from the denominator — the project's settled informativeness
+definition, matching the calibrated 16-state Sankoff cost matrix) and
+family_entropy_4 (plain 4-symbol reduction, kept only for comparison) are
+pure functions of genome_mask. extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes
+drives this for every cached layer, lazily (only what's missing).
algorithms::subsample::sample_index draws up to n non-monomorphic
+families index-wide, proportionally per layer: for each layer, a target
+quota sized from that layer's share of the index-wide eligible count, filled
+by a circular walk with a Bernoulli(p) draw per candidate — p optionally
+weighted by an unnormalized Gaussian kernel over the candidate's entropy
+(EntropyBias { mu, sigma }, read from the entropy annex). Accepted
+candidates are resolved in batches (scan_layer_families) and checked
+against survives_masking; rejected-after-masking candidates don't count
+toward quota, triggering further "topup" rounds (bounded, MAX_TOPUP_ROUNDS
+= 5) rather than silently under-filling. Every SurvivingFamily reaching a
+caller has already survived both the draw and post-masking informativeness —
+no second filter pass needed downstream.
algorithms::scan::sampled_scan is the single entry point every sampled
+computation goes through: one sample_index pass, fanned out to whichever
+LayerReducer implementations the caller actually needs, each receiving the
+exact same per-layer SurvivingFamily batch (an Arc, cloned not copied).
+This is what guarantees several outputs requested together
+(--pseudo-alignment and --sankoff and a snp-* --distance in one CLI
+invocation) come from one random sample, not independently-drawn ones —
+there is no other driver of sample_index anywhere in this crate.
Two LayerReducer implementations exist today:
CanonicalKmerOf::minimizer() (obikseq/src/kmer.rs) — a direct O(k)
- bit-arithmetic minimiser for a single isolated k-mer, replacing a
- RollingStat instance fed byte-by-byte through an ASCII round-trip (used
- by helpers::partition_of, called for every generated family variant).
- ~3x wall-clock improvement on its own, confirmed by sampling
- (obiskbuilder::rolling_stat/obikentropy frames disappeared from the
- hot path). CanonicalKmerOf::partition() added alongside it (wraps
- minimizer().seq_hash() & mask, the same routing rule
- KmerPartition/RoutableSuperKmer use).outgoing.par_iter() in
- build_layer_sibling_annex) parallelised at the partition level — one
- Rayon task per non-empty outgoing[dest] bucket. For k=31/m=11, a
- central-base substitution changes the winning minimiser (and thus the
- destination partition) only when that window overlaps the central base:
- ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants
- route right back to the partition already being built. That self bucket
- ends up far larger than any other, so the per-partition split pinned one
- thread to it alone while the rest of the pool finished instantly —
- confirmed by sampling: one thread solid in MphfLayer::find, everyone
- else idle. Fixed by splitting each non-empty bucket into
- total_queries / n_workers (capped 4096) chunks before par_iter(),
- preserving per-partition mmap locality (each chunk stays contiguous
- within one partition) while letting Rayon spread an oversized bucket
- across several threads. Net effect of both fixes together: ~3 cores
- average → ~10-13 cores average on the same run, and a projected total
- build time of ~1h15 down to ~30min on the real phyloskims_sal_vac run
- this was measured against.TracedBar's ETA (obisys/src/progress.rs) was silently starved: the
- custom progress message and the self-computed ETA text used to share one
- pb.set_message() slot, with the ETA holding off for 2s after any custom
- message — fine when custom messages are rare, broken once
- build_sibling_annex's per-partition callback fires more often than
- that. Fixed by keeping the two texts in separate fields, composed
- together on every render instead of one overwriting the other.AlignmentReducer (alignment.rs) — accumulates SnpAlignment (one
+ IUPAC-coded row per included genome), reading family.masked directly.PairwiseReducer (pairwise.rs) — accumulates PairwiseTally/
+ PartitionDispersion, also reading family.masked directly (substitution/
+ agreement counts and cardinality both derive from the same masked state
+ the alignment uses).Tried and reverted — do not repeat blindly:
+PairwiseTally stores per-genome-pair substitution/agreement/cardinality
+counts, unfiltered, from which every downstream output is cheap O(n²)
+post-processing (never a second scan):
build_sibling_annex with
- obikindex::PartitionRunner (already used by merge/build_layers),
- splitting a fixed core budget between outer (partition) and inner
- (pipeline + resolution) concurrency so their product wouldn't exceed the
- budget. Measured worse: throughput dropped over time (26
- partitions/5min → 38/11-12min) and peak resolution concurrency fell from
- ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which
- scales very well on its own — to make room for outer concurrency, and
- running several partitions' resolution at once scatters access across
- multiple partitions' mmap regions at once, working against the
- locality outgoing's per-partition grouping exists for. PartitionRunner
- stayed exported from obikindex (new_capped too) since it's
- general-purpose, but nothing in obikphylo calls it./(n_workers*8), cap 1024,
- instead of /n_workers, cap 4096) to smooth the residual sawtooth.
- Measured ~10% slower, wider dips, not narrower. Reverted to the
- original chunk sizing.algorithms::snp_distance::distance_matrix — closed-form snp-*
+ corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions-
+ only), optionally gamma-corrected (GammaShape, PartitionDispersion::estimate_alpha
+ for Auto).algorithms::sankoff::SankoffBundle::finish — assembles the
+ --sankoff/--tnt/--phyg/--iqtree pipeline's inputs
+ (RawSnpDistanceOutput, BasePairTally, CardinalityTally) from an
+ already-scanned AlignmentReducer/PairwiseReducer pair, plus
+ SankoffBundle::snp_distance to reuse that exact sample for a snp-*
+ --distance in the same invocation without a second scan.algorithms::cardcomp — pure functions turning CardinalityTally/
+ BasePairTally into row-stochastic transition probability matrices, then
+ the calibrated 16-state Sankoff cost matrix (pairwise_cost_matrix).Known remaining limitation, not yet worth fixing: within one layer, the
-four stages (sequential unitigs.bin read → parallel generation →
-parallel resolution → sequential annex write) never overlap — confirmed by
-1s-interval sampling: generation alone occupies ~17 threads evenly, but the
-next layer's read/generation never starts until the current layer's
-resolution and write are both done. This produces a real, periodic (~layer
-duration) alternation between "many cores" and "few cores" that neither of
-the fixes above touches, since both operate within one layer's resolution
-step. The only remaining lever is overlapping consecutive layers (e.g. a
-depth-2 pipeline: start layer N+1's read/generation while layer N's
-resolution/write is still running) — a real restructuring, not a parameter
-tweak, and explicitly not to be combined with the reverted
-budget-capping idea above (let each phase use however many cores it
-naturally wants; only the scheduling needs to overlap). Deferred, not
-started.
family_scan.rs::scan_layer_families (shared by snp_pseudo_alignment,
-sibling_annex_stats, cardinality_tally, scan_family_pairs) already
-implements most of a dispatch/accumulate/resolve pipeline: generation
-(cheap, CPU-only — builds outgoing[dest_partition] from FamilyMask and
-buckets cross-partition queries) runs on an obipipeline::throttle +
-make_pipe! stage, decoupled from resolution (I/O-bound, rayon::par_iter
-across partitions, one generated batch resolved at a time, never several
-concurrently — this ordering is deliberate, see the module's own docs on a
-reverted concurrent-batch-resolution attempt that scattered mmap access).
-The fast/slow mode gate (PartitionCache::fast_mode, cache.rs:162-163)
-already exists: n_layers <= 7 (checked once from the first non-empty
-partition's PartitionMeta::n_layers, documented as identical across every
-partition of an index — a structural, build-time property, never a
-per-partition state) decides whether FamilyMask's recorded layer_value
-can be trusted to skip straight to the right layer
-(find_presence_batch_fast) or must fall back to scanning every layer of
-the destination partition (find_presence_batch).
Real gap, confirmed not implemented: resolution is triggered by the
-source batch finishing (FAMILY_BATCH = 65536 minorants read from the
-scanned layer), not by an output accumulator filling up. Since most
-central-base variants of a family route back to the same partition being
-scanned (~48% per the k=31/m=11 measurement above), a FAMILY_BATCH's
-outgoing[dest] is large for the local/self partition and thin for the
-other ~255 (or however many) destination partitions — each of those gets
-resolved at low query density every batch instead of being accumulated
-across several source batches until resolving it is worthwhile. This is
-distinct from, and not fixed by, the fast/slow layer gate above.
Redesign sketched (not built): per-destination accumulators decoupled from
-FAMILY_BATCH, flushed on reaching a size threshold instead of on source-batch
-completion — a "hot" accumulator for the partition being scanned (sharded
-one-per-generation-worker, no lock, since all n_workers pipeline workers
-write to it concurrently — this differs from an earlier, simpler mental
-model of "one thread owns one layer's local collector," which doesn't hold
-here since n_workers threads cooperate on scanning one layer at a time,
-not one thread per layer) and "cold" mutex-per-partition accumulators for
-the rest, low contention expected since traffic to any single cold
-destination is a small fraction of total.
This breaks the current strict-iteration-order delivery of on_family
-(today: a reorder buffer keyed by batch, since a whole FAMILY_BATCH
-resolves atomically). With cross-batch accumulation, a family only becomes
-complete once every accumulator holding one of its outgoing queries has
-flushed, at unpredictable, independent times — no longer streamable
-strictly in order without a large, unbounded pending buffer. Resolution
-sketched: replace order-dependent consumers with coordinate-addressed
-writes instead of order-dependent appends (see PseudoAlignment idea
-below) wherever possible, since sibling_annex_stats's reduction (plain
-counts) is already order-independent and needs nothing here.
Superseded 2026-08-15 by the --subsample/--shannon design below,
-which sidesteps the accumulator redesign for now: bounding the number of
-families actually resolved per layer (via sampling) keeps per-layer
-resolution volume small enough that the batch-density problem above stops
-mattering in practice for these two consumers. The accumulator redesign
-remains relevant for a future unsampled, full-index run, but is not
-required to ship --subsample/--shannon.
The reference run (phyloskims_sal_vac-scale bacterial test set,
-iqtree.fasta) produced a dense alignment for 13 genomes × 383,965 sites
-(4.8 MB) — trivially small. The in-progress plant index is expected to
-carry on the order of 9 billion minorant families; a dense byte-per-cell
-alignment at that column count is unbuildable regardless of genome count
-(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000–
-8,000 genomes on a large machine, which makes the per-cell cost dominant in
-the other dimension too. Pruning the retained family set before
-materializing anything is mandatory, not an optimization.
Already free: family_size() < 2 (no sibling variant registered at
-all) is a zero-cost structural filter, read directly off FamilyMask bits,
-already applied in snp_pseudo_alignment. Insufficient alone — per the
-~80-85% mono-family estimate from earlier discussion, this only brings 9
-billion down to roughly 1.3-1.8 billion, still unusable.
Entropy definition — settled 2026-08-15, correcting an earlier wrong
-turn. The project does not encode families as IUPAC ambiguity codes
-interpreted the classical way (Fitch-parsimony subset-compatibility, or
-ML's "one true state, uncertain which"); see
-docmd/theory/evolutionary_distances.md ("Why the IUPAC/DNA encoding used
-for the first --snp test was wrong") and the Sankoff resolution that
-followed it. The real model is a genuine 16-state alphabet (the powerset of
-{A,C,G,T}, ∅ included as a real state) scored with a calibrated
-pairwise cost matrix (obikphylo::cardcomp::pairwise_cost_matrix,
-cmd/phylo/sankoff.rs), not a compatibility/subset relation between
-states. Under that model, each of the 16 states — including multi-bit ones
-like AC — is a first-class, independently-costed state, not an
-uncertainty encoding of a single true base. So: entropy over the 15
-non-empty states (∅ excluded, matching the earlier decision to exclude
-genomes where the family is absent) is the correct informativeness
-measure for this project — not a 4-symbol reduction, which would discard
-exactly the cardinality/composition information the calibrated cost matrix
-is built to exploit.
--subsample / --shannon — sampling strategy (decided 2026-08-15)Goal: make both the pseudo-alignment (--snp) and a Shannon-entropy
-diagnostic usable at any index scale, from the 13-genome bacterial
-reference run up to the 9-billion-family plant index, without requiring the
-batched-accumulator redesign above.
--subsample N (integer, families to retain): bounds the pseudo-alignment
-to N minorant families, sampled proportionally per layer among
-non-monomorphic minorants (family_size >= 2) — this sidesteps the need
-for a true global reservoir merge across layers while still approximating a
-uniform sample over the whole index, and directly answers the earlier open
-question of global-vs-per-layer selection scope.
Three passes, in order:
-sibling_family_size_histogram, extended to report a
- per-layer breakdown rather than one index-wide aggregate): for each
- layer, count_layer = number of non-monomorphic minorants. Gives
- total_count = Σ count_layer.N_layer = round(N ×
- count_layer / total_count). Since N_layer is a proportion of
- count_layer, it can never exceed it as long as N <= total_count — the
- one edge case is total_count <= N, in which case sampling is skipped
- entirely and every non-monomorphic minorant of every layer is kept
- (no reservoir needed, N was never a real constraint). Otherwise:
- Algorithm-R reservoir sampling over the layer's non-monomorphic minorant
- indices, producing N_layer iteration-order indices directly, no
- intermediate full list ever materialized.scan_layer_families engine, unchanged): re-scan the layer, generating
- and resolving cross-partition queries only for the indices selected
- in step 2 (cheap membership test against a small per-layer index set) —
- this is what keeps --subsample cheap even on an unsampled-scale index,
- since the cross-partition resolution volume is bounded by N, not by
- the layer's true size.Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir
-sampling would waste step-3's expensive resolution work on candidates later
-evicted by the reservoir. Step 1 must fully complete (every layer) before
-step 2 can start for any layer, since total_count is a global quantity.
--shannon (no argument): emits a CSV of per-family Shannon entropy
-(15 non-empty states, ∅/absent genomes excluded from the denominator, per
-the settled definition above). Independent of --subsample — entropy is
-computed and written per family as soon as its genome_mask resolves,
-O(1) memory per family, so it streams fine even unsampled at full index
-scale (a time cost, not a memory one). Combined with --subsample N, it
-delivers the original exploratory diagnostic (e.g. --subsample 1000000
---shannon) directly from this general machinery, rather than a
-purpose-built one-off script.
Validated end-to-end (2026-08-15) against real data: --sibling-hist on
-phyloskims_sal_vac (91 real genomes, k=31/m=11, 256 partitions × 2
-layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238
-total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier
-guess, corrected here). --subsample/--shannon on the smaller 20-genome
-bacterial reference (benchmark/global_index_presence) produced a sample
-size within rounding of the request (99,742/100,000) and a
-[0.8,1.2)-bucket share (40.7%) matching the full unsampled population
-(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a
-real bug in reservoir_sample_layer (see next section), now fixed.
Bug found and fixed (2026-08-15): family_idx numbering mismatch.
-scan_layer_families's family_idx counts every minorant of a layer
-(monomorphic ones included, since iter_minorants_batch filters only on
-is_minorant()), not the raw annex slot (SiblingAnnex::get(slot) spans
-every k-mer, minorant or not) and not a counter over non-monomorphic
-minorants alone. subsample.rs's reservoir_sample_layer originally
-stored raw slot numbers in its HashSet<usize> selection, which drifts
-away from family_idx as soon as any monomorphic minorant is seen —
-i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by
-tracking two separate counters: family_idx (every minorant, matching
-scan_layer_families) and seen (non-monomorphic minorants only, what
-Algorithm R actually samples over) — only family_idx values are ever
-stored in the selection set. The existing unit test never exercised this
-(its fixture has exactly one non-monomorphic family, always hitting the
-"keep everything" shortcut) — a stronger fixture with several interleaved
-monomorphic/non-monomorphic families would be needed to catch a regression
-here automatically; not yet written.
Motivation: on real data (bacterial reference, full unsampled run), only
-~5‰ of non-monomorphic minorants fall in the [0.5, 1.5] bit band judged
-phylogenetically interesting (entropy too low = uninformative near-invariant
-site; too high = saturated/noisy, see family_entropy's 15-state
-discussion) — roughly 1 in 10,000 minorants overall. Computing exact
-entropy for every candidate just to discard 99.99% of them is wasteful at
-the full 9-billion scale.
The idea: PersistentBitMatrix::col_view(c) gives a genome's whole
-presence column as a BitSliceView (sequential, no MPHF, no cross-partition
-routing — purely local to one layer's own matrix). Accumulating
-TempCompactIntVecBuilder::inc_present(col) (already exists in
-obicompactvec/src/builder.rs:121, along with add/min/max/diff on
-IntSliceView — no new low-level API needed) over every column of a layer
-produces coverage[slot]: how many genomes carry each exact k-mer, in one
-sequential per-layer pass, entirely decoupled from family/sibling
-structure. Persisted once per layer, a family's members' coverage could
-then be looked up via a plain local MPHF index() (cheap) instead of a
-full cross-partition presence resolution (find_presence_batch) — the
-expensive part today is specifically the cross-partition/cross-layer
-routing to a sibling's own matrix, not the bit-reading itself, and
-coverage[slot] sidesteps that routing entirely by moving the cost into a
-one-time, purely local, embarrassingly-parallel build step.
Why not implemented: coverage[slot] is a per-member marginal —
-summing members' coverages to approximate a family's entropy silently
-assumes no genome carries more than one member at once. It cannot
-represent or detect joint co-occurrence (a genome carrying both A and
-C at once, i.e. a combined 15-symbol state) at all, which is exactly the
-phenomenon family_entropy's 15-state definition exists to capture (see
-CardinalityTally/cardinality_transition_probs, the project's own
-existing machinery for this same co-occurrence structure, built for the
-Sankoff matrix calibration). A family that is in truth uniformly AC
-across every carrying genome would look like a well-balanced 2-state split
-under the marginal approximation (entropy ≈ 1) while its true 15-state
-entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the
-"saturated, uninformative" tail this pre-filter would need to catch,
-undermining the point. It stays plausible as a coarse filter for the
-low tail only (a dominant single member's marginal share reliably
-predicts low true entropy too), but not as a stand-in for the high tail —
-not pursued further for now.
--free-loss/--tnt pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below)Measured on phyloskims_sal_vac (91 genomes): obikmer phylo --subsample 500000
---free-loss --tnt logs four sequential stages —
-raw_snp_distance (1413s), base_pair_tally (1456s),
-cardinality_tally (2078s), snp_pseudo_alignment (143s). Reading the
-code (obikmer/src/cmd/phylo/mod.rs:210-242,
-obikphylo/src/siblings/distance.rs, cardinality.rs, alignment.rs)
-surfaced two compounding problems, not one:
KmerPartition/PartitionCache and calling scan_family_pairs/
- scan_layer_families independently — nothing computed in one stage is
- reused by another. base_pair_tally is explicitly documented
- (distance.rs:138-143) as a second full pass over the same
- traversal raw_snp_distance already did, needed only because
- raw_snp_distance doesn't keep the resolved bases, only aggregate
- counts. cardinality_tally and snp_pseudo_alignment are each a
- third and fourth independent full pass. Per the module's own earlier
- profiling note (family_scan.rs:26-28, cited already above), this
- traversal is page-fault/mmap-bound, not compute-bound — the ~10-12%
- CPU efficiency ("contention" status) observed on these three slow
- stages is consistent with I/O stalls scaled by repeated full scans,
- not lock contention (there are no Mutex/RwLock anywhere in
- siblings/*.rs; shared writes use per-slot AtomicU8::fetch_or).raw_snp_distance and cardinality_tally don't honor
- --subsample at all — both call scan_layer_families with
- Selection::All hardcoded, and args.subsample isn't even threaded
- into their function signatures (mod.rs:212, mod.rs:226). Only
- snp_pseudo_alignment(args.subsample) builds a real reservoir-sampled
- Selection::Some(set) (via compute_selections, alignment.rs:89).
- So today, --subsample 500000 only bounds the pseudo-alignment step —
- the SNP-distance matrix, the Sankoff base-pair calibration, and the
- cardinality histogram are always computed over the full, unsampled
- index regardless of the flag. This is not merely "different subsamples
- per stage" (which would already be a problem worth fixing) — it's that
- three of the four stages never subsample, which explains most of the
- ~10x runtime gap against snp_pseudo_alignment on its own.Decided requirement: all four stages consume one shared selection, -computed once, not each stage either scanning everything or drawing its -own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair -calibration, the cardinality histogram, and the pseudo-alignment all -describe the same set of families — the calibration and the alignment it -calibrates are now guaranteed to agree on which sites exist.
-Correction to the "single pass" framing above: base_pair_tally/
-cardinality_tally both need raw_snp_distance's complete aggregate
-SNP/shared counts before they can derive included[i,j] (the
-ratio_ceiling filter) — a genuine sequential dependency (included
-can't be known until every pair's aggregate count is final), not an
-artifact of the old code's structure. So the fix is two passes over
-the shared selection, not one: pass A computes the aggregate counts (and
-derives included); pass B fuses base_pair_tally + cardinality_tally
-+ the pseudo-alignment (mutually independent once included is known)
-into a single scan. Still a 4→2 reduction, and — per the clarification
-that settled this — pass A itself now runs over the same shared
-selection pass B uses (not the full unsampled index): "les ratios, on
-les fait sur les sites sélectionnés, c'est tout, les autres sites
-n'existent pas" — once a selection is chosen, both passes are bounded by
-it, so on a real --subsample/--entropy run pass A is cheap too, not
-just pass B.
Entropy-biased selection's own resolution to the "forward-looking -complication" (entropy must be known before selection, but selection -happens during the same scan that would resolve it): see "Entropy-biased -selection" below — resolved via a persisted per-layer entropy annex, not -by restructuring the scan into an inline pre-pass.
-Refines the "forward-looking complication" above with a concrete
-mechanism. Instead of a hard [low, high] entropy band (or any other
-exact cutoff) deciding which non-monomorphic minorant families are
-eligible, selection is weighted by an unnormalized Gaussian kernel
-centered on a target entropy: w(entropy) = exp(-(entropy - μ)² / (2σ²))
-— deliberately not the normalized Gaussian density (which would peak
-below 1 and complicate the "probability" reading) — this kernel form
-equals 1 exactly at entropy = μ and decays smoothly to 0 away from it,
-so it reads directly as an acceptance weight: the further a family's
-entropy from the target, the less likely it is picked, with no hard
-in/out boundary — a few "bad" sites can still get in, by design. μ
-(default ~1.0) and σ (default ~0.5) are meant to be user-tunable.
Mechanism: joint probability, not a weighted reservoir. Not
-Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex
-proposal, superseded before implementation) — instead, a single
-independent accept/reject draw per qualifying candidate: draw
-u ~ Uniform(0,1), accept iff u < p₀ · w(entropy). p₀ is a single
-index-wide rate, N / total_count (total_count = the sum of
-non_monomorphic_counts across every layer, N = --subsample's
-target), applied identically at every layer — this alone already gives
-each layer its proportional share (the same effect the old uniform
-reservoir's explicit per-layer n_layer = N · count_layer / total_count
-computation achieved, but without needing to compute it: applying one
-rate uniformly is mathematically the same as proportioning per layer).
-p₀ = 1.0 when there is no --subsample at all — --entropy alone is a
-pure soft entropy filter over the whole index, no size target. Properties:
-(1) strictly generalizes the existing uniform sampler — with σ large
-enough that w ≈ 1 everywhere, this reduces to the old uniform N/total_count
-draw; (2) yields "approximately N", not exactly N — expected accepted
-count is N · mean(w), always ≤ N — an intentional relaxation, matching
-"sous-échantillonnage à environ n" rather than the old reservoir's exact-N
-guarantee; (3) one streaming pass, one random draw per candidate, no
-reservoir state.
Resolving "entropy must be known before selection, but selection
-happens during the same resolving scan": solved with a persisted,
-per-layer entropy annex (obikphylo/src/siblings/entropy_annex.rs,
-EntropyAnnex/EntropyAnnexBuilder), not by restructuring the scan.
-Mirrors SiblingAnnex's mmap-backed, read-only-after-build convention,
-but indexed by family_idx (every minorant of the layer, monomorphic
-included — the same numbering Selection/scan_layer_families already
-use), one f32 entropy15 value per entry, -1.0 sentinel for monomorphic/
-not-yet-computed. First use of --entropy/--entropy-sd on an index
-pays a one-time cost (ensure_entropy_annexes in entropy.rs) — every
-later run (any μ/σ, any command) reads the file positionally, no
-re-scan, restoring the usual Selection::Some "skip resolving excluded
-families" speedup that a naive "weigh during the resolving scan" design
-would have permanently forfeited.
Bug found and fixed (2026-08-15): ensure_entropy_annexes scanned with
-Selection::All instead of bounding to non-monomorphic minorants.
-Monomorphism (family_size() < 2) is knowable directly from the annex
-bits alone, no per-genome resolution needed — but the original
-implementation called scan_layer_families with Selection::All
-anyway, so fill_sub_matrix_carries (the expensive per-genome
-resolution) ran for every minorant, ~98% of which are monomorphic
-(measured elsewhere in this doc) and had their genome_mask immediately
-discarded once the callback checked family_size() < 2. Fixed by adding
-[subsample::non_monomorphic_selection_layer] — a cheap, annex-only,
-non-sampling pass (same shape as reservoir_sample_layer, but keeping
-every non-monomorphic minorant's family_idx instead of a bounded
-reservoir) — and passing Selection::Some(&eligible) instead of
-Selection::All, so the expensive resolution now runs only for the ~2%
-of minorants that can actually produce a real entropy value. A
-debug_assert!(mask.family_size() >= 2, ...) inside the
-scan_layer_families callback guards the invariant.
Resolved: the existing hard "non-monomorphic minorant" eligibility -filter stays a hard gate upstream of the Gaussian weighting — only -qualifying families ever get a stored entropy value or a weighted draw.
-CLI, implemented: two phylo options, --entropy <μ> and
---entropy-sd <σ> (obikmer/src/cmd/phylo/args.rs). The entropic filter
-activates as soon as either is given (mod.rs, computed once into an
-Option<EntropyBias> threaded through --snp/--family-overlap/
---shannon/the fused sankoff pipeline below). If active but one or both
-are unset, defaults are μ = 1.0, σ = 0.5.
--shannon's CSV carries both entropy15 (family_entropy — the settled
-15-non-empty-state definition, see above) and entropy4 (family_entropy_4
-— plain nucleotide reduction), computed from the same already-resolved
-genome_mask, not from the marginal approximation above. A genome carrying
-several bases at once contributes to each base's count (counted once per
-base present, not fractionally split, not folded into one combined state)
-— a genome polymorphic for the family is present at more than one base by
-construction, so it is expected to count more than once; the denominator is
-the total base-occurrence count, not the genome count (the two coincide
-only when no genome carries more than one base). Kept side by side
-specifically to measure, on real data, how much the two diverge — not yet
-analyzed.
PersistentSparseBitMatrix — implemented and measured (2026-08-15)A row-major (k-mer-major), deduplicated sparse alternative to
-obicompactvec::PersistentBitMatrix, motivated by the same sparsity that
-drove --subsample/--shannon above, but pursued as a foundational
-storage-layer change rather than an index-level workaround. Full design
-history, rationale, and rejected alternatives (external Elias-Fano crates,
-cacheline-ef, a single unsplit dict_id array) are in the dedicated
-implementation plan (vivid-mapping-tiger.md at the time of writing — the
-content below is the durable summary, not a pointer to a session-scoped
-file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi,
-"Compact Data Structures for Collections of Sets" (2025,
-biblio/Alanko et al. - Compact Data Structures for Collections of
-Sets.pdf) — this design implements only their exact-duplicate special
-case (a plain dedup dictionary), not their full subset-containment
-hierarchy.
Design: four on-disk components, each mmap-backed, built once per
-layer (matching how the rest of the build pipeline already works — never
-the whole multi-billion-row index at once): an is_multi rank-capable
-flag per row (singleton vs. multi-genome), a fixed-bit-width array for
-singleton rows (genome index directly, ceil(log2(n_cols)) bits), a
-separate fixed-bit-width array for multi-genome rows (dict_id,
-ceil(log2(n_distinct_multi_sets)) bits — kept apart from the singleton
-array specifically because n_distinct_multi_sets can be large in
-absolute terms even when multi-genome rows are a small fraction of all
-rows, and a single shared array would force every row, singletons
-included, to pay the wider width), and a deduplicated dictionary of
-distinct multi-genome sets (Elias-Fano-encoded byte offsets + a
-varint-encoded values blob). New low-level primitives added to
-obicompactvec to build this: PersistentFixedIntVec (arbitrary,
-runtime-parameterized bit width, width 0 included — needed once a real
-bug surfaced, see below), PersistentRankSelectBitVec (rank1/rank0/select1
-on top of the crate's existing count_ones, using
-common_traits::SelectInWord), EliasFano (composes the two). A new
-BinaryMatrix trait (n, n_cols, row/fill_row, fill_sub_matrix,
-count_ones) unifies dense and sparse at the one call site that needs
-both interchangeably (obikphylo::siblings::cache::Mat) — column-oriented
-methods (col, col_view, the partial_*_dist_matrix family) stay
-dense-only.
Two real bugs caught by tests, not by inspection: (1) EliasFano::open
-re-derived its low-bits width from the persisted low-vector file's own
-width byte; the zero-width case was built with a dummy 1-bit placeholder
-(the builder rejected true width 0), so every reopened value silently
-doubled. Fixed by making PersistentFixedIntVec genuinely support width 0
-(no storage, get always 0) instead of working around the limitation in
-EliasFano. (2) An empty row (cardinality 0 — not expected on a real
-built index, but not guarded against either) was recorded as a singleton
-at genome 0, indistinguishable on read-back from a real singleton at
-genome 0. Fixed by routing cardinality-0 rows through the dictionary path
-(a genuine empty entry) instead of the singleton shortcut. Both caught by
-obicompactvec's test suite (142 tests, including disk-reopen round-trips
-that drop every builder/mmap before reopening fresh), not by manual
-review — worth remembering next time a "this edge case can't happen in
-practice" shortcut is tempting.
Measured on real data (layer_1 of phyloskims_sal_vac's
-part_00018, 30,246,774 rows, 91 genomes — #[ignore]d benchmarks in
-obikphylo/src/siblings/tests.rs):
| - | dense | -sparse | -ratio | -
|---|---|---|---|
| on-disk size | -328.1MB | -43.7MB | -7.5x smaller | -
| build time / peak RSS | -— | -4.26s / 628MB | -(per-layer, in-memory construction — comfortable) | -
| row access, sequential (2M reads) | -43ns/row | -32ns/row | -sparse faster (smaller structure, better cache fit) | -
| row access, random (2M reads) | -409ns/row | -85ns/row | -sparse ~4.8x faster (the real --shannon/family-lookup shape) |
-
| column access, one full column (30.2M rows) | -11.5ms | -993ms | -sparse 86x slower (no native column method — every read decodes a full row to keep one bit) | -
The row-access wins (both directions) weren't the design's stated goal — -compactness was — but turn out real: dense's genome-major layout scatters -a single row read across a much bigger file, which costs more than -sparse's rank/select/varint decode once the file is this much smaller. -The column-access cost is the flip side of the same layout choice, and is -exactly what the next item below exists to fix.
-Next, not yet planned: rewrite partial_jaccard_dist_matrix/
-partial_hamming_dist_matrix/etc. (obicompactvec/src/bitmatrix/pairwise.rs)
-as a row-major co-occurrence accumulation (O(Σ_rows k²), per-row
-increments into an NxN genome-pair counter — the known alternative to
-today's column-fold, plausibly cheaper on data this sparse, not just a
-fallback) so obikindex's --metric/distance-matrix path can use the
-sparse type without the measured 86x column-access penalty. Needs its own
-design pass (in particular how it plugs into the BitPartials/
-ColumnWeights traits so both matrix types keep serving --metric)
-before implementation — not just "port the loop", a genuinely different
-algorithm.
Also still deferred, unchanged from the implementation plan: full Alanko
-et al. subset-hierarchy compression (only the exact-duplicate special case
-is built), a sparse PersistentCompactIntMatrix (count matrices), and
-BRWT-style column-correlation exploitation.
pack and the sibling-annex build path (2026-08-15)PersistentSparseBitMatrix went from a validated but unused type to a
-real, selectable on-disk format:
extensions::SiblingExt (CLI-facing trait on IndexCache) covers only the
+operations with no cross-invocation sharing concern: build_sibling_annex,
+shannon_entropy_csv, sibling_family_size_histogram/sibling_annex_stats,
+family_overlap. The sampled SNP path (sampled_scan, AlignmentReducer,
+PairwiseReducer, distance_matrix, SankoffBundle) is not wrapped in
+trait methods — the CLI (obikmer::cmd::phylo) composes them directly,
+because it alone knows the union of reducers a given invocation needs
+(--pseudo-alignment/Sankoff group/snp-* --distance can combine in one
+call). A per-output trait method would have to either duplicate that
+composition logic or reintroduce the "two independent scans" problem this
+design replaced.
Layer<D>: obikindex::layer::Layer<D>'s presence-only methods
- (n_cols, sub_matrix, fill_sub_matrix) are generic over any
- D: LayerData<Item = Box<[bool]>> + BinaryMatrix, not hardcoded to
- PersistentBitMatrix — PersistentSparseBitMatrix implements
- LayerData (open/read) the same way. find_slot/index_batch were
- already generic over any D: LayerData, so they needed no change.
- Verified by obikindex::layer's
- presence_layer_generic_over_sparse_matches_dense test: build a dense
- presence layer, convert it to sparse via build_from_dense, open both
- as Layer<PersistentBitMatrix>/Layer<PersistentSparseBitMatrix> on
- the same directory, assert n_cols/sub_matrix/find_slot agree.
- (This test must stay at k=4 with mutually non-colliding canonical
- 4-mers across its input sequences — K/M are process-wide
- AtomicUsizes in test builds, not thread-local, so a test using a
- different k races every other test in the same crate binary; a k=11
- version of this test passed alone but failed under the full
- obikindex::layer suite for exactly that reason before being fixed.)obikphylo::siblings::cache::Mat gained a third variant,
- SparsePresence(Layer<PersistentSparseBitMatrix>), alongside Count
- and Presence — every method (find_slot, index_batch,
- iter_minorants_batch, n_cols, fill_sub_matrix_carries) dispatches
- to it identically to Presence, since both go through the same generic
- Layer<D> code. PartitionCache::build picks the variant per layer by
- checking for presence/is_multi.prsb (the sparse format's own marker
- file, see the design section above) before falling back to the dense
- open path.pack_sparse_bit_matrix (new, obicompactvec::bitmatrix::sparse):
- pack --sparse's entry point. Idempotent (checks is_multi.prsb
- first); packs to dense matrix.pbmx first if that hasn't happened yet
- (the dense→sparse transpose needs random row access, which only the
- packed/columnar dense forms give), then build_from_denses the sparse
- form into the same directory and deletes matrix.pbmx — old-format
- files are removed only after the new format is fully written, mirroring
- pack_bit_matrix's own crash-safety convention.obikmer pack --sparse threads a sparse: bool through
- KmerIndex::pack_matrices (all other call sites — select, merge,
- finalize_indexed — pass false, unchanged dense behaviour). Count
- matrices are untouched by --sparse (no sparse PersistentCompactIntMatrix
- — see "still deferred" above).obikphylo::siblings::tests::
- sibling_annex_works_after_pack_sparse builds a two-genome index, packs
- it --sparse, asserts is_multi.prsb exists, then runs
- build_sibling_annex and checks the resulting FamilyMasks match the
- dense-path test (sibling_annex_one_sibling_each) exactly — proves the
- sparse format round-trips through the real build pipeline
- (PartitionCache sparse-detection included), not just the
- obicompactvec/obikindex::layer unit layers below it.Full workspace cargo test (all crates, unit + doc tests) green after
-this change.
Replaces the "four independent scans" problem above and implements -"Entropy-biased selection" above, end to end:
-SankoffBundleExt::sankoff_bundle (new,
- obikphylo/src/siblings/sankoff_bundle.rs) — the --sankoff/--tnt/
- --phyg/--iqtree block in obikmer/src/cmd/phylo/mod.rs now calls
- this once instead of three separate raw_snp_distance/
- base_pair_tally/cardinality_tally calls. One PartitionCache, one
- shared (possibly subsampled/entropy-biased) selection computed once via
- compute_selections, two scans over it: pass A (aggregate SNP/shared
- counts, --exclude-genome zeroing, then included[i,j]), pass B
- (base_pair_tally + cardinality_tally + the pseudo-alignment, fused
- into one scan_layer_families call per layer, all three read off the
- same resolved genome_mask). snp_pseudo_alignment/
- shannon_entropy_csv (still used standalone by --snp/
- --family-overlap/--shannon) both gained an entropy_bias parameter
- too, so entropy-biased selection isn't sankoff-specific.sankoff_bundle_matches_old_separate_calls
- (obikphylo/src/siblings/tests.rs) asserts sankoff_bundle's four
- outputs are bit-identical to calling the old, separate
- raw_snp_distance/base_pair_tally/cardinality_tally/
- snp_pseudo_alignment on the same fixture with no subsample — the
- fusion is a performance change, not a behavior change.EntropyAnnex/EntropyAnnexBuilder (new,
- obikphylo/src/siblings/entropy_annex.rs) and ensure_entropy_annexes
- (entropy.rs) implement the persisted-entropy mechanism from
- "Entropy-biased selection" above. entropy_annex_builds_on_demand_and_biases_selection
- (tests.rs) proves, on a fixture with one known-entropy family: the
- annex file doesn't exist before any entropy-biased call; compute_selections
- builds it on first use; μ set to the family's exact entropy with
- p₀ = 1.0 selects it deterministically (u < 1.0 always, for
- u ∈ [0,1)); μ set far away with tiny σ deterministically excludes
- it (w underflows to exactly 0.0); the persisted value matches
- --shannon's own family_entropy computation to 1e-6.EntropyBias (pub, obikphylo::siblings::EntropyBias { mu, sigma })
- is the one new public type threading --entropy/--entropy-sd through
- every Option<EntropyBias>-accepting method — resolved once in
- obikmer/src/cmd/phylo/mod.rs from args.entropy/args.entropy_sd
- (activation: either given; defaults 1.0/0.5 for whichever is unset).Full workspace cargo test green after this change (167 unit tests in
-obicompactvec+obikindex::layer+obikphylo alone, plus every other
-crate's suite, no regressions).
Still open, not part of this change (per "Correction to the 'single
-pass' framing" above): --raw-snp-distance/--raw-snp-counts (the
-standalone diagnostic flags, not the --sankoff pipeline) still always
-scan the full unsampled index — never threaded --subsample/--entropy,
-out of scope here since the reported problem was specifically about the
---sankoff/--tnt pipeline's redundant/inconsistent scans, not these
-two standalone flags.
query never benefits from sparse row-major access (found 2026-08-19, not implemented)Benchmarked obikmer query against global_index_presence (dense-packed)
-vs. global_index_presence_sparse (pack --sparse), 100k simulated reads
-× 2 specimens (benchmark/, see
-benchmark_query_testing.md).
-Correctness: 0 mismatches — sparse and dense return bit-identical query
-results. Performance: sparse consistently slower than dense (~30-50%,
-reproducible across two runs with warm disk cache), the opposite of
-pack --sparse's stated intent ("faster for single-row access... like
-query").
Root cause, read from source, not measured in isolation:
-KmerPartition::query_partition_with (obikindex/src/partition/query_layer.rs:155-220)
-is architecturally column-major: stage 2 walks for g in 0..n_cols { for
-slot in hit_slots { layer.col_value(g, slot) } }, documented (correctly)
-as the right locality strategy for the packed/columnar formats, where
-col_value → PersistentBitMatrix::get is a genuine O(1) mmap'd column
-read (persistent.rs:110-113).
For Self::Sparse, that same get(c, slot) (persistent.rs:114-118)
-allocates a full n_cols-wide buffer and calls fill_row — materializing
-the entire row — just to return one cell. Called from inside the
-column-major double loop, this reconstructs the same row once per genome
-column touched: O(hits × n_cols) full-row rebuilds instead of O(hits).
-PersistentSparseBitMatrix's own native row-major decode
-(for_each_genome_in_row, sparse.rs:164-177, used correctly by its own
-row/fill_row/fill_sub_matrix) is never reached from the query path
-at all.
fill_sub_matrix (the existing BinaryMatrix trait primitive,
-traits.rs:13-37) is not the right replacement for query either, even
-once its own dispatch bug is fixed (see next section) — its output shape
-is inherently column-dense: out[col] gets an entry for every column,
-including columns with zero hits among the requested slots. On real
-sparse data (a hit typically touching a handful of genomes out of dozens)
-that's still O(n_cols) output regardless of true sparsity. What query
-actually wants is the sparse triple stream (slot, col, value) it already
-consumes as QueryHit::Value — not a materialized sub-matrix.
Proposed primitive (design only, not implemented — explicit ask: keep -count matrices not excluded, even though effort right now is -presence/absence only):
-Not a closure-driven for_each — a real Iterator, one concrete struct
-per matrix format, so the traversal state (current position in the sorted
-slot list, current column, permutation, sparse-row decode cursor…) lives
-in named struct fields instead of being threaded implicitly through
-recursion or a captured closure. RPITIT (stable since 1.75, and this
-workspace is edition 2024) means the trait method can return it without
-naming or boxing the concrete type:
/// Yields every nonzero cell among `slots`, in implementation-defined order.
-fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator<Item = (usize, usize, u32)> + 'a;
-// item: (idx into `slots`, col, value)
-This is the one foundational primitive per format — both fill_sub_matrix
-and a for_each-style callback become trivial, free consumers of it
-(.for_each(f) is already Iterator::for_each from std; fill_sub_matrix
-becomes "drain the iterator, scatter into out[][]"), instead of two
-independently-maintained traversals that can silently diverge (see the bug
-below — this is exactly how it happened).
PersistentSparseBitMatrix: the struct is nearly free to write —
- it wraps the existing (currently private) for_each_genome_in_row
- per-row decode, advancing to the next slots entry on exhaustion. O(Σ
- row nnz), zero n_cols-wide allocation.PersistentBitMatrix::{Packed,Columnar}: revised — cheaper than
- first thought, by reusing the same split already used for
- fill_matrix's own implementation instead of hand-writing a resumable
- state machine at the matrix level. The base-vector layer
- (BitSliceView, views.rs) already separates the two concerns:
- fill_batch_sorted (views.rs:55-60, sorted-slot batch lookup) and a
- genuine per-bit Iterator (BitSliceIter, views.rs:94+) sit side by
- side there, one level below the matrix. Adding a
- "positions among sorted_slots where the bit is set" iterator at that
- same vector level is a filter over the existing get() — no new
- state machine, since std::iter::Filter already is one, generated by
- the compiler:
- fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator<Item = usize> + 's {
- sorted_slots.iter().copied().filter(move |&slot| self.get(slot))
-}
-nonzero_iter then composes these per column with
- flat_map over 0..n_cols (each column's hits, tagged with c,
- slot mapped back through the sort permutation fill_batch/
- fill_batch_sorted already carry) — again a combinator chain, not a
- hand-rolled struct. Same algorithm, same mmap/sort locality as today's
- fill_sub_matrix; just assembled from std iterator adaptors instead
- of a loop body writing into a buffer, mirroring the vector/matrix split
- the codebase already uses for fill_batch_sorted rather than
- introducing a new shape.Implicit: trivial (slots.iter().map(|&i| (i, 0, 1)), one column,
- always present).PersistentCompactIntMatrix (counts): same treatment as
- Packed/Columnar — no sparse count format exists yet ("Explicitly
- deferred" per traits.rs:9-12), so no native low-effort case the way
- Sparse has one, but not excluded either: the iterator's Item is
- already (usize, usize, u32), not bool, specifically so presence
- (0/1) and counts (arbitrary u32) share one primitive instead of a
- bool/u32 split forcing counts out of the design. Ready for a native
- sparse-count struct later without a signature change.Would let query_partition_with's stage 2 collapse to one
-for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) }
-per layer, format-agnostic, each backend's struct deciding the actual
-traversal.
This also closes the existing dispatch bug for free, by construction:
-PersistentBitMatrix::fill_sub_matrix (persistent.rs:190-215, the enum
-wrapper backing BinaryMatrix's trait impl) today does not delegate to
-PersistentSparseBitMatrix::fill_sub_matrix for Self::Sparse — it
-reimplements the same naive per-(column, slot) fill_row_bool loop
-instead, bypassing the efficient native method one file over
-(sparse.rs:249-258). obikphylo::siblings::cache::Mat
-(cache.rs:138-145) independently built its own parallel enum wrapper
-that dispatches correctly — evidence this was worked around rather than
-fixed at the source: two hand-written traversals for the same format,
-free to drift apart, and they did. If fill_sub_matrix itself is
-rewritten as "drain nonzero_iter, scatter into out[][]", there is only
-one traversal per format left to get right — the bug class doesn't just
-get fixed once, it stops being possible to reintroduce.
Built as designed above, with one deviation from the original sketch:
-nonzero_iter ended up Box<dyn Iterator<...>>, not a bare impl
-Iterator, because Columnar/Packed/Sparse/Implicit are genuinely
-different concrete types and this method isn't on a trait (kept off
-BinaryMatrix deliberately — that trait is used as dyn BinaryMatrix in
-tests/sparse.rs, and RPITIT methods aren't dyn-compatible). One Box
-per nonzero_iter call, not per cell — negligible next to what it
-replaces.
BitSliceView::nonzero_among_sorted / IntSliceView::nonzero_among_sorted
- (obicompactvec/src/views.rs): the vector-level filter/filter_map
- primitive, exactly as sketched — no new state machine, std's own.PersistentSparseBitMatrix::nonzero_iter (bitmatrix/sparse.rs): native,
- std::iter::from_fn over one buffered row at a time via the existing
- for_each_genome_in_row — no n_cols-wide allocation, ever.PersistentBitMatrix::nonzero_iter (bitmatrix/persistent.rs): dispatches
- to the above for Sparse; for Columnar/Packed, loops columns,
- collects each column's nonzero_among_sorted hits via .extend() (not
- flat_map — a flat_map closure can't lazily return something
- borrowing its own captured sort permutation across separate calls
- without either boxing per-column or fighting the borrow checker; eager
- collection into one Vec sidesteps it, at zero cost since
- fill_sub_matrix already fully materialized anyway). Implicit trivial.PersistentBitMatrix::fill_sub_matrix and sub_matrix rewritten to
- drain nonzero_iter — the dispatch bug is gone because there is now
- only one traversal per format, not because the old one was patched.
- PersistentCompactIntMatrix::nonzero_iter added the same way (counts
- not excluded, per the earlier ask) — no native low-effort case, since no
- sparse count format exists, but on the same primitive, ready for one.KmerPartition::query_partition_with (obikindex/src/partition/query_layer.rs):
- stage 2's column-major for g { for slot { col_value } } replaced by one
- layer.nonzero_iter(&slot_list) call per layer, format-agnostic.nonzero_iter_matches_dense, nonzero_iter_matches_row, and —
- the one that actually targets the dispatch bug rather than each type's
- own correctness — enum_wrapper_dispatches_to_native_sparse (builds
- PersistentBitMatrix::Sparse(...) directly, not through open, since
- open only auto-detects Sparse from a presence/ dir layout).
- cargo test --workspace: green, no regressions.Measured: re-ran the benchmark/ query branch (100k reads × 2
-specimens, same setup as the original finding). Correctness still 0
-mismatches. The dense/sparse performance gap is gone — previously sparse
-~30-50% slower than dense, reproducibly; now within ~1-3% either way
-(7.42s dense vs 7.60s sparse for Escherichia_coli--K-12_MG1655; 5.25s vs
-5.30s for Saccharolobus_islandicus--M.16.4) — noise-level, not a
-systematic gap. pack --sparse's claimed query win isn't confirmed
-outright by this (sparse should arguably now beat dense on truly sparse
-real data, not just tie), but the pathological regression is fixed.
PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)Closes the gap flagged throughout this document ("no sparse count format
-exists yet", traits.rs:9-12's "Explicitly deferred"): obicompactvec
-already had PersistentSparseCompactIntMatrix (row-major, built on top of
-PersistentSparseBitMatrix as its "which columns are non-zero" support,
-values not deduplicated — see that struct's own doc comment), but it was
-never wired into PersistentCompactIntMatrix, the dense-dispatching enum
-every real consumer (TypedLayer<PersistentCompactIntMatrix>,
-KmerLayer::Count) actually holds. Concretely: kmer_index.rs::
-pack_matrices(sparse=true) already called pack_sparse_compact_int_matrix
-on every layer's counts/ — but PersistentCompactIntMatrix::open had no
-code path back to what that just wrote, so a Count layer became
-unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
-anyone ran pack --sparse on an index with count layers. Root cause, not a
-workaround: add the missing Sparse variant.
Enum + dispatch (intmatrix.rs): PersistentCompactIntMatrix::Sparse
- (PersistentSparseCompactIntMatrix), detected in open/detect_storage
- via a singleton_values.pciv marker (mirrors PersistentBitMatrix's own
- sparse_meta.json check), reported via storage_kind(). col/
- col_view/col_persist panic/Unsupported on Sparse, same convention
- as the bit side. sub_matrix/fill_sub_matrix and nonzero_iter
- unified the same way PersistentBitMatrix's already are (drain
- nonzero_iter, one traversal per format — see "Implemented
- (2026-08-20)" above); nonzero_iter had to become Box<dyn Iterator<...>>
- for the same reason (Columnar/Packed/Sparse are different concrete
- types). No change needed in obikindex at all — KmerLayer::Count
- already only ever holds TypedLayer<PersistentCompactIntMatrix>, so the
- enum absorbing Sparse fixes the unreadable-layer bug for free, same as
- PersistentBitMatrix::Sparse already did on the presence side.
CountPartials, non-naive (sparse_intmatrix.rs): unlike
- PersistentSparseBitMatrix's dict-driven col_weights_and_pair_counts,
- values here aren't deduplicated (two rows can share the same non-zero
- column set via the same dict_id while carrying different counts), so
- the "weight by how many rows share a dict entry" shortcut doesn't carry
- over. What does: a single row-major pass (row_major_pairwise, decodes
- each row once via for_each_cell_in_row, nests over that row's own
- co-present columns) — O(Σ k̄²) over populated rows instead of the naive
- O(n_cols² × n) column-pair rescan, same complexity class as the bit
- side minus the dict multiplicity discount. Kernels used: min(a,b)
- (bray, relfreq-bray — both vanish when either side is absent, so no
- correction needed), a·b and √(a·b) (euclidean/relfreq-euclidean and
- hellinger — these do need a correction, reconstructed from per-column
- marginals via Σ(a-b)² = Σa²+Σb²-2Σab, since (a-0)² = a² ≠ 0 unlike
- the min-based formulas). threshold_jaccard(1) shortcuts straight to
- support's own BitPartials::partial_jaccard (threshold 1 is exactly
- presence); threshold_jaccard(0) is closed-form (every u32 is ≥ 0).
Two pre-existing bugs found and fixed while wiring the threshold==1
- shortcut (bitmatrix/sparse.rs, BitPartials for
- PersistentSparseBitMatrix, present since the 2026-08-15 implementation
- above, never caught because no test compared Sparse's raw partial_*
- output against dense on real data — only the diagonal-blind
- jaccard_dist_matrix/hamming_dist_matrix finalisations were tested):
partial_jaccard's diagonal was (0, 2×col_weights[i]) instead of a
- genuine self-comparison (col_weights[i], col_weights[i]) —
- col_weights_and_pair_counts's inter never pairs a column with
- itself by construction.partial_hamming's off-diagonal formula itself was wrong: total -
- union (count of rows where neither column is present) instead of
- the actual Hamming distance col_weights[i] + col_weights[j] -
- 2×inter[i,j] (symmetric-difference size). Only coincides with the
- correct value when col_weights[i] + col_weights[j] == total, so
- small/synthetic test data could easily have hidden it.Neither surfaced through jaccard_dist_matrix/hamming_dist_matrix
- (both explicitly zero their own diagonal at finalisation, and the
- off-diagonal partial_hamming bug had gone untested against dense
- entirely) — only visible to a caller of the raw partial_* methods
- directly, which is exactly what partial_threshold_jaccard(1)'s new
- shortcut became. Fixed at the source, not patched around at the call
- site; regression test added:
- tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal.
tests::intmatrix::sparse_roundtrip_matches_columnar/
- sparse_roundtrip_from_packed (the open-dispatch fix, both build
- paths); tests::intmatrix::sparse_count_partials_match_dense (all six
- CountPartials formulas, thresholds 0/1/2/3, against Columnar on
- asymmetric-presence data — this is what caught the diagonal gap in the
- int side's own new code before it shipped, the same way it exposed the
- two pre-existing bit-side bugs above); obikindex's
- count_layer_transparently_reads_sparse_after_pack — the actual
- end-to-end regression test for the original "layer unreadable after
- pack --sparse" bug, built → packed sparse → reopened, compared against
- the pre-pack dense read. cargo test -p obicompactvec -p obikindex:
- green, no regressions (180 + 12 tests).sample_index's skip_layers parameter (checkpointed-resume support) is
+ never called with a non-zero value by any current caller — --session
+ persistence was removed pending a redesign on top of sampled_scan
+ (single shared trunk, not per-driver checkpoint plumbing). The parameter
+ stays as the seam a future resume mechanism would use.verify_presence/verify_merge_presence, which only check index
content against the .npz truth, never the query API.pack --sparse produces a presence-matrix format documented (see
- siblings.md) as faster for single-row
- access (query) and slower for column-oriented access (phylo --metric).
+ obicompactvec.md, "Sparse matrix formats") as faster
+ for single-row access (query) and slower for column-oriented access
+ (phylo --metric).
global_index_presence/ built by merge_presence.sh is always packed
dense (packing is a stage inside merge, not a separate pack
invocation) — there was no dense/sparse regression check.Column-first to match the on-disk column-major layout. fill_sub_matrix sorts slots once, then calls each column's fill_batch in turn — no redundant per-column sort. On PersistentSparseBitMatrix (k-mer-major, no column method) this degrades to a row-by-row decode; see siblings.md.
Column-first to match the on-disk column-major layout. fill_sub_matrix sorts slots once, then calls each column's fill_batch in turn — no redundant per-column sort. On PersistentSparseBitMatrix (k-mer-major, no column method) this goes through nonzero_iter instead, decoding one row at a time; see obicompactvec.md, "Sparse matrix formats".
LayeredStore<S> is a generic aggregation wrapper over Vec<S>. It propagates three traits from obicompactvec::traits up the hierarchy via blanket impls:
Sparsesparse_meta.json + PFIV/Elias-Fano component files, row-major. Int: same support files (built on PersistentSparseBitMatrix internally) plus singleton_values.pciv/multi_values.pciv/multi_offsets for the per-row, non-deduplicated valuespack --sparse; see siblings.md for the sparse-vs-dense access-pattern trade-offpack --sparse; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-offImplicit (bit only)PersistentBitMatrix::open(layer_dir) auto-detects the variant, in order: matrix.pbmx → Packed, presence/meta.json → Columnar, presence/sparse_meta.json → Sparse, layer_meta.json (no presence dir at all) → Implicit. PersistentCompactIntMatrix::open(layer_dir) mirrors the same priority order minus Implicit (there's no implicit count matrix — counts always have at least one on-disk column): matrix.pcmx → Packed, counts/meta.json → Columnar, counts/singleton_values.pciv → Sparse. col_view/col/sub_matrix panic on Sparse/Implicit where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through row/fill_row.
Unlike the bit side, PersistentSparseCompactIntMatrix's values are not deduplicated across rows — two rows can share the same non-zero column set (same dict_id in the shared support) while carrying different counts — so its CountPartials impl can't reuse the support's dict-multiplicity shortcut the way BitPartials for PersistentSparseBitMatrix does. It still avoids the naive O(n_cols² × n) column-pair scan via a single row-major pass (row_major_pairwise in sparse_intmatrix.rs), reconstructing the squared-difference formulas (euclidean/relfreq_euclidean/hellinger) from per-column marginals via Σ(a-b)² = Σa²+Σb²-2Σab — see siblings.md's "PersistentCompactIntMatrix::Sparse — implemented" entry for the full derivation.
col_view(c) returns the appropriate view directly:
// PersistentBitMatrix
pub fn col_view(&self, c: usize) -> BitSliceView<'_>
@@ -1212,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
No wrapper enums (BitColView, IntColView): the caller receives a Copy view struct immediately usable with any view method or bulk builder method.
pack_compact_int_matrix and pack_bit_matrix convert columnar → packed format.
+Sparse matrix formats
+Motivated by real k-mer-family data being extremely row-sparse (most rows —
+k-mers — present in only a handful of genome columns out of dozens to
+thousands): a row-major, deduplicated alternative to the dense column-major
+formats above, built once per layer like the rest of the pipeline (never the
+whole multi-billion-row index at once). Implements the exact-duplicate
+special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
+Structures for Collections of Sets" (2025,
+biblio/Alanko et al. - Compact Data Structures for Collections of
+Sets.pdf) — a plain dedup dictionary, not their full subset-containment
+hierarchy.
+PersistentSparseBitMatrix
+Four mmap-backed on-disk components: an is_multi rank-capable flag per row
+(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
+(genome index directly, ceil(log2(n_cols)) bits), a separate fixed-width
+array for multi-genome rows (dict_id, ceil(log2(n_distinct_multi_sets))
+bits — kept apart from the singleton array because n_distinct_multi_sets
+can be large in absolute terms even when multi-genome rows are a small
+fraction of all rows, and a shared array would force every row, singletons
+included, to pay the wider width), and a deduplicated dictionary of distinct
+multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
+blob). Supporting primitives: PersistentFixedIntVec (arbitrary,
+runtime-parameterised bit width, width 0 included — needed once an
+EliasFano reopen bug surfaced, see below), PersistentRankSelectBitVec
+(rank1/rank0/select1 on top of the crate's existing count_ones, via
+common_traits::SelectInWord), EliasFano (composes the two). BinaryMatrix
+(n, n_cols, row/fill_row, fill_sub_matrix, count_ones) is the
+trait unifying dense and sparse at the call sites that need both
+interchangeably; column-oriented methods (col, col_view, the
+partial_*_dist_matrix family) stay dense-only — Sparse/Implicit panic
+there (see the PersistentBitMatrix::open note above).
+Measured on real data (one layer, 30.2M rows, 91 genomes):
+
+
+
+
+dense
+sparse
+ratio
+
+
+
+
+on-disk size
+328.1MB
+43.7MB
+7.5x smaller
+
+
+build time / peak RSS
+—
+4.26s / 628MB
+per-layer, in-memory construction
+
+
+row access, sequential (2M reads)
+43ns/row
+32ns/row
+sparse faster (smaller structure, better cache fit)
+
+
+row access, random (2M reads)
+409ns/row
+85ns/row
+sparse ~4.8x faster — the --shannon/family-lookup access shape
+
+
+column access, one full column (30.2M rows)
+11.5ms
+993ms
+sparse 86x slower — no native column method, every read decodes a full row to keep one bit
+
+
+
+The row-access wins (compactness was the stated design goal, not speed) come
+from dense's genome-major layout scattering a single row read across a much
+bigger file — costlier than sparse's rank/select/varint decode once the file
+is this much smaller. Column access is the flip side of the same layout
+choice; nonzero_iter (below) is what removes it from the hot path that
+actually matters (query).
+Two reopen-correctness bugs, now fixed at the source: (1) EliasFano::open
+re-derived its low-bits width from the persisted low-vector's own width
+byte; the width-0 case was built with a dummy 1-bit placeholder (the
+builder used to reject true width 0), silently doubling every reopened
+value. Fixed by making PersistentFixedIntVec genuinely support width 0
+(no storage, get always 0). (2) An empty row (cardinality 0) used to be
+recorded as a singleton at genome 0, indistinguishable on read-back from a
+real singleton at genome 0. Fixed by routing cardinality-0 rows through
+the dictionary path (a genuine empty entry).
+Still deferred: full Alanko et al. subset-hierarchy compression (only the
+exact-duplicate case is built), a sparse PersistentCompactIntMatrix
+variant beyond what's described below, BRWT-style column-correlation
+exploitation, and a row-major rewrite of
+partial_jaccard_dist_matrix/partial_hamming_dist_matrix/etc. as a
+per-row co-occurrence accumulation (O(Σ_rows k̄²)) so obikindex's
+--metric distance-matrix path stops paying the column-access penalty on
+Sparse — needs its own design pass (how it plugs into
+BitPartials/ColumnWeights so both matrix types keep serving --metric),
+not just "port the loop".
+nonzero_iter — the one traversal per format
+fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a>
+(item: index into slots, column, value) is the single foundational
+per-format traversal primitive every consumer builds on — fill_sub_matrix
+and a for_each-style callback are both trivial, free consumers of it
+(.for_each(f) is Iterator::for_each; fill_sub_matrix drains the
+iterator and scatters into out[][]), instead of two independently
+hand-maintained traversals that can silently diverge. Box<dyn Iterator>,
+not a bare impl Iterator: Columnar/Packed/Sparse/Implicit are
+genuinely different concrete types and the method isn't on a
+dyn-compatible trait (BinaryMatrix is used as dyn BinaryMatrix
+elsewhere, and RPITIT methods aren't dyn-compatible) — one Box per
+nonzero_iter call, not per cell.
+Per-format implementation: PersistentSparseBitMatrix decodes natively,
+buffered one row at a time (std::iter::from_fn over the existing
+for_each_genome_in_row, no n_cols-wide allocation ever). Columnar/
+Packed loop columns, collecting each column's hits via
+BitSliceView::nonzero_among_sorted/IntSliceView::nonzero_among_sorted
+(a filter/filter_map over the vector-level get(), no new state
+machine — std's own Filter iterator already is one) into one Vec via
+.extend() (not flat_map: a flat_map closure can't lazily return
+something borrowing its own captured sort permutation across separate calls
+without either boxing per-column or fighting the borrow checker — eager
+collection sidesteps it at zero extra cost, since fill_sub_matrix already
+fully materialises anyway). Implicit is trivial (slots.iter().map(|&i| (i, 0, 1))).
+This closed a real dispatch bug by construction, not by patching it:
+PersistentBitMatrix::fill_sub_matrix/sub_matrix used to reimplement a
+naive per-(column, slot) loop instead of delegating to
+PersistentSparseBitMatrix's own efficient native method for Self::Sparse
+— two hand-written traversals for the same format, free to drift apart, and
+they had (obikphylo's own cache layer had independently built a parallel
+enum wrapper that did dispatch correctly, evidence the bug was worked
+around rather than fixed at the source). With fill_sub_matrix/sub_matrix
+rewritten to drain nonzero_iter, there is only one traversal per format
+left to get right. KmerPartition::query_partition_with
+(obikindex/src/partition/query_layer.rs) — previously column-major
+(for g { for slot { col_value } }, which for Sparse rebuilt a full row
+per single cell touched, O(hits × n_cols) instead of O(hits) — the original
+motivation for this primitive) — now issues one layer.nonzero_iter(&slot_list)
+call per layer, format-agnostic. Dense/sparse query performance parity
+confirmed on real data (previously sparse 30-50% slower than dense,
+reproducibly; now within ~1-3% either way — noise level).
+PersistentSparseCompactIntMatrix
+The count-matrix analogue, built on PersistentSparseBitMatrix as its
+"which columns are non-zero" support, values not deduplicated (unlike
+the bit side, two rows can share the same non-zero column set — same
+dict_id — while carrying different counts, so the dict-multiplicity
+shortcut BitPartials for PersistentSparseBitMatrix uses doesn't carry
+over). CountPartials here: a single row-major pass (row_major_pairwise
+in sparse_intmatrix.rs, decodes each row once via
+for_each_cell_in_row, nests over that row's own co-present columns) —
+O(Σ k̄²) over populated rows, avoiding the naive O(n_cols² × n)
+column-pair rescan. min(a,b)-based kernels (bray, relfreq_bray) need
+no correction — they vanish when either side is absent. a·b and √(a·b)
+kernels (euclidean/relfreq_euclidean/hellinger) do need one, since
+(a-0)² = a² ≠ 0 unlike the min-based formulas — reconstructed from
+per-column marginals via Σ(a-b)² = Σa²+Σb²-2Σab.
+threshold_jaccard(1) shortcuts to BitPartials::partial_jaccard
+(threshold 1 is exactly presence); threshold_jaccard(0) is closed-form
+(every u32 is ≥ 0).
+Wiring the threshold==1 shortcut surfaced two pre-existing formula bugs in
+BitPartials for PersistentSparseBitMatrix (present since that type's own
+introduction, never caught because no test had compared Sparse's raw
+partial_* output against dense on real data — only the diagonal-blind
+jaccard_dist_matrix/hamming_dist_matrix finalisations were tested),
+fixed at the source:
+
+partial_jaccard's diagonal used to be (0, 2×col_weights[i]) instead
+ of a genuine self-comparison (col_weights[i], col_weights[i]) —
+ col_weights_and_pair_counts's inter never pairs a column with itself
+ by construction, so the diagonal needs its own case.
+partial_hamming's off-diagonal formula itself was wrong: it computed
+ total - union (rows where neither column is present) instead of the
+ actual Hamming distance col_weights[i] + col_weights[j] - 2×inter[i,j]
+ (symmetric-difference size) — the two only coincide when
+ col_weights[i] + col_weights[j] == total, which small/synthetic test
+ data can easily satisfy by accident.
+
Aggregation traits (matrix level)
ColumnWeights
trait ColumnWeights: Send + Sync {
diff --git a/DevDoc/implementation/obipipeline.refs/index.html b/DevDoc/implementation/obipipeline.refs/index.html
index 46dace87..ba0cda60 100644
--- a/DevDoc/implementation/obipipeline.refs/index.html
+++ b/DevDoc/implementation/obipipeline.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/obipipeline/index.html b/DevDoc/implementation/obipipeline/index.html
index edebe6b6..8c864f3f 100644
--- a/DevDoc/implementation/obipipeline/index.html
+++ b/DevDoc/implementation/obipipeline/index.html
@@ -1293,7 +1293,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/obitaxonomy/index.html b/DevDoc/implementation/obitaxonomy/index.html
index b1886a24..a9d00b9e 100644
--- a/DevDoc/implementation/obitaxonomy/index.html
+++ b/DevDoc/implementation/obitaxonomy/index.html
@@ -1305,7 +1305,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/partition_layer_cache/index.html b/DevDoc/implementation/partition_layer_cache/index.html
index 1cce3095..07dd753c 100644
--- a/DevDoc/implementation/partition_layer_cache/index.html
+++ b/DevDoc/implementation/partition_layer_cache/index.html
@@ -1563,7 +1563,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/persistent_bit_vec.refs/index.html b/DevDoc/implementation/persistent_bit_vec.refs/index.html
index 452c7db1..bccfdd30 100644
--- a/DevDoc/implementation/persistent_bit_vec.refs/index.html
+++ b/DevDoc/implementation/persistent_bit_vec.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/persistent_bit_vec/index.html b/DevDoc/implementation/persistent_bit_vec/index.html
index 3e40d825..a8701785 100644
--- a/DevDoc/implementation/persistent_bit_vec/index.html
+++ b/DevDoc/implementation/persistent_bit_vec/index.html
@@ -1405,7 +1405,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
index 9e306acb..7d6b5deb 100644
--- a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
+++ b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/persistent_compact_int_vec/index.html b/DevDoc/implementation/persistent_compact_int_vec/index.html
index 05ea4d9f..ab5d8aa2 100644
--- a/DevDoc/implementation/persistent_compact_int_vec/index.html
+++ b/DevDoc/implementation/persistent_compact_int_vec/index.html
@@ -1388,7 +1388,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/pipeline.refs/index.html b/DevDoc/implementation/pipeline.refs/index.html
index 3ab74967..985e28c7 100644
--- a/DevDoc/implementation/pipeline.refs/index.html
+++ b/DevDoc/implementation/pipeline.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/pipeline/index.html b/DevDoc/implementation/pipeline/index.html
index a697eee7..fc08d198 100644
--- a/DevDoc/implementation/pipeline/index.html
+++ b/DevDoc/implementation/pipeline/index.html
@@ -1254,7 +1254,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/select/index.html b/DevDoc/implementation/select/index.html
index c98f84e1..c3cb4add 100644
--- a/DevDoc/implementation/select/index.html
+++ b/DevDoc/implementation/select/index.html
@@ -1382,7 +1382,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/storage.refs/index.html b/DevDoc/implementation/storage.refs/index.html
index 03bfe3b3..06d67528 100644
--- a/DevDoc/implementation/storage.refs/index.html
+++ b/DevDoc/implementation/storage.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/storage/index.html b/DevDoc/implementation/storage/index.html
index 7fd77097..ce4b99a7 100644
--- a/DevDoc/implementation/storage/index.html
+++ b/DevDoc/implementation/storage/index.html
@@ -1326,7 +1326,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/superkmer.refs/index.html b/DevDoc/implementation/superkmer.refs/index.html
index 946e4da1..6622b610 100644
--- a/DevDoc/implementation/superkmer.refs/index.html
+++ b/DevDoc/implementation/superkmer.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/superkmer/index.html b/DevDoc/implementation/superkmer/index.html
index d5267a43..6d6069fe 100644
--- a/DevDoc/implementation/superkmer/index.html
+++ b/DevDoc/implementation/superkmer/index.html
@@ -1232,7 +1232,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/unitig_evidence.refs/index.html b/DevDoc/implementation/unitig_evidence.refs/index.html
index 1dfbaa9c..4118486a 100644
--- a/DevDoc/implementation/unitig_evidence.refs/index.html
+++ b/DevDoc/implementation/unitig_evidence.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/implementation/unitig_evidence/index.html b/DevDoc/implementation/unitig_evidence/index.html
index a66adbd3..32ef44a3 100644
--- a/DevDoc/implementation/unitig_evidence/index.html
+++ b/DevDoc/implementation/unitig_evidence/index.html
@@ -1354,7 +1354,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/index.html b/DevDoc/index.html
index e6629e0a..06cc62b0 100644
--- a/DevDoc/index.html
+++ b/DevDoc/index.html
@@ -1228,7 +1228,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/index.refs/index.html b/DevDoc/index.refs/index.html
index 5f0a7f61..b229367c 100644
--- a/DevDoc/index.refs/index.html
+++ b/DevDoc/index.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/installation/index.html b/DevDoc/installation/index.html
index 42e8a762..d044a12d 100644
--- a/DevDoc/installation/index.html
+++ b/DevDoc/installation/index.html
@@ -1314,7 +1314,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/kmers.refs/index.html b/DevDoc/kmers.refs/index.html
index b12faff2..f789c1f2 100644
--- a/DevDoc/kmers.refs/index.html
+++ b/DevDoc/kmers.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/kmers/index.html b/DevDoc/kmers/index.html
index a707331c..89637ba8 100644
--- a/DevDoc/kmers/index.html
+++ b/DevDoc/kmers/index.html
@@ -1227,7 +1227,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/sitemap.xml.gz b/DevDoc/sitemap.xml.gz
index 03a78e93..15bde70b 100644
Binary files a/DevDoc/sitemap.xml.gz and b/DevDoc/sitemap.xml.gz differ
diff --git a/DevDoc/theory/encoding.refs/index.html b/DevDoc/theory/encoding.refs/index.html
index fe650220..44c46f36 100644
--- a/DevDoc/theory/encoding.refs/index.html
+++ b/DevDoc/theory/encoding.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/theory/encoding/index.html b/DevDoc/theory/encoding/index.html
index e8a2dbb1..abfa12e4 100644
--- a/DevDoc/theory/encoding/index.html
+++ b/DevDoc/theory/encoding/index.html
@@ -1199,7 +1199,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/theory/entropy.refs/index.html b/DevDoc/theory/entropy.refs/index.html
index 49c1b70d..1a9e9081 100644
--- a/DevDoc/theory/entropy.refs/index.html
+++ b/DevDoc/theory/entropy.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/theory/entropy/index.html b/DevDoc/theory/entropy/index.html
index 5e201572..f913aeca 100644
--- a/DevDoc/theory/entropy/index.html
+++ b/DevDoc/theory/entropy/index.html
@@ -1265,7 +1265,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/theory/evolutionary_distances/index.html b/DevDoc/theory/evolutionary_distances/index.html
index 77c3d6cf..57634792 100644
--- a/DevDoc/theory/evolutionary_distances/index.html
+++ b/DevDoc/theory/evolutionary_distances/index.html
@@ -932,6 +932,17 @@
+
+
+
+
+
+
+ Automatic α estimation (--gamma-shape auto)
+
+
+
+
@@ -943,6 +954,45 @@
+
+
+
+
+
+
+ α-estimation literature survey and design discussion (2026-09-11)
+
+
+
+
+
+
+
+
+
+
+ Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
+
+
+
+
+
+
@@ -1691,7 +1741,7 @@
- Sibling annex (discussion)
+ Sibling annex
@@ -2268,6 +2318,17 @@
+
+
+
+
+
+
+ Automatic α estimation (--gamma-shape auto)
+
+
+
+
@@ -2279,6 +2340,45 @@
+
+
+
+
+
+
+ α-estimation literature survey and design discussion (2026-09-11)
+
+
+
+
+
+
+
+
+
+
+ Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
+
+
+
+
+
+
@@ -4476,9 +4576,75 @@ F84/TN93 and is included below. snp- prefix on every CLI value.
+Γ rate-heterogeneity modifier, applicable to snp-jc, snp-k2p,
snp-k81, snp-t92, snp-f84, snp-hky85, snp-tn93 (not snp-raw,
nothing to correct; not snp-logdet, no standard gamma formulation) — same
-formula as the base correction, weighted by a shape parameter α supplied
-by the user (--gamma-shape <alpha>), not estimated by ML. A modifier on
-existing values, not a separate enum arm per distance.
+formula as the base correction, weighted by a shape parameter α either
+supplied by the user (--gamma-shape <alpha>) or estimated from the data
+(--gamma-shape auto/estimate, method-of-moments — not ML; see
+"Automatic α estimation" below). A modifier on existing values, not a
+separate enum arm per distance.
+Automatic α estimation (--gamma-shape auto)
+Correction (verified against the primary source, 2026-09-11): Jin &
+Nei (1990) itself ("Limitations of the Evolutionary Parsimony Method of
+Phylogenetic Analysis", Mol. Biol. Evol. 7(2):82–102 — the paper this
+whole +Γ correction is cited from, confirmed algebraically to match this
+codebase's corrected_log/k2p exactly against their eq. A4, general, and
+A8, the a = 1 case) contains no data-driven α-estimation procedure.
+Their own recommendation (p. 98) is a fixed default: "we suggest that the
+gamma distance with a = 1 [eq. A8] be used. However, one may choose a
+different gamma distance, estimating a from data. Wilson et al. (1989)
+recently used a distance with a = 1/2 for restriction-site data of
+mitochondrial DNA in hominoids." — i.e. Jin & Nei explicitly punt
+data-driven estimation to a different paper (Wilson et al. 1989), not
+read/verified here. The estimator below is therefore not "Jin & Nei's
+method" under any framing — that attribution (present in an earlier
+revision of this section) was wrong, not just under-cited.
+Implemented (PartitionDispersion, obikphylo/src/siblings/algorithms/pairwise.rs)
+as an independent method-of-moments estimator, unrelated to any specific
+published procedure: pools substitution counts by partition rather
+than by genome pair, during the same reduce_pairwise pass that builds
+PairwiseTally (no second scan).
+For partition i: n_i = substitutions pooled over every genome pair,
+L_i = eligible loci pooled over every genome pair, R_i = n_i / L_i.
+Modeling among-site rate heterogeneity the same way as the +Γ correction
+itself (a Gamma(α, α)-distributed, mean-1, multiplicative rate shared by
+every locus in a partition — the classical Poisson-Gamma/negative-binomial
+mixture, the general identity behind gamma-rate-heterogeneity corrections,
+also behind Uzzell & Corbin's (1971) original observation that substitution
+counts across sites/regions are over-dispersed relative to Poisson):
+\[
+\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
+\]
+Weighting each partition's squared deviation by its own L_i cancels the
+Poisson term before attributing what's left to α:
+\[
+\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad
+V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad
+\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
+\]
+where \bar L is the mean partition size. Returns "no estimate" (falls
+back to the uncorrected formula, warns) when fewer than 2 partitions have
+data, \hat\mu \le 0, or V doesn't exceed the Poisson floor
+\hat\mu/\bar L — no detectable over-dispersion, α would be unbounded.
+Caveat, stated explicitly rather than left implicit: unlike every
+closed-form correction in snp_distance.rs (each verified line-by-line
+against ape's dist_dna.c, and now also against Jin & Nei 1990 directly
+for the base +Γ formula), this estimator is derived from first
+principles (the general Poisson-Gamma/negative-binomial identity) with no
+primary-source procedure behind it at all — not Jin & Nei's (confirmed
+above), and Wilson et al. (1989), the paper they point to instead, hasn't
+been read/verified either. Mathematically self-consistent (re-derived and
+checked, not guessed), but a from-scratch method, not a literature
+implementation. If --gamma-shape needs a value with a literature
+pedigree rather than an estimated one, Jin & Nei's own stated default,
+α = 1 (--gamma-shape 1), is the better-supported choice today.
+Deliberately not gated by --sankoff-ratio-ceiling the way
+base_pair_tally is (same precedent as cardinality_tally — see its own
+doc comment): that filter excludes individual saturated pairs from a
+composition estimate computed once at the very end, from the complete
+PairwiseTally; the partition axis needed here only exists transiently,
+one partition at a time, while PairwiseTally is still being built — long
+before any pair's final SNP ratio (and thus its ratio_ceiling eligibility)
+is known. --exclude-genome isn't applied either, matching
+reduce_pairwise's own raw per-pair fold.
Implemented now: snp-raw, snp-jc, snp-k2p, snp-k81, snp-f81,
snp-t92, snp-tn93, snp-tv, all with +Γ except raw/tv — see
"Exact formulas" below. snp-tajima-nei, snp-f84, snp-hky85,
@@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same
mechanical substitution but weren't independently checked against an
ape-equivalent reference for those three specifically — flagged here, not
silently assumed correct.
+α-estimation literature survey and design discussion (2026-09-11)
+Follow-up discussion after implementing PartitionDispersion::estimate_alpha
+(previous section), triggered by the user pointing out that the literature
+conflates two distinct problems: estimating α from an alignment versus
+using a given α in a Jin-Nei-style corrected distance. Jin & Nei (1990)
+itself is squarely in the second camp — α is a user-supplied parameter
+there, not something their paper estimates (confirmed by reading the paper
+directly, see previous section). This section catalogs the actual
+α-estimation literature and records which approaches were considered and
+why each was or wasn't adopted.
+Bibliography (methodological papers, not exhaustive users of +Γ):
+
+
+
+Method
+Reference
+Needs a tree?
+Estimates "true" α or a task-specific one?
+Cost
+
+
+
+
+ML, continuous Γ (foundational)
+Gu, Fu & Li (1995), MBE 12:546–557 — read in full
+yes, topology fixed a priori
+true α (+ θ jointly, invariant+Γ)
+very high, ≤5–6 taxa in practice
+
+
+ML, discrete Γ
+Yang (1993/94), MBE 10:1396; JME 39:306
+yes
+true α
+high
+
+
+Parsimony-based ML on inferred changes
+Yang & Kumar (1996), MBE 13:650
+yes, equal branch lengths assumed
+true α, biased when branches unequal
+low
+
+
+Corrected substitutions/site → NB fit
+Gu & Zhang (1997), MBE 14:1106 — read in full
+yes, + ancestral reconstruction
+true α, close to ML accuracy
+low (given a tree)
+
+
+Topology-optimizing grid search
+Guindon & Gascuel (2002), MBE 19:534 — read in full
+yes, but built from the candidate distances themselves (BIONJ)
+not true α — deliberately biased upward for topological accuracy
+low–moderate (grid × tree build)
+
+
+Free (non-parametric) rate distribution
+Susko et al. (2003), Syst. Biol. 52:594
+yes
+tests the Γ assumption itself
+high
+
+
+Parsimony counts/site → NB fit
+Wakeley (1993), JME 37:613 — read in full
+yes, NJ tree + Fitch parsimony ancestral reconstruction
+true α, biased upward by parsimony (conservative)
+low (given a tree)
+
+
+Poisson-Gamma/negative-binomial, foundational
+Uzzell & Corbin (1971), Science 172:1089 — not read, only seen via citations
+unknown — likely also count-per-site/tree-based, unverified
+true α (indirect, via NB overdispersion)
+unknown
+
+
+Bayesian posterior over α
+BPP/MrBayes/BEAST2/RevBayes
+yes
+posterior, not point estimate
+very high
+
+
+
+Correction (2026-09-11, after reading Wakeley in full): an earlier
+revision of this table filed Wakeley (1993) under "no tree needed" —
+wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei
+1987, on the real mtDNA data; coalescent-simulated for the simulation
+study), reconstruct ancestral states at internal nodes by Fitch (1971)
+parsimony, count the minimum number of changes per site this implies,
+then fit either a two-rate Poisson mixture or a gamma-distributed-rates
+model (⟹ negative binomial, parameters by the methods of Cohen 1965 and
+Johnson & Kotz 1969) to the resulting per-site count distribution — the
+same general "tree + ancestral reconstruction + per-site counts + NB fit"
+shape as Gu & Zhang (1997), just using raw parsimony counts directly
+rather than Gu & Zhang's multiple-hit-corrected k̂. Confirmed by
+simulation (Wakeley's own Fig. 1) that parsimony underestimates both the
+mean and (more severely) the variance of per-site change counts, which
+biases α upward (toward apparent uniformity) — consistent with Gu &
+Zhang's own citation of Wakeley for exactly this bias, and with the
+general "parsimony overestimates α" pattern noted throughout this
+literature. Real hypervariable-region-1 mtDNA data gave α̂ ≈ 0.44–0.60
+per data set, 0.47 combined — despite the conservative bias, still
+solidly in the "strong heterogeneity" range, which is the paper's own
+point: the method is biased but usefully conservative, not useless.
+Where PartitionDispersion::estimate_alpha actually sits: nowhere in
+this table's tree-dependent rows. It shares only the abstract
+mathematical identity (Poisson-Gamma mixture ⟹ negative binomial
+over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a
+genuinely different, tree-free and ancestral-reconstruction-free route:
+pairwise genome-to-genome substitution counts pooled by partition
+instead of per-site counts pooled across an alignment's columns after
+ancestral state reconstruction. As far as this survey has established
+(Uzzell & Corbin not read, so not fully ruled out), no tree and no
+ancestral reconstruction of any kind appears to be a genuinely
+distinguishing property of PartitionDispersion, not something it
+inherits from prior art — consistent with it being an independent
+derivation rather than a literature implementation (see previous
+section's caveat, now on firmer footing).
+Why a tree-free estimate is even valid — the identifiability question.
+Raised directly by the user after reading Gu, Fu & Li (1995), which states
+explicitly: "one cannot estimate the rate heterogeneity when the number
+of sequences is <3." This is a real non-identifiability result, not a
+practical inconvenience: with only 2 sequences, a site's history is
+summarized by one binary observation (differ / don't differ), collapsed
+across all sites into two aggregate numbers (transition and transversion
+proportions) — divergence time and among-site rate variance are
+confounded in that pair of numbers, with no way to separate them. Every
+method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3
+sequences sharing one evolutionary history (a tree) specifically to get
+multiple independent looks at the same site's rate across different
+lineages, which is what identifies its variance.
+PartitionDispersion doesn't violate this, because it isn't estimating
+the same quantity. The literature above estimates rate variation across
+sites, within one shared tree. PartitionDispersion estimates rate
+variation across partitions, pooled over every genome pair in the
+index — each genome pair stands in for one independent draw of a
+partition's relative rate, the same identifying role multiple lineages
+play in the tree-based methods, just substituting "genome pair" for
+"lineage" and "partition" for "site." With P genome pairs (potentially
+in the hundreds here) all contributing to every partition's pooled count,
+the ≥3-samples identifiability requirement is met by the pair count, not
+by tree depth — closer in spirit to Jin & Nei's own original motivating
+scenario (comparing several genomic regions across one fixed panel of
+taxa) than to single-tree site-rate estimation.
+This substitution carries its own assumption, which must be named rather
+than left implicit: that a partition's relative rate (fast/slow) is
+reasonably stable across genome pairs, not just across sites within
+one pair — the direct analogue of the "no lineage-specific rate variation,
+only site-specific" assumption every tree-based +Γ method already makes,
+just moved from lineage→pair and site→partition. If this breaks — e.g. an
+index mixing very closely related and very divergent genome pairs, where a
+partition's saturation behavior differs qualitatively between the two —
+the estimate could be misled in a way a tree-based method would at least
+have the topology to detect and a tree-free, pool-everything method
+cannot. Not yet tested against real data with strongly heterogeneous
+pairwise divergence; worth keeping in mind as the main open validity
+question for --gamma-shape auto, not the citation question (now settled)
+this whole discussion started from.
+Gu, Fu & Li (1995), read in full — the foundational ML method, not a
+candidate to implement. Models rate variation as invariant+Γ (θ =
+proportion of invariant sites, α = gamma shape among the variable sites),
+and derives the exact joint likelihood over nucleotide configurations
+across all n sequences on a fixed, a-priori-known tree topology —
+for 3 sequences this is closed-form (Jukes-Cantor), for n sequences it's
+a sum over 4^n configurations weighted by coefficients tied to the
+specific topology's branching structure (their eq. 32–34), with branch
+lengths, α, and θ jointly optimized by Newton-Raphson (Hessian over all
+free parameters). The site-rate integral has the same closed form
+E = θ + (1-θ)(1+D/α)^{-α} that underlies +Γ distance corrections
+generally (the same mechanism as corrected_log, not a coincidence — Jin &
+Nei's own formula is the θ=0 special case applied to a pairwise D
+rather than a whole-tree one). Explicitly stated by the authors as
+infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a
+candidate for obikmer's typical genome counts. Notable contribution worth
+keeping in mind regardless: ρ = (1+θα)/(1+α) is shown to be a more robust
+summary of rate heterogeneity than α alone whenever θ isn't negligible
+— α and θ are confounded (nearly-invariant sites can come from either a
+small α's left tail or from a nonzero θ), so α alone can be very
+unstable while ρ stays well-behaved. PartitionDispersion has no θ
+term at all (no invariant-site component), so this confound doesn't arise
+for it the same way, but it's worth remembering if an invariant+Γ variant
+is ever considered.
+Gu & Zhang (1997), read in full — rejected for now. Their procedure:
+(1) a tree with least-squares branch lengths must already exist; (2)
+ancestral states are reconstructed at every internal node (they use Zhang &
+Nei 1997's likelihood method, but note plain parsimony would work, just
+less accurately); (3) per site, branches are split into "changed"/"unchanged"
+given the ancestral reconstruction, and the expected substitution count
+k̂ (corrected for multiple hits) is obtained by solving their eq. (7)
+(or eq. 12 for the generalized model) numerically — a root-find per site,
+not a closed form; (4) the site-level k̂ values (real-valued, not
+integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to
+get α — again no closed form, numerical optimization. This is a real
+per-site pipeline requiring a tree before distances/α can be computed —
+the reverse of obikmer's current dependency order (snp_distance computes
+distances, which --nj/--upgma only turn into a tree afterward). Their
+own answer to this chicken-and-egg problem is an iterative
+distance→tree→α→distance loop, which they themselves flag as expensive.
+Adopting this would mean a new ancestral-reconstruction subsystem (even a
+parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB
+ML fit — a much bigger addition than PartitionDispersion, not something
+to build without a concrete need beyond "closer to Gu&Zhang's own
+simulated accuracy than our moment estimator."
+Guindon & Gascuel (2002), read in full — plausible future addition, not implemented.
+Their Q criterion: for a candidate α, build a tree (they use BIONJ) from
+the α-corrected distance matrix; for every internal branch, group taxa into
+the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree
+distances \(\bar\delta_{AB}\), \(\bar\delta_{CD}\), etc., and let S ≤ M ≤ L
+be the three pairings' sums ({AB,CD}, {AC,BD}, {AD,BC}); the branch's
+reliability score is Q_branch = L − M (zero when the four points are
+perfectly tree-additive). The whole-tree Q is the mean over internal
+branches (negative/zero branches excluded). α* = argmin_α Q(α) over a
+grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where
+sensitivity is highest). Complexity: O(n²l) once for the base
+frequencies, then O(n²r) for the r candidate distance matrices, then
+O(n³r) worst case for building r trees and evaluating Q on each
+(same order as the tree-building step itself, so "free" relative to it).
+Central finding, the opposite direction from an initial misreading in
+this discussion: α_opt (Guindon-Gascuel's topology-optimal value) is
+always ≥ the true α, not the reverse — underestimating rate
+heterogeneity (i.e., picking a larger α than reality) reduces the variance
+of distance estimates and thereby improves NJ/BIONJ topological accuracy,
+especially when the molecular clock roughly holds (where α_opt → ∞, i.e.
+no correction at all is topologically best). This is an empirical
+regularity observed across their tested conditions (η ∈ {0.5, 2.0}, true
+α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard
+guarantee would be another unverified-claim mistake of the kind this whole
+discussion has been correcting.
+Proposed integration (not yet implemented): since α_true ≤ α_opt
+empirically, PartitionDispersion::estimate_alpha()'s output could seed
+the lower bound of Guindon-Gascuel's grid (search [α̂, ∞) instead of
+[0.1, 5000]), narrowing the grid substantially and reusing the existing
+estimator rather than needing an independent "true α" source — this also
+sidesteps Gu & Zhang's tree-first dependency problem entirely, since
+Guindon-Gascuel's own grid already builds a tree per candidate. Still
+requires: (a) a BIONJ implementation (see below), (b) the Q-criterion
+subtree-grouping logic, (c) a grid-search driver. Not started.
+BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found.
+Searched crates.io directly (API query for "bionj": 0 results) and read the
+docs.rs pages of the most plausible candidates: phylotree (builds/reads/
+manipulates trees but has no distance-matrix reconstruction — no NJ, no
+BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected
+after checking the actual docs), speedytree (NJ only — Canonical/RapidNJ/
+Hybrid variants, explicitly no BIONJ), nj (plain NJ only), phylo (no
+distance-based reconstruction at all). obikphylo::neighbor_joining
+(tree.rs) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in
+how branch lengths are weighted during agglomeration (accounts for
+distance-estimate variance, not just the sum), a modest delta over an
+existing NJ implementation rather than a rewrite, but would need to be
+written in-house if the Guindon-Gascuel integration above is ever pursued
+(their own paper notes other tree-building methods gave similar results in
+their simulations, so plain NJ may be an acceptable substitute if BIONJ
+proper is judged not worth the effort).
+Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11
+Correctness bug, not just a performance one. --sankoff/--tnt/--phyg/
+--iqtree (via SiblingExt::sankoff_bundle) and a snp-* --distance
+(via SiblingExt::snp_distance) always consume the exact same selection
+parameters when requested together in one obikmer phylo invocation
+(n/free_loss/no_ambiguity/excluded-set/entropy_bias — the CLI has
+no way to give them different values in one run, see cmd/phylo/args.rs).
+Before this fix, each independently called sample_index, and
+sample_layer's rand::rng() (subsample.rs:235) is a thread-local
+generator that advances across calls, not reseeded each time — so the
+second call silently drew a different random sample of sites than the
+first, even with identical parameters. Verified empirically (throwaway
+8-genome index): running snp-jc alone vs. combined with --sankoff
+produced visibly different distance matrices from the same index/params.
+This defeated the actual point of combining these flags — running several
+algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a snp-* distance) on
+one identical site selection for direct comparison.
+Fix: SankoffBundle (sankoff.rs) now retains its PairwiseTally/
+PartitionDispersion internally (previously consumed into raw/
+base_pair_tally/cardinality_tally and dropped) and exposes
+SankoffBundle::snp_distance(kind, gamma_shape), which computes the
+matrix from that same already-sampled tally — no second sample_index
+call. The shared post-sampling logic (the --gamma-shape support check,
+alpha resolution, final matrix build — previously all inline in the
+snp_distance() free function) was factored into
+pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape) in
+snp_distance.rs, called by both the standalone snp_distance() (after
+its own fresh sampling) and SankoffBundle::snp_distance (reusing the
+bundle's). cmd/phylo/mod.rs keeps the Option<SankoffBundle> alive past
+the Sankoff-family if block and, when a snp-* --distance is also
+requested, calls bundle.snp_distance(...) instead of
+cache.snp_distance(...) whenever a bundle was built — since both branches
+are driven by the same args.* fields, the parameters trivially always
+match when both fire; no runtime "do the params match" check needed.
+Verified end-to-end: the reused path logs (reusing the Sankoff bundle's
+sample) and the snp_distance stage timer reads 0ms (formula(kind)
+is O(n²) post-processing, no I/O), vs. ~150-230ms for a fresh sample on
+the same tiny test index.
+Explicit --session (implemented 2026-09-12)
+The in-process fix above only covered reuse within one command.
+--session DIR (cmd/phylo/args.rs) names a directory (outside the
+index) that persists the snp-* --distance sample — the
+PairwiseTally/PartitionDispersion pair sample_index would otherwise
+rebuild from scratch — across separate obikmer phylo invocations.
+Crate placement, as planned: a new obiksession crate
+(src/obiksession/), domain-agnostic — no PairwiseTally/"site" concept
+anywhere in it. Session::open(dir, params: &[u8], force: bool) handles
+directory lifecycle and the params-conflict check against an opaque byte
+blob; Session::store(name, bytes)/Session::restore(name) -> Option<Mmap>
+handle a checksummed artifact cache. obikphylo
+(siblings/algorithms/session_cache.rs, shared by snp_distance.rs and
+sankoff.rs — see below) is the consumer: it serializes its own
+PairwiseTally/PartitionDispersion/SnpAlignment to bytes (via rkyv)
+and hands them to obiksession, which never sees their type.
+Serialization: rkyv 0.8.18, added as planned. One deviation from the
+original zero-copy pitch, scoped down deliberately: artifacts are restored
+via rkyv::from_bytes (full owned deserialize) rather than rkyv::access
+(zero-copy over the mmap). True zero-copy would require every
+PairwiseTally query method (categories, base_freq, pair, ...) to
+work generically over Archived<PairwiseTally> as well as the owned type
+— a separate, larger change not needed to get the actual win (skipping
+sample_index's expensive re-scan of the sibling annex; deserializing an
+already-in-memory-sized count array is comparatively cheap CPU, not I/O).
+#[derive(Archive, Serialize, Deserialize)] was added to PairStats,
+PairwiseTally, and PartitionDispersion (pairwise.rs) for this.
+Startup behavior, as designed: Session::open creates DIR if
+missing (fresh, params recorded); if DIR already holds different saved
+params, cmd/phylo reports a hard error and exits rather than silently
+preferring one side, unless --session-force is given (which discards the
+directory's cached artifacts and starts over under the new params) —
+implements exactly the "explicit escape hatch, not silent override"
+decision from the design discussion. obiksession itself doesn't decode
+either side's bytes to produce a field-by-field diff (true to "opaque
+blob," left to whoever needs it); cmd/phylo's conflict message is
+currently a single generic sentence listing which flags could be the
+cause, not a computed diff — an honest v1 simplification, not a limitation
+of obiksession itself.
+Per-artifact caching, as designed: only PairwiseTally/
+PartitionDispersion are cached (snp_distance.rs's restore_tally/
+store_tally) — ratio_ceiling/gamma_shape remain pure post-processing
+over whatever tally is in hand, cached or fresh, exactly as planned.
+Locking, reconsidered and kept as the whole-session DirLock. First
+written up as a corner cut ("per-entry locking was sketched, not
+implemented"). On actually working through the concrete race, whole-session
+locking turned out to be the correct choice, not a simplification:
+--session-force resets a session by deleting its whole artifacts/
+directory. If the directory lock were only held briefly (during
+Session::open, then released so a long-running Session doesn't block
+other processes from touching different artifacts), a second process
+calling open(..., force: true) could acquire the lock, see mismatched
+params, and wipe artifacts/ while a first process's already-open
+Session is mid-checkpoint — genuine corruption risk, not just wasted
+work. Holding the lock for the Session's entire lifetime makes that
+sequence impossible: a force reset must wait for every currently-open
+Session on that directory to be dropped first. The concurrency this
+gives up (two different processes reading/writing different artifacts
+of the same session at once) isn't exercised by anything today — only one
+code path (snp_distance/sankoff_bundle, sequentially per invocation)
+uses sessions at all.
+Per-layer chunked/resumable dumps — implemented. subsample::sample_index
+gained a skip_layers: usize parameter (skip the first N (partition,
+layer) pairs, in its own deterministic iteration order, entirely — no
+I/O, no on_layer call) and its on_layer callback gained a raw_index
+argument (that pair's 0-based position in the same order), so a caller can
+tell exactly how far a run has gotten. snp_distance/sankoff_bundle
+checkpoint their tally/dispersion (and, for sankoff_bundle, the
+in-progress SnpAlignment) to the session every
+session_cache::CHECKPOINT_INTERVAL_LAYERS (8) layers or
+CHECKPOINT_INTERVAL (30s) of wall time, whichever comes first — bounding
+how much completed scanning a crash can lose, without paying a full
+tally-rewrite on every one of what can be thousands of layers. A progress
+artifact records how many layers are reflected in the checkpoint, with a
+PROGRESS_COMPLETE (u64::MAX) sentinel once every layer has been
+processed (distinct from any real layer count, so "finished" is never
+confused with "stopped after N layers where N happens to equal the total").
+Verified with a real kill: built a 10-genome/64-partition index, ran
+--distance snp-jc --session DIR under timeout 0.4, confirmed a
+checkpoint at progress=8 (exactly CHECKPOINT_INTERVAL_LAYERS) was on
+disk, then re-ran the same command and confirmed it resumed from layer 8
+and completed successfully — 0-effort verification that the mechanism
+itself is real, not just plausible-sounding.
+Important correction from that same test — resume is not bit-reproducible,
+and shouldn't be. The output matrix from the resumed run differed from
+an uninterrupted control run against the same index/params. Root cause:
+sample_layer's random draws come from rand::rng(), a per-process,
+OS-seeded generator with no continuity across a process boundary — the
+layers processed before a kill and the layers processed after a resume
+come from two independent random streams. An earlier revision of this doc
+(and of the code comments) claimed resume "reproduces the exact same
+sample an uninterrupted run would have" — that's wrong, and was corrected
+after the user pointed out the actual reasoning error: true randomness has
+no notion of continuity in the first place, so there was never a
+correctness requirement for a resumed run to match an uninterrupted one
+bit-for-bit. The sample produced after a resume is exactly as legitimate a
+random draw as one from an uninterrupted run — just a different one,
+same as running the command twice without --session already gives two
+different samples today. That variance is explicitly wanted (see below),
+not a defect to fix. Comments in subsample.rs/session_cache.rs were
+corrected to state this plainly instead of the false reproducibility claim.
+Explicitly declined: deterministic/seeded sampling. Raised as a
+possible fix for the above (seed sample_layer's RNG from a per-session
+value plus (partition, layer), making a resume reproduce bit-identical
+results, and, if applied everywhere, making any two runs with identical
+parameters produce identical samples). The user explicitly declined this
+for now: repeated runs are relied on to measure sampling variance, and a
+--seed flag would need its own explicit design if wanted later — this is
+not an accidental gap, it's a stated preference to revisit only if asked.
+SankoffBundle integration — implemented. sankoff_bundle accepts
+the same session: Option<&Session> and uses the same artifact names
+(session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT})
+as snp_distance, plus its own ALIGNMENT_ARTIFACT (sankoff.rs) folded
+in lockstep with the tally at every checkpoint (same on_layer call
+produces both, so they can never drift out of sync). Consequence verified
+directly: a tally checkpointed by --sankoff --session DIR is restored by
+a later, separate --distance snp-k2p --session DIR invocation with no
+--sankoff at all, and vice versa — cross-consumer reuse, not just
+same-command reuse, exactly the synergy this design aimed for.
+Per-artifact caching, as designed: only the tally/dispersion (plus,
+for sankoff_bundle, the alignment) are cached — ratio_ceiling/
+gamma_shape remain pure post-processing over whatever tally is in hand,
+cached or fresh, exactly as planned.
+Still not done — genuine follow-up, not a justified tradeoff:
+exhaustive-sample caching for --subsample-varying exploration. The
+design discussion's suggestion — cache the exhaustive (n = None) sample
+once and subsample from it on each run instead of one session per exact
+n — wasn't implemented; a session today is still scoped to one exact
+parameter tuple, --subsample included.
Output format: PHYLIP-relaxed by default for the distance matrix
Implemented. The primary distance-matrix output
(_dist.csv today) gains multiple formats: PHYLIP-relaxed becomes the
diff --git a/DevDoc/theory/indexing.refs/index.html b/DevDoc/theory/indexing.refs/index.html
index 890844ab..8dcb615c 100644
--- a/DevDoc/theory/indexing.refs/index.html
+++ b/DevDoc/theory/indexing.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/theory/indexing/index.html b/DevDoc/theory/indexing/index.html
index afbd21e6..be5ea79a 100644
--- a/DevDoc/theory/indexing/index.html
+++ b/DevDoc/theory/indexing/index.html
@@ -1199,7 +1199,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/theory/minimizer.refs/index.html b/DevDoc/theory/minimizer.refs/index.html
index efa795a8..26fbf103 100644
--- a/DevDoc/theory/minimizer.refs/index.html
+++ b/DevDoc/theory/minimizer.refs/index.html
@@ -1124,7 +1124,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDoc/theory/minimizer/index.html b/DevDoc/theory/minimizer/index.html
index f7d4cdc0..b789b4f2 100644
--- a/DevDoc/theory/minimizer/index.html
+++ b/DevDoc/theory/minimizer/index.html
@@ -1243,7 +1243,7 @@
- Sibling annex (discussion)
+ Sibling annex
diff --git a/DevDocMD/architecture/siblings.md b/DevDocMD/architecture/siblings.md
index 352956f4..a9e32377 100644
--- a/DevDocMD/architecture/siblings.md
+++ b/DevDocMD/architecture/siblings.md
@@ -1,1024 +1,160 @@
-# Sibling annex — architecture (discussion)
-
-Status: architecture decided (2026-08-14). Implementation not yet mandated.
-
-## Two index spaces, uncorrelated
-
-Every kmer stored in a `Layer` lives in two independent index spaces:
-
-- **Iteration order**: its position when enumerating `unitigs.bin` (the
- superkmer file), deterministic but arbitrary with respect to slot.
-- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns.
-
-The two are not correlated by any formula. Converting from one to the other
-requires either recomputing the MPHF (kmer → slot) or scanning the iteration
-stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a
-one-way function, not an invertible bijection with a stored inverse. Any
-method that reconstructs a kmer from a bare slot number is wrong by
-construction, regardless of the mechanism used (MPHF re-hash, or evidence
-decode + direct unitig read). See `MphfLayer::kmer_at`
-(`obikindex/src/layer/mphf_layer.rs`) — flagged for removal, currently called
-from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
-"Pending work" status below).
-
-## Two pipelines, never mixed
-
-| | origin of the kmer | membership known? | correct mapping |
-|---|---|---|---|
-| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check |
-| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence |
-
-Evidence exists solely to answer "is this external kmer a member of the
-layer" for the query pipeline. Using it (or the MPHF) to go the other way —
-recover a kmer from a slot, or re-verify a kmer that was just produced by
-iterating the layer — is a conceptual error: evidence can be probabilistic
-(`Approx` mode), so any slot→kmer attempt is unsound in general, and
-pointless even in `Exact`/`Hybrid` mode since the kmer was already known.
-
-## Sibling annex: an iteration-pipeline artifact only
-
-The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`,
-`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a
-family minorant and which family members are present in the index. Its only
-consumers (`obikphylo/siblings/stats.rs`, `family_scan.rs`) enumerate it
-exhaustively (`0..annex.len()`); no query-pipeline code path touches it.
-
-**Decision**: the annex must be persisted in iteration order, not slot
-order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex
-file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
-no `kmer_at`. It also enables specialized iterators building on this zip:
-minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.
-
-Today the annex is built and stored in **slot** order
-(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated
-via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is
-threaded through the whole cross-partition reconciliation pipeline (variant
-generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This
-must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading
-the **iteration index** instead of the slot end to end — eliminating
-`kmer_at` from the build path entirely, not just the read path. No
-slot-indexed intermediate is needed even during construction; the
-iteration-order id is sufficient throughout.
-
-The cross-partition side of the same pipeline is unaffected: checking
-whether a generated family-variant kmer exists in another partition is a
-genuine query-pipeline operation (the variant's membership in the *target*
-partition is unknown) and must keep going through
-`KmerPartition::query_partition_with` (MPHF + evidence), never a raw
-`index()`.
-
-## Pending work — done
-
-The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
-traits over `obikindex::KmerIndex`/`obikindex::layer::Layer`, replacing the
-old `obikindex::siblings` module) builds and reads the annex purely in
-iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
-batch variants, mirroring `Layer`'s own `KmerIter`/`KmerBatchIter`
-shape). `MphfLayer::kmer_at` has no remaining callers.
-
-A separate, unrelated bug surfaced during this work and was fixed
-(2026-08-14): `MphfLayer::enumerate_kmers_batch` computed its
-`batch_start_index` via the stdlib `.enumerate()` adapter, which counts
-*batches* (0, 1, 2…), not the cumulative k-mer offset the annex is actually
-keyed on — every batch past the first wrote its mask/annex entries at the
-wrong iteration-order position. Fixed by tracking a running offset instead;
-regression tests added (`sibling_annex_no_empty_masks_after_build`,
-`sibling_histogram_does_not_panic_on_partial_last_batch`).
-
-## Performance: `build_sibling_annex` parallelism (2026-08-14)
-
-Investigated on a real multi-genome run (`phyloskims_sal_vac`, k=31/m=11).
-Baseline: mostly one active core, with short multi-core bursts — average
-~3 cores.
-
-**Fixes that helped, kept:**
-
-- `CanonicalKmerOf::minimizer()` (`obikseq/src/kmer.rs`) — a direct O(k)
- bit-arithmetic minimiser for a single isolated k-mer, replacing a
- `RollingStat` instance fed byte-by-byte through an ASCII round-trip (used
- by `helpers::partition_of`, called for every generated family variant).
- ~3x wall-clock improvement on its own, confirmed by sampling
- (`obiskbuilder::rolling_stat`/`obikentropy` frames disappeared from the
- hot path). `CanonicalKmerOf::partition()` added alongside it (wraps
- `minimizer().seq_hash() & mask`, the same routing rule
- `KmerPartition`/`RoutableSuperKmer` use).
-- Cross-partition resolution (`outgoing.par_iter()` in
- `build_layer_sibling_annex`) parallelised at the *partition* level — one
- Rayon task per non-empty `outgoing[dest]` bucket. For k=31/m=11, a
- central-base substitution changes the winning minimiser (and thus the
- destination partition) only when that window overlaps the central base:
- ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants
- route right back to the partition already being built. That self bucket
- ends up far larger than any other, so the per-partition split pinned one
- thread to it alone while the rest of the pool finished instantly —
- confirmed by sampling: one thread solid in `MphfLayer::find`, everyone
- else idle. Fixed by splitting each non-empty bucket into
- `total_queries / n_workers` (capped 4096) chunks *before* `par_iter()`,
- preserving per-partition mmap locality (each chunk stays contiguous
- within one partition) while letting Rayon spread an oversized bucket
- across several threads. Net effect of both fixes together: ~3 cores
- average → ~10-13 cores average on the same run, and a projected total
- build time of ~1h15 down to ~30min on the real `phyloskims_sal_vac` run
- this was measured against.
-- `TracedBar`'s ETA (`obisys/src/progress.rs`) was silently starved: the
- custom progress message and the self-computed ETA text used to share one
- `pb.set_message()` slot, with the ETA holding off for 2s after any custom
- message — fine when custom messages are rare, broken once
- `build_sibling_annex`'s per-partition callback fires more often than
- that. Fixed by keeping the two texts in separate fields, composed
- together on every render instead of one overwriting the other.
-
-**Tried and reverted — do not repeat blindly:**
-
-- Parallelising the *outer* partition loop in `build_sibling_annex` with
- `obikindex::PartitionRunner` (already used by `merge`/`build_layers`),
- splitting a fixed core budget between outer (partition) and inner
- (pipeline + resolution) concurrency so their product wouldn't exceed the
- budget. Measured *worse*: throughput dropped over time (26
- partitions/5min → 38/11-12min) and peak resolution concurrency fell from
- ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which
- scales very well on its own — to make room for outer concurrency, and
- running several partitions' resolution at once scatters access across
- multiple partitions' mmap regions at once, working against the
- locality `outgoing`'s per-partition grouping exists for. `PartitionRunner`
- stayed exported from `obikindex` (`new_capped` too) since it's
- general-purpose, but nothing in `obikphylo` calls it.
-- Splitting resolution chunks even finer (`/(n_workers*8)`, cap 1024,
- instead of `/n_workers`, cap 4096) to smooth the residual sawtooth.
- Measured ~10% *slower*, wider dips, not narrower. Reverted to the
- original chunk sizing.
-
-**Known remaining limitation, not yet worth fixing:** within one layer, the
-four stages (sequential `unitigs.bin` read → parallel generation →
-parallel resolution → sequential annex write) never overlap — confirmed by
-1s-interval sampling: generation alone occupies ~17 threads evenly, but the
-next layer's read/generation never starts until the current layer's
-resolution and write are both done. This produces a real, periodic (~layer
-duration) alternation between "many cores" and "few cores" that neither of
-the fixes above touches, since both operate *within* one layer's resolution
-step. The only remaining lever is overlapping consecutive layers (e.g. a
-depth-2 pipeline: start layer N+1's read/generation while layer N's
-resolution/write is still running) — a real restructuring, not a parameter
-tweak, and explicitly *not* to be combined with the reverted
-budget-capping idea above (let each phase use however many cores it
-naturally wants; only the *scheduling* needs to overlap). Deferred, not
-started.
-
-## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)
-
-`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`,
-`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already
-implements most of a dispatch/accumulate/resolve pipeline: generation
-(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and
-buckets cross-partition queries) runs on an `obipipeline::throttle` +
-`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter`
-*across partitions*, one generated batch resolved at a time, never several
-concurrently — this ordering is deliberate, see the module's own docs on a
-reverted concurrent-batch-resolution attempt that scattered mmap access).
-The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`)
-already exists: `n_layers <= 7` (checked once from the first non-empty
-partition's `PartitionMeta::n_layers`, documented as identical across every
-partition of an index — a structural, build-time property, never a
-per-partition state) decides whether `FamilyMask`'s recorded `layer_value`
-can be trusted to skip straight to the right layer
-(`find_presence_batch_fast`) or must fall back to scanning every layer of
-the destination partition (`find_presence_batch`).
-
-**Real gap, confirmed not implemented**: resolution is triggered by the
-*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the
-scanned layer), not by an *output* accumulator filling up. Since most
-central-base variants of a family route back to the same partition being
-scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s
-`outgoing[dest]` is large for the local/self partition and thin for the
-other ~255 (or however many) destination partitions — each of those gets
-resolved at low query density every batch instead of being accumulated
-across several source batches until resolving it is worthwhile. This is
-distinct from, and not fixed by, the fast/slow layer gate above.
-
-Redesign sketched (not built): per-destination accumulators decoupled from
-`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch
-completion — a "hot" accumulator for the partition being scanned (sharded
-one-per-generation-worker, no lock, since all `n_workers` pipeline workers
-write to it concurrently — this differs from an earlier, simpler mental
-model of "one thread owns one layer's local collector," which doesn't hold
-here since `n_workers` threads cooperate on scanning *one* layer at a time,
-not one thread per layer) and "cold" mutex-per-partition accumulators for
-the rest, low contention expected since traffic to any single cold
-destination is a small fraction of total.
-
-This breaks the current strict-iteration-order delivery of `on_family`
-(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH`
-resolves atomically). With cross-batch accumulation, a family only becomes
-complete once *every* accumulator holding one of its outgoing queries has
-flushed, at unpredictable, independent times — no longer streamable
-strictly in order without a large, unbounded pending buffer. Resolution
-sketched: replace order-dependent consumers with coordinate-addressed
-writes instead of order-dependent appends (see `PseudoAlignment` idea
-below) wherever possible, since `sibling_annex_stats`'s reduction (plain
-counts) is already order-independent and needs nothing here.
-
-**Superseded 2026-08-15** by the `--subsample`/`--shannon` design below,
-which sidesteps the accumulator redesign for now: bounding the number of
-families actually resolved per layer (via sampling) keeps per-layer
-resolution volume small enough that the batch-density problem above stops
-mattering in practice for these two consumers. The accumulator redesign
-remains relevant for a future *unsampled, full-index* run, but is not
-required to ship `--subsample`/`--shannon`.
-
-## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15)
-
-The reference run (`phyloskims_sal_vac`-scale bacterial test set,
-`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites
-(4.8 MB) — trivially small. The in-progress plant index is expected to
-carry on the order of 9 billion minorant families; a dense byte-per-cell
-alignment at that column count is unbuildable regardless of genome count
-(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000–
-8,000 genomes on a large machine, which makes the per-cell cost dominant in
-the other dimension too. Pruning the retained family set before
-materializing anything is mandatory, not an optimization.
-
-**Already free**: `family_size() < 2` (no sibling variant registered at
-all) is a zero-cost structural filter, read directly off `FamilyMask` bits,
-already applied in `snp_pseudo_alignment`. Insufficient alone — per the
-~80-85% mono-family estimate from earlier discussion, this only brings 9
-billion down to roughly 1.3-1.8 billion, still unusable.
-
-**Entropy definition — settled 2026-08-15, correcting an earlier wrong
-turn.** The project does **not** encode families as IUPAC ambiguity codes
-interpreted the classical way (Fitch-parsimony subset-compatibility, or
-ML's "one true state, uncertain which"); see
-`docmd/theory/evolutionary_distances.md` ("Why the IUPAC/DNA encoding used
-for the first `--snp` test was wrong") and the Sankoff resolution that
-followed it. The real model is a genuine 16-state alphabet (the powerset of
-`{A,C,G,T}`, `∅` included as a real state) scored with a *calibrated
-pairwise cost matrix* (`obikphylo::cardcomp::pairwise_cost_matrix`,
-`cmd/phylo/sankoff.rs`), not a compatibility/subset relation between
-states. Under that model, each of the 16 states — including multi-bit ones
-like `AC` — is a first-class, independently-costed state, not an
-uncertainty encoding of a single true base. So: **entropy over the 15
-non-empty states (`∅` excluded, matching the earlier decision to exclude
-genomes where the family is absent) is the correct informativeness
-measure** for this project — not a 4-symbol reduction, which would discard
-exactly the cardinality/composition information the calibrated cost matrix
-is built to exploit.
-
-## `--subsample` / `--shannon` — sampling strategy (decided 2026-08-15)
-
-Goal: make both the pseudo-alignment (`--snp`) and a Shannon-entropy
-diagnostic usable at any index scale, from the 13-genome bacterial
-reference run up to the 9-billion-family plant index, without requiring the
-batched-accumulator redesign above.
-
-**`--subsample N`** (integer, families to retain): bounds the pseudo-alignment
-to `N` minorant families, sampled **proportionally per layer** among
-non-monomorphic minorants (`family_size >= 2`) — this sidesteps the need
-for a true global reservoir merge across layers while still approximating a
-uniform sample over the whole index, and directly answers the earlier open
-question of global-vs-per-layer selection scope.
-
-Three passes, in order:
-
-1. **Global count** (cheap, structural, parallel across layers — same shape
- as the existing `sibling_family_size_histogram`, extended to report a
- **per-layer** breakdown rather than one index-wide aggregate): for each
- layer, `count_layer` = number of non-monomorphic minorants. Gives
- `total_count = Σ count_layer`.
-2. **Per-layer proportional reservoir sampling** (cheap, structural, one
- pass per layer, no cross-partition resolution): `N_layer = round(N ×
- count_layer / total_count)`. Since `N_layer` is a proportion of
- `count_layer`, it can never exceed it as long as `N <= total_count` — the
- one edge case is `total_count <= N`, in which case sampling is skipped
- entirely and *every* non-monomorphic minorant of every layer is kept
- (no reservoir needed, `N` was never a real constraint). Otherwise:
- Algorithm-R reservoir sampling over the layer's non-monomorphic minorant
- indices, producing `N_layer` iteration-order indices directly, no
- intermediate full list ever materialized.
-3. **Filtered resolution** (the expensive step, the existing
- `scan_layer_families` engine, unchanged): re-scan the layer, generating
- and resolving cross-partition queries **only** for the indices selected
- in step 2 (cheap membership test against a small per-layer index set) —
- this is what keeps `--subsample` cheap even on an unsampled-scale index,
- since the cross-partition resolution volume is bounded by `N`, not by
- the layer's true size.
-
-Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir
-sampling would waste step-3's expensive resolution work on candidates later
-evicted by the reservoir. Step 1 must fully complete (every layer) before
-step 2 can start for any layer, since `total_count` is a global quantity.
-
-**`--shannon`** (no argument): emits a CSV of per-family Shannon entropy
-(15 non-empty states, `∅`/absent genomes excluded from the denominator, per
-the settled definition above). Independent of `--subsample` — entropy is
-computed and written per family as soon as its `genome_mask` resolves,
-O(1) memory per family, so it streams fine even unsampled at full index
-scale (a time cost, not a memory one). Combined with `--subsample N`, it
-delivers the original exploratory diagnostic (e.g. `--subsample 1000000
---shannon`) directly from this general machinery, rather than a
-purpose-built one-off script.
-
-Validated end-to-end (2026-08-15) against real data: `--sibling-hist` on
-`phyloskims_sal_vac` (91 real genomes, k=31/m=11, 256 partitions × 2
-layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238
-total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier
-guess, corrected here). `--subsample`/`--shannon` on the smaller 20-genome
-bacterial reference (`benchmark/global_index_presence`) produced a sample
-size within rounding of the request (99,742/100,000) and a
-[0.8,1.2)-bucket share (40.7%) matching the full unsampled population
-(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a
-real bug in `reservoir_sample_layer` (see next section), now fixed.
-
-**Bug found and fixed (2026-08-15): `family_idx` numbering mismatch.**
-`scan_layer_families`'s `family_idx` counts *every minorant* of a layer
-(monomorphic ones included, since `iter_minorants_batch` filters only on
-`is_minorant()`), not the raw annex slot (`SiblingAnnex::get(slot)` spans
-every k-mer, minorant or not) and not a counter over non-monomorphic
-minorants alone. `subsample.rs`'s `reservoir_sample_layer` originally
-stored raw slot numbers in its `HashSet` selection, which drifts
-away from `family_idx` as soon as *any* monomorphic minorant is seen —
-i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by
-tracking two separate counters: `family_idx` (every minorant, matching
-`scan_layer_families`) and `seen` (non-monomorphic minorants only, what
-Algorithm R actually samples over) — only `family_idx` values are ever
-stored in the selection set. The existing unit test never exercised this
-(its fixture has exactly one non-monomorphic family, always hitting the
-"keep everything" shortcut) — a stronger fixture with several interleaved
-monomorphic/non-monomorphic families would be needed to catch a regression
-here automatically; not yet written.
-
-## Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15)
-
-Motivation: on real data (bacterial reference, full unsampled run), only
-~5‰ of non-monomorphic minorants fall in the `[0.5, 1.5]` bit band judged
-phylogenetically interesting (entropy too low = uninformative near-invariant
-site; too high = saturated/noisy, see `family_entropy`'s 15-state
-discussion) — roughly 1 in 10,000 minorants overall. Computing exact
-entropy for every candidate just to discard 99.99% of them is wasteful at
-the full 9-billion scale.
-
-**The idea**: `PersistentBitMatrix::col_view(c)` gives a genome's whole
-presence column as a `BitSliceView` (sequential, no MPHF, no cross-partition
-routing — purely local to one layer's own matrix). Accumulating
-`TempCompactIntVecBuilder::inc_present(col)` (already exists in
-`obicompactvec/src/builder.rs:121`, along with `add`/`min`/`max`/`diff` on
-`IntSliceView` — no new low-level API needed) over every column of a layer
-produces `coverage[slot]`: how many genomes carry each exact k-mer, in one
-sequential per-layer pass, entirely decoupled from family/sibling
-structure. Persisted once per layer, a family's members' coverage could
-then be looked up via a plain local MPHF `index()` (cheap) instead of a
-full cross-partition presence resolution (`find_presence_batch`) — the
-expensive part today is specifically the cross-partition/cross-layer
-routing to a sibling's own matrix, not the bit-reading itself, and
-`coverage[slot]` sidesteps that routing entirely by moving the cost into a
-one-time, purely local, embarrassingly-parallel build step.
-
-**Why not implemented**: `coverage[slot]` is a per-member marginal —
-summing members' coverages to approximate a family's entropy silently
-assumes no genome carries more than one member at once. It cannot
-represent or detect joint co-occurrence (a genome carrying both `A` and
-`C` at once, i.e. a combined 15-symbol state) at all, which is exactly the
-phenomenon `family_entropy`'s 15-state definition exists to capture (see
-`CardinalityTally`/`cardinality_transition_probs`, the project's own
-existing machinery for this same co-occurrence structure, built for the
-Sankoff matrix calibration). A family that is in truth uniformly `AC`
-across every carrying genome would look like a well-balanced 2-state split
-under the marginal approximation (entropy ≈ 1) while its true 15-state
-entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the
-"saturated, uninformative" tail this pre-filter would need to catch,
-undermining the point. It stays plausible as a coarse filter for the
-*low* tail only (a dominant single member's marginal share reliably
-predicts low true entropy too), but not as a stand-in for the high tail —
-not pursued further for now.
-
-## `--free-loss`/`--tnt` pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below)
-
-Measured on `phyloskims_sal_vac` (91 genomes): `obikmer phylo --subsample 500000
---free-loss --tnt` logs four sequential stages —
-`raw_snp_distance` (1413s), `base_pair_tally` (1456s),
-`cardinality_tally` (2078s), `snp_pseudo_alignment` (143s). Reading the
-code (`obikmer/src/cmd/phylo/mod.rs:210-242`,
-`obikphylo/src/siblings/distance.rs`, `cardinality.rs`, `alignment.rs`)
-surfaced two compounding problems, not one:
-
-1. **Four separate full scans of the annex**, each opening its own
- `KmerPartition`/`PartitionCache` and calling `scan_family_pairs`/
- `scan_layer_families` independently — nothing computed in one stage is
- reused by another. `base_pair_tally` is explicitly documented
- (`distance.rs:138-143`) as a second full pass over the same
- traversal `raw_snp_distance` already did, needed only because
- `raw_snp_distance` doesn't keep the resolved bases, only aggregate
- counts. `cardinality_tally` and `snp_pseudo_alignment` are each a
- third and fourth independent full pass. Per the module's own earlier
- profiling note (`family_scan.rs:26-28`, cited already above), this
- traversal is page-fault/mmap-bound, not compute-bound — the ~10-12%
- CPU efficiency ("contention" status) observed on these three slow
- stages is consistent with I/O stalls scaled by repeated full scans,
- not lock contention (there are no `Mutex`/`RwLock` anywhere in
- `siblings/*.rs`; shared writes use per-slot `AtomicU8::fetch_or`).
-2. **Worse: `raw_snp_distance` and `cardinality_tally` don't honor
- `--subsample` at all** — both call `scan_layer_families` with
- `Selection::All` hardcoded, and `args.subsample` isn't even threaded
- into their function signatures (`mod.rs:212`, `mod.rs:226`). Only
- `snp_pseudo_alignment(args.subsample)` builds a real reservoir-sampled
- `Selection::Some(set)` (via `compute_selections`, `alignment.rs:89`).
- So today, `--subsample 500000` only bounds the pseudo-alignment step —
- the SNP-distance matrix, the Sankoff base-pair calibration, and the
- cardinality histogram are always computed over the **full, unsampled**
- index regardless of the flag. This is not merely "different subsamples
- per stage" (which would already be a problem worth fixing) — it's that
- three of the four stages never subsample, which explains most of the
- ~10x runtime gap against `snp_pseudo_alignment` on its own.
-
-**Decided requirement**: all four stages consume **one shared selection**,
-computed once, not each stage either scanning everything or drawing its
-own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair
-calibration, the cardinality histogram, and the pseudo-alignment all
-describe the same set of families — the calibration and the alignment it
-calibrates are now guaranteed to agree on which sites exist.
-
-**Correction to the "single pass" framing above**: `base_pair_tally`/
-`cardinality_tally` both need `raw_snp_distance`'s *complete* aggregate
-SNP/shared counts before they can derive `included[i,j]` (the
-`ratio_ceiling` filter) — a genuine sequential dependency (`included`
-can't be known until every pair's aggregate count is final), not an
-artifact of the old code's structure. So the fix is **two** passes over
-the shared selection, not one: pass A computes the aggregate counts (and
-derives `included`); pass B fuses `base_pair_tally` + `cardinality_tally`
-+ the pseudo-alignment (mutually independent once `included` is known)
-into a single scan. Still a 4→2 reduction, and — per the clarification
-that settled this — pass A itself now runs over the *same shared
-selection* pass B uses (not the full unsampled index): "les ratios, on
-les fait sur les sites sélectionnés, c'est tout, les autres sites
-n'existent pas" — once a selection is chosen, both passes are bounded by
-it, so on a real `--subsample`/`--entropy` run pass A is cheap too, not
-just pass B.
-
-**Entropy-biased selection's own resolution to the "forward-looking
-complication"** (entropy must be known before selection, but selection
-happens during the same scan that would resolve it): see "Entropy-biased
-selection" below — resolved via a persisted per-layer entropy annex, not
-by restructuring the scan into an inline pre-pass.
-
-## Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15)
-
-Refines the "forward-looking complication" above with a concrete
-mechanism. Instead of a hard `[low, high]` entropy band (or any other
-exact cutoff) deciding which non-monomorphic minorant families are
-eligible, selection is weighted by an **unnormalized Gaussian kernel**
-centered on a target entropy: `w(entropy) = exp(-(entropy - μ)² / (2σ²))`
-— deliberately not the normalized Gaussian density (which would peak
-below 1 and complicate the "probability" reading) — this kernel form
-equals 1 exactly at `entropy = μ` and decays smoothly to 0 away from it,
-so it reads directly as an acceptance weight: the further a family's
-entropy from the target, the less likely it is picked, with no hard
-in/out boundary — a few "bad" sites can still get in, by design. `μ`
-(default ~1.0) and `σ` (default ~0.5) are meant to be user-tunable.
-
-**Mechanism: joint probability, not a weighted reservoir**. Not
-Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex
-proposal, superseded before implementation) — instead, a single
-independent accept/reject draw per qualifying candidate: draw
-`u ~ Uniform(0,1)`, accept iff `u < p₀ · w(entropy)`. `p₀` is a single
-**index-wide** rate, `N / total_count` (`total_count` = the sum of
-`non_monomorphic_counts` across every layer, `N` = `--subsample`'s
-target), applied identically at every layer — this alone already gives
-each layer its proportional share (the same effect the old uniform
-reservoir's explicit per-layer `n_layer = N · count_layer / total_count`
-computation achieved, but without needing to compute it: applying one
-rate uniformly is mathematically the same as proportioning per layer).
-`p₀ = 1.0` when there is no `--subsample` at all — `--entropy` alone is a
-pure soft entropy filter over the whole index, no size target. Properties:
-(1) **strictly generalizes the existing uniform sampler** — with `σ` large
-enough that `w ≈ 1` everywhere, this reduces to the old uniform `N/total_count`
-draw; (2) yields "approximately N", not exactly N — expected accepted
-count is `N · mean(w)`, always `≤ N` — an intentional relaxation, matching
-"sous-échantillonnage à environ n" rather than the old reservoir's exact-N
-guarantee; (3) one streaming pass, one random draw per candidate, no
-reservoir state.
-
-**Resolving "entropy must be known before selection, but selection
-happens during the same resolving scan"**: solved with a **persisted,
-per-layer entropy annex** (`obikphylo/src/siblings/entropy_annex.rs`,
-`EntropyAnnex`/`EntropyAnnexBuilder`), not by restructuring the scan.
-Mirrors `SiblingAnnex`'s mmap-backed, read-only-after-build convention,
-but indexed by `family_idx` (every minorant of the layer, monomorphic
-included — the same numbering `Selection`/`scan_layer_families` already
-use), one `f32` entropy15 value per entry, `-1.0` sentinel for monomorphic/
-not-yet-computed. First use of `--entropy`/`--entropy-sd` on an index
-pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`) — every
-later run (any `μ`/`σ`, any command) reads the file positionally, no
-re-scan, restoring the usual `Selection::Some` "skip resolving excluded
-families" speedup that a naive "weigh during the resolving scan" design
-would have permanently forfeited.
-
-**Bug found and fixed (2026-08-15): `ensure_entropy_annexes` scanned with
-`Selection::All` instead of bounding to non-monomorphic minorants.**
-Monomorphism (`family_size() < 2`) is knowable directly from the annex
-bits alone, no per-genome resolution needed — but the original
-implementation called `scan_layer_families` with `Selection::All`
-anyway, so `fill_sub_matrix_carries` (the expensive per-genome
-resolution) ran for *every* minorant, ~98% of which are monomorphic
-(measured elsewhere in this doc) and had their `genome_mask` immediately
-discarded once the callback checked `family_size() < 2`. Fixed by adding
-[`subsample::non_monomorphic_selection_layer`] — a cheap, annex-only,
-non-sampling pass (same shape as `reservoir_sample_layer`, but keeping
-every non-monomorphic minorant's `family_idx` instead of a bounded
-reservoir) — and passing `Selection::Some(&eligible)` instead of
-`Selection::All`, so the expensive resolution now runs only for the ~2%
-of minorants that can actually produce a real entropy value. A
-`debug_assert!(mask.family_size() >= 2, ...)` inside the
-`scan_layer_families` callback guards the invariant.
-
-**Resolved**: the existing hard "non-monomorphic minorant" eligibility
-filter stays a hard gate upstream of the Gaussian weighting — only
-qualifying families ever get a stored entropy value or a weighted draw.
-
-**CLI, implemented**: two `phylo` options, `--entropy <μ>` and
-`--entropy-sd <σ>` (`obikmer/src/cmd/phylo/args.rs`). The entropic filter
-activates as soon as *either* is given (`mod.rs`, computed once into an
-`Option` threaded through `--snp`/`--family-overlap`/
-`--shannon`/the fused sankoff pipeline below). If active but one or both
-are unset, defaults are `μ = 1.0`, `σ = 0.5`.
-
-## Two entropy definitions kept side by side, for comparison (2026-08-15)
-
-`--shannon`'s CSV carries both `entropy15` (`family_entropy` — the settled
-15-non-empty-state definition, see above) and `entropy4` (`family_entropy_4`
-— plain nucleotide reduction), computed from the same already-resolved
-`genome_mask`, not from the marginal approximation above. A genome carrying
-several bases at once contributes to *each* base's count (counted once per
-base present, not fractionally split, not folded into one combined state)
-— a genome polymorphic for the family is present at more than one base by
-construction, so it is expected to count more than once; the denominator is
-the total base-occurrence count, not the genome count (the two coincide
-only when no genome carries more than one base). Kept side by side
-specifically to measure, on real data, how much the two diverge — not yet
-analyzed.
-
-## `PersistentSparseBitMatrix` — implemented and measured (2026-08-15)
-
-A row-major (k-mer-major), deduplicated sparse alternative to
-`obicompactvec::PersistentBitMatrix`, motivated by the same sparsity that
-drove `--subsample`/`--shannon` above, but pursued as a foundational
-storage-layer change rather than an index-level workaround. Full design
-history, rationale, and rejected alternatives (external Elias-Fano crates,
-`cacheline-ef`, a single unsplit `dict_id` array) are in the dedicated
-implementation plan (`vivid-mapping-tiger.md` at the time of writing — the
-content below is the durable summary, not a pointer to a session-scoped
-file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi,
-"Compact Data Structures for Collections of Sets" (2025,
-`biblio/Alanko et al. - Compact Data Structures for Collections of
-Sets.pdf`) — this design implements only their exact-duplicate special
-case (a plain dedup dictionary), not their full subset-containment
-hierarchy.
-
-**Design**: four on-disk components, each mmap-backed, built once per
-layer (matching how the rest of the build pipeline already works — never
-the whole multi-billion-row index at once): an `is_multi` rank-capable
-flag per row (singleton vs. multi-genome), a fixed-bit-width array for
-singleton rows (genome index directly, `ceil(log2(n_cols))` bits), a
-separate fixed-bit-width array for multi-genome rows (`dict_id`,
-`ceil(log2(n_distinct_multi_sets))` bits — kept apart from the singleton
-array specifically because `n_distinct_multi_sets` can be large in
-absolute terms even when multi-genome rows are a small *fraction* of all
-rows, and a single shared array would force every row, singletons
-included, to pay the wider width), and a deduplicated dictionary of
-distinct multi-genome sets (Elias-Fano-encoded byte offsets + a
-varint-encoded values blob). New low-level primitives added to
-`obicompactvec` to build this: `PersistentFixedIntVec` (arbitrary,
-runtime-parameterized bit width, width 0 included — needed once a real
-bug surfaced, see below), `PersistentRankSelectBitVec` (rank1/rank0/select1
-on top of the crate's existing `count_ones`, using
-`common_traits::SelectInWord`), `EliasFano` (composes the two). A new
-`BinaryMatrix` trait (`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`,
-`count_ones`) unifies dense and sparse at the one call site that needs
-both interchangeably (`obikphylo::siblings::cache::Mat`) — column-oriented
-methods (`col`, `col_view`, the `partial_*_dist_matrix` family) stay
-dense-only.
-
-**Two real bugs caught by tests, not by inspection**: (1) `EliasFano::open`
-re-derived its low-bits width from the persisted low-vector file's own
-width byte; the zero-width case was built with a dummy 1-bit placeholder
-(the builder rejected true width 0), so every reopened value silently
-doubled. Fixed by making `PersistentFixedIntVec` genuinely support width 0
-(no storage, `get` always 0) instead of working around the limitation in
-`EliasFano`. (2) An empty row (cardinality 0 — not expected on a real
-built index, but not guarded against either) was recorded as a singleton
-at genome 0, indistinguishable on read-back from a *real* singleton at
-genome 0. Fixed by routing cardinality-0 rows through the dictionary path
-(a genuine empty entry) instead of the singleton shortcut. Both caught by
-`obicompactvec`'s test suite (142 tests, including disk-reopen round-trips
-that drop every builder/mmap before reopening fresh), not by manual
-review — worth remembering next time a "this edge case can't happen in
-practice" shortcut is tempting.
-
-**Measured on real data** (`layer_1` of `phyloskims_sal_vac`'s
-`part_00018`, 30,246,774 rows, 91 genomes — `#[ignore]`d benchmarks in
-`obikphylo/src/siblings/tests.rs`):
-
-| | dense | sparse | ratio |
-|---|---|---|---|
-| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
-| build time / peak RSS | — | 4.26s / 628MB | (per-layer, in-memory construction — comfortable) |
-| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse **faster** (smaller structure, better cache fit) |
-| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** (the real `--shannon`/family-lookup shape) |
-| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** (no native column method — every read decodes a full row to keep one bit) |
-
-The row-access wins (both directions) weren't the design's stated goal —
-compactness was — but turn out real: dense's genome-major layout scatters
-a single row read across a much bigger file, which costs more than
-sparse's rank/select/varint decode once the file is this much smaller.
-The column-access cost is the flip side of the same layout choice, and is
-exactly what the next item below exists to fix.
-
-**Next, not yet planned**: rewrite `partial_jaccard_dist_matrix`/
-`partial_hamming_dist_matrix`/etc. (`obicompactvec/src/bitmatrix/pairwise.rs`)
-as a row-major co-occurrence accumulation (`O(Σ_rows k²)`, per-row
-increments into an `NxN` genome-pair counter — the known alternative to
-today's column-fold, plausibly cheaper on data this sparse, not just a
-fallback) so `obikindex`'s `--metric`/distance-matrix path can use the
-sparse type without the measured 86x column-access penalty. Needs its own
-design pass (in particular how it plugs into the `BitPartials`/
-`ColumnWeights` traits so both matrix types keep serving `--metric`)
-before implementation — not just "port the loop", a genuinely different
-algorithm.
-
-Also still deferred, unchanged from the implementation plan: full Alanko
-et al. subset-hierarchy compression (only the exact-duplicate special case
-is built), a sparse `PersistentCompactIntMatrix` (count matrices), and
-BRWT-style column-correlation exploitation.
-
-## Wired into `pack` and the sibling-annex build path (2026-08-15)
-
-`PersistentSparseBitMatrix` went from a validated but unused type to a
-real, selectable on-disk format:
-
-- **Generic `Layer`**: `obikindex::layer::Layer`'s presence-only methods
- (`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
- `D: LayerData- > + BinaryMatrix`, not hardcoded to
- `PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
- `LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
- already generic over any `D: LayerData`, so they needed no change.
- Verified by `obikindex::layer`'s
- `presence_layer_generic_over_sparse_matches_dense` test: build a dense
- presence layer, convert it to sparse via `build_from_dense`, open both
- as `Layer
`/`Layer` on
- the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree.
- (This test must stay at `k=4` with mutually non-colliding canonical
- 4-mers across its input sequences — `K`/`M` are process-wide
- `AtomicUsize`s in test builds, not thread-local, so a test using a
- different `k` races every other test in the same crate binary; a k=11
- version of this test passed alone but failed under the full
- `obikindex::layer` suite for exactly that reason before being fixed.)
-- **`obikphylo::siblings::cache::Mat`** gained a third variant,
- `SparsePresence(Layer)`, alongside `Count`
- and `Presence` — every method (`find_slot`, `index_batch`,
- `iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches
- to it identically to `Presence`, since both go through the same generic
- `Layer` code. `PartitionCache::build` picks the variant per layer by
- checking for `presence/is_multi.prsb` (the sparse format's own marker
- file, see the design section above) before falling back to the dense
- open path.
-- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`):
- `pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb`
- first); packs to dense `matrix.pbmx` first if that hasn't happened yet
- (the dense→sparse transpose needs random row access, which only the
- packed/columnar dense forms give), then `build_from_dense`s the sparse
- form into the same directory and deletes `matrix.pbmx` — old-format
- files are removed only after the new format is fully written, mirroring
- `pack_bit_matrix`'s own crash-safety convention.
-- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through
- `KmerIndex::pack_matrices` (all other call sites — `select`, `merge`,
- `finalize_indexed` — pass `false`, unchanged dense behaviour). Count
- matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix`
- — see "still deferred" above).
-- **End-to-end coverage**: `obikphylo::siblings::tests::
- sibling_annex_works_after_pack_sparse` builds a two-genome index, packs
- it `--sparse`, asserts `is_multi.prsb` exists, then runs
- `build_sibling_annex` and checks the resulting `FamilyMask`s match the
- dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
- sparse format round-trips through the real build pipeline
- (`PartitionCache` sparse-detection included), not just the
- `obicompactvec`/`obikindex::layer` unit layers below it.
-
-Full workspace `cargo test` (all crates, unit + doc tests) green after
-this change.
-
-## Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15)
-
-Replaces the "four independent scans" problem above and implements
-"Entropy-biased selection" above, end to end:
-
-- **`SankoffBundleExt::sankoff_bundle`** (new,
- `obikphylo/src/siblings/sankoff_bundle.rs`) — the `--sankoff`/`--tnt`/
- `--phyg`/`--iqtree` block in `obikmer/src/cmd/phylo/mod.rs` now calls
- this once instead of three separate `raw_snp_distance`/
- `base_pair_tally`/`cardinality_tally` calls. One `PartitionCache`, one
- shared (possibly subsampled/entropy-biased) selection computed once via
- `compute_selections`, two scans over it: pass A (aggregate SNP/shared
- counts, `--exclude-genome` zeroing, then `included[i,j]`), pass B
- (`base_pair_tally` + `cardinality_tally` + the pseudo-alignment, fused
- into one `scan_layer_families` call per layer, all three read off the
- same resolved `genome_mask`). `snp_pseudo_alignment`/
- `shannon_entropy_csv` (still used standalone by `--snp`/
- `--family-overlap`/`--shannon`) both gained an `entropy_bias` parameter
- too, so entropy-biased selection isn't sankoff-specific.
-- **Regression proof**: `sankoff_bundle_matches_old_separate_calls`
- (`obikphylo/src/siblings/tests.rs`) asserts `sankoff_bundle`'s four
- outputs are bit-identical to calling the old, separate
- `raw_snp_distance`/`base_pair_tally`/`cardinality_tally`/
- `snp_pseudo_alignment` on the same fixture with no subsample — the
- fusion is a performance change, not a behavior change.
-- **`EntropyAnnex`/`EntropyAnnexBuilder`** (new,
- `obikphylo/src/siblings/entropy_annex.rs`) and **`ensure_entropy_annexes`**
- (`entropy.rs`) implement the persisted-entropy mechanism from
- "Entropy-biased selection" above. `entropy_annex_builds_on_demand_and_biases_selection`
- (`tests.rs`) proves, on a fixture with one known-entropy family: the
- annex file doesn't exist before any entropy-biased call; `compute_selections`
- builds it on first use; `μ` set to the family's exact entropy with
- `p₀ = 1.0` selects it deterministically (`u < 1.0` always, for
- `u ∈ [0,1)`); `μ` set far away with tiny `σ` deterministically excludes
- it (`w` underflows to exactly `0.0`); the persisted value matches
- `--shannon`'s own `family_entropy` computation to `1e-6`.
-- **`EntropyBias`** (`pub`, `obikphylo::siblings::EntropyBias { mu, sigma }`)
- is the one new public type threading `--entropy`/`--entropy-sd` through
- every `Option`-accepting method — resolved once in
- `obikmer/src/cmd/phylo/mod.rs` from `args.entropy`/`args.entropy_sd`
- (activation: either given; defaults `1.0`/`0.5` for whichever is unset).
-
-Full workspace `cargo test` green after this change (167 unit tests in
-`obicompactvec`+`obikindex::layer`+`obikphylo` alone, plus every other
-crate's suite, no regressions).
-
-**Still open, not part of this change** (per "Correction to the 'single
-pass' framing" above): `--raw-snp-distance`/`--raw-snp-counts` (the
-standalone diagnostic flags, not the `--sankoff` pipeline) still always
-scan the full unsampled index — never threaded `--subsample`/`--entropy`,
-out of scope here since the reported problem was specifically about the
-`--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these
-two standalone flags.
-
-## `query` never benefits from sparse row-major access (found 2026-08-19, not implemented)
-
-Benchmarked `obikmer query` against `global_index_presence` (dense-packed)
-vs. `global_index_presence_sparse` (`pack --sparse`), 100k simulated reads
-× 2 specimens (`benchmark/`, see
-[benchmark_query_testing.md](../implementation/benchmark_query_testing.md)).
-Correctness: 0 mismatches — sparse and dense return bit-identical query
-results. Performance: sparse consistently *slower* than dense (~30-50%,
-reproducible across two runs with warm disk cache), the opposite of
-`pack --sparse`'s stated intent ("faster for single-row access... like
-query").
-
-**Root cause, read from source, not measured in isolation:**
-`KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs:155-220`)
-is architecturally column-major: stage 2 walks `for g in 0..n_cols { for
-slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly)
-as the right locality strategy for the packed/columnar formats, where
-`col_value` → `PersistentBitMatrix::get` is a genuine O(1) mmap'd column
-read (`persistent.rs:110-113`).
-
-For `Self::Sparse`, that same `get(c, slot)` (`persistent.rs:114-118`)
-allocates a full `n_cols`-wide buffer and calls `fill_row` — materializing
-the *entire row* — just to return one cell. Called from inside the
-column-major double loop, this reconstructs the same row once per genome
-column touched: O(hits × n_cols) full-row rebuilds instead of O(hits).
-`PersistentSparseBitMatrix`'s own native row-major decode
-(`for_each_genome_in_row`, `sparse.rs:164-177`, used correctly by its own
-`row`/`fill_row`/`fill_sub_matrix`) is never reached from the query path
-at all.
-
-**`fill_sub_matrix` (the existing `BinaryMatrix` trait primitive,
-`traits.rs:13-37`) is not the right replacement for `query` either**, even
-once its own dispatch bug is fixed (see next section) — its output shape
-is inherently column-dense: `out[col]` gets an entry for every column,
-including columns with zero hits among the requested slots. On real
-sparse data (a hit typically touching a handful of genomes out of dozens)
-that's still O(n_cols) output regardless of true sparsity. What `query`
-actually wants is the sparse triple stream `(slot, col, value)` it already
-consumes as `QueryHit::Value` — not a materialized sub-matrix.
-
-**Proposed primitive** (design only, not implemented — explicit ask: keep
-count matrices *not excluded*, even though effort right now is
-presence/absence only):
-
-Not a closure-driven `for_each` — a real `Iterator`, one concrete struct
-per matrix format, so the traversal state (current position in the sorted
-slot list, current column, permutation, sparse-row decode cursor…) lives
-in named struct fields instead of being threaded implicitly through
-recursion or a captured closure. RPITIT (stable since 1.75, and this
-workspace is edition 2024) means the trait method can return it without
-naming or boxing the concrete type:
-
-```rust
-/// Yields every nonzero cell among `slots`, in implementation-defined order.
-fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator- + 'a;
-// item: (idx into `slots`, col, value)
-```
-
-This is the one foundational primitive per format — both `fill_sub_matrix`
-and a `for_each`-style callback become trivial, free consumers of it
-(`.for_each(f)` is already `Iterator::for_each` from std; `fill_sub_matrix`
-becomes "drain the iterator, scatter into `out[][]`"), instead of two
-independently-maintained traversals that can silently diverge (see the bug
-below — this is exactly how it happened).
-
-- **On `PersistentSparseBitMatrix`**: the struct is nearly free to write —
- it wraps the existing (currently private) `for_each_genome_in_row`
- per-row decode, advancing to the next `slots` entry on exhaustion. O(Σ
- row nnz), zero `n_cols`-wide allocation.
-- **On `PersistentBitMatrix::{Packed,Columnar}`**: revised — cheaper than
- first thought, by reusing the same split already used for
- `fill_matrix`'s own implementation instead of hand-writing a resumable
- state machine at the matrix level. The base-vector layer
- (`BitSliceView`, `views.rs`) already separates the two concerns:
- `fill_batch_sorted` (`views.rs:55-60`, sorted-slot batch lookup) and a
- genuine per-bit `Iterator` (`BitSliceIter`, `views.rs:94+`) sit side by
- side there, one level below the matrix. Adding a
- "positions among `sorted_slots` where the bit is set" iterator at that
- same vector level is a `filter` over the existing `get()` — no new
- state machine, since `std::iter::Filter` already *is* one, generated by
- the compiler:
- ```rust
- fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator
- + 's {
- sorted_slots.iter().copied().filter(move |&slot| self.get(slot))
- }
- ```
- The matrix-level `nonzero_iter` then composes these per column with
- `flat_map` over `0..n_cols` (each column's hits, tagged with `c`,
- slot mapped back through the sort permutation `fill_batch`/
- `fill_batch_sorted` already carry) — again a combinator chain, not a
- hand-rolled struct. Same algorithm, same mmap/sort locality as today's
- `fill_sub_matrix`; just assembled from `std` iterator adaptors instead
- of a loop body writing into a buffer, mirroring the vector/matrix split
- the codebase already uses for `fill_batch_sorted` rather than
- introducing a new shape.
-- **`Implicit`**: trivial (`slots.iter().map(|&i| (i, 0, 1))`, one column,
- always present).
-- **On `PersistentCompactIntMatrix` (counts)**: same treatment as
- `Packed`/`Columnar` — no sparse count format exists yet ("Explicitly
- deferred" per `traits.rs:9-12`), so no native low-effort case the way
- `Sparse` has one, but not excluded either: the iterator's `Item` is
- already `(usize, usize, u32)`, not `bool`, specifically so presence
- (`0`/`1`) and counts (arbitrary `u32`) share one primitive instead of a
- bool/u32 split forcing counts out of the design. Ready for a native
- sparse-count struct later without a signature change.
-
-Would let `query_partition_with`'s stage 2 collapse to one
-`for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) }`
-per layer, format-agnostic, each backend's struct deciding the actual
-traversal.
-
-**This also closes the existing dispatch bug for free, by construction**:
-`PersistentBitMatrix::fill_sub_matrix` (`persistent.rs:190-215`, the enum
-wrapper backing `BinaryMatrix`'s trait impl) today does *not* delegate to
-`PersistentSparseBitMatrix::fill_sub_matrix` for `Self::Sparse` — it
-reimplements the same naive per-(column, slot) `fill_row_bool` loop
-instead, bypassing the efficient native method one file over
-(`sparse.rs:249-258`). `obikphylo::siblings::cache::Mat`
-(`cache.rs:138-145`) independently built its own parallel enum wrapper
-that dispatches correctly — evidence this was worked around rather than
-fixed at the source: two hand-written traversals for the same format,
-free to drift apart, and they did. If `fill_sub_matrix` itself is
-rewritten as "drain `nonzero_iter`, scatter into `out[][]`", there is only
-one traversal per format left to get right — the bug class doesn't just
-get fixed once, it stops being possible to reintroduce.
-
-## Implemented (2026-08-20)
-
-Built as designed above, with one deviation from the original sketch:
-`nonzero_iter` ended up `Box
>`, not a bare `impl
-Iterator`, because `Columnar`/`Packed`/`Sparse`/`Implicit` are genuinely
-different concrete types and this method isn't on a trait (kept off
-`BinaryMatrix` deliberately — that trait is used as `dyn BinaryMatrix` in
-`tests/sparse.rs`, and RPITIT methods aren't dyn-compatible). One `Box`
-per `nonzero_iter` call, not per cell — negligible next to what it
-replaces.
-
-- `BitSliceView::nonzero_among_sorted` / `IntSliceView::nonzero_among_sorted`
- (`obicompactvec/src/views.rs`): the vector-level `filter`/`filter_map`
- primitive, exactly as sketched — no new state machine, `std`'s own.
-- `PersistentSparseBitMatrix::nonzero_iter` (`bitmatrix/sparse.rs`): native,
- `std::iter::from_fn` over one buffered row at a time via the existing
- `for_each_genome_in_row` — no `n_cols`-wide allocation, ever.
-- `PersistentBitMatrix::nonzero_iter` (`bitmatrix/persistent.rs`): dispatches
- to the above for `Sparse`; for `Columnar`/`Packed`, loops columns,
- collects each column's `nonzero_among_sorted` hits via `.extend()` (not
- `flat_map` — a `flat_map` closure can't lazily return something
- borrowing its own captured sort permutation across separate calls
- without either boxing per-column or fighting the borrow checker; eager
- collection into one `Vec` sidesteps it, at zero cost since
- `fill_sub_matrix` already fully materialized anyway). `Implicit` trivial.
-- `PersistentBitMatrix::fill_sub_matrix` and `sub_matrix` rewritten to
- drain `nonzero_iter` — the dispatch bug is gone because there is now
- only one traversal per format, not because the old one was patched.
- `PersistentCompactIntMatrix::nonzero_iter` added the same way (counts
- not excluded, per the earlier ask) — no native low-effort case, since no
- sparse count format exists, but on the same primitive, ready for one.
-- `KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs`):
- stage 2's column-major `for g { for slot { col_value } }` replaced by one
- `layer.nonzero_iter(&slot_list)` call per layer, format-agnostic.
-- Tests: `nonzero_iter_matches_dense`, `nonzero_iter_matches_row`, and —
- the one that actually targets the dispatch bug rather than each type's
- own correctness — `enum_wrapper_dispatches_to_native_sparse` (builds
- `PersistentBitMatrix::Sparse(...)` directly, not through `open`, since
- `open` only auto-detects `Sparse` from a `presence/` dir layout).
- `cargo test --workspace`: green, no regressions.
-
-**Measured**: re-ran the `benchmark/` query branch (100k reads × 2
-specimens, same setup as the original finding). Correctness still 0
-mismatches. The dense/sparse performance gap is gone — previously sparse
-~30-50% slower than dense, reproducibly; now within ~1-3% either way
-(7.42s dense vs 7.60s sparse for `Escherichia_coli--K-12_MG1655`; 5.25s vs
-5.30s for `Saccharolobus_islandicus--M.16.4`) — noise-level, not a
-systematic gap. `pack --sparse`'s claimed query win isn't confirmed
-outright by this (sparse should arguably now *beat* dense on truly sparse
-real data, not just tie), but the pathological regression is fixed.
-
-## `PersistentCompactIntMatrix::Sparse` — implemented (2026-08-26)
-
-Closes the gap flagged throughout this document ("no sparse count format
-exists yet", `traits.rs:9-12`'s "Explicitly deferred"): `obicompactvec`
-already had `PersistentSparseCompactIntMatrix` (row-major, built on top of
-`PersistentSparseBitMatrix` as its "which columns are non-zero" support,
-values *not* deduplicated — see that struct's own doc comment), but it was
-never wired into `PersistentCompactIntMatrix`, the dense-dispatching enum
-every real consumer (`TypedLayer`,
-`KmerLayer::Count`) actually holds. Concretely: `kmer_index.rs::
-pack_matrices(sparse=true)` already called `pack_sparse_compact_int_matrix`
-on every layer's `counts/` — but `PersistentCompactIntMatrix::open` had no
-code path back to what that just wrote, so a `Count` layer became
-unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment
-anyone ran `pack --sparse` on an index with count layers. Root cause, not a
-workaround: add the missing `Sparse` variant.
-
-- **Enum + dispatch** (`intmatrix.rs`): `PersistentCompactIntMatrix::Sparse
- (PersistentSparseCompactIntMatrix)`, detected in `open`/`detect_storage`
- via a `singleton_values.pciv` marker (mirrors `PersistentBitMatrix`'s own
- `sparse_meta.json` check), reported via `storage_kind()`. `col`/
- `col_view`/`col_persist` panic/`Unsupported` on `Sparse`, same convention
- as the bit side. `sub_matrix`/`fill_sub_matrix` and `nonzero_iter`
- unified the same way `PersistentBitMatrix`'s already are (drain
- `nonzero_iter`, one traversal per format — see "Implemented
- (2026-08-20)" above); `nonzero_iter` had to become `Box>`
- for the same reason (`Columnar`/`Packed`/`Sparse` are different concrete
- types). No change needed in `obikindex` at all — `KmerLayer::Count`
- already only ever holds `TypedLayer`, so the
- enum absorbing `Sparse` fixes the unreadable-layer bug for free, same as
- `PersistentBitMatrix::Sparse` already did on the presence side.
-
-- **`CountPartials`, non-naive** (`sparse_intmatrix.rs`): unlike
- `PersistentSparseBitMatrix`'s dict-driven `col_weights_and_pair_counts`,
- values here aren't deduplicated (two rows can share the same non-zero
- column set via the same `dict_id` while carrying different counts), so
- the "weight by how many rows share a dict entry" shortcut doesn't carry
- over. What does: a single row-major pass (`row_major_pairwise`, decodes
- each row once via `for_each_cell_in_row`, nests over that row's own
- co-present columns) — `O(Σ k̄²)` over populated rows instead of the naive
- `O(n_cols² × n)` column-pair rescan, same complexity class as the bit
- side minus the dict multiplicity discount. Kernels used: `min(a,b)`
- (bray, relfreq-bray — both vanish when either side is absent, so no
- correction needed), `a·b` and `√(a·b)` (euclidean/relfreq-euclidean and
- hellinger — these *do* need a correction, reconstructed from per-column
- marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`, since `(a-0)² = a² ≠ 0` unlike
- the `min`-based formulas). `threshold_jaccard(1)` shortcuts straight to
- `support`'s own `BitPartials::partial_jaccard` (threshold 1 is exactly
- presence); `threshold_jaccard(0)` is closed-form (every `u32` is `≥ 0`).
-
-- **Two pre-existing bugs found and fixed while wiring the `threshold==1`
- shortcut** (`bitmatrix/sparse.rs`, `BitPartials for
- PersistentSparseBitMatrix`, present since the 2026-08-15 implementation
- above, never caught because no test compared `Sparse`'s raw `partial_*`
- output against dense on real data — only the diagonal-blind
- `jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested):
- 1. `partial_jaccard`'s diagonal was `(0, 2×col_weights[i])` instead of a
- genuine self-comparison `(col_weights[i], col_weights[i])` —
- `col_weights_and_pair_counts`'s `inter` never pairs a column with
- itself by construction.
- 2. `partial_hamming`'s off-diagonal formula itself was wrong: `total -
- union` (count of rows where *neither* column is present) instead of
- the actual Hamming distance `col_weights[i] + col_weights[j] -
- 2×inter[i,j]` (symmetric-difference size). Only coincides with the
- correct value when `col_weights[i] + col_weights[j] == total`, so
- small/synthetic test data could easily have hidden it.
-
- Neither surfaced through `jaccard_dist_matrix`/`hamming_dist_matrix`
- (both explicitly zero their own diagonal at finalisation, and the
- off-diagonal `partial_hamming` bug had gone untested against dense
- entirely) — only visible to a caller of the raw `partial_*` methods
- directly, which is exactly what `partial_threshold_jaccard(1)`'s new
- shortcut became. Fixed at the source, not patched around at the call
- site; regression test added:
- `tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal`.
-
-- **Tests**: `tests::intmatrix::sparse_roundtrip_matches_columnar`/
- `sparse_roundtrip_from_packed` (the `open`-dispatch fix, both build
- paths); `tests::intmatrix::sparse_count_partials_match_dense` (all six
- `CountPartials` formulas, thresholds 0/1/2/3, against `Columnar` on
- asymmetric-presence data — this is what caught the diagonal gap in the
- int side's own new code before it shipped, the same way it exposed the
- two pre-existing bit-side bugs above); `obikindex`'s
- `count_layer_transparently_reads_sparse_after_pack` — the actual
- end-to-end regression test for the original "layer unreadable after
- `pack --sparse`" bug, built → packed sparse → reopened, compared against
- the pre-pack dense read. `cargo test -p obicompactvec -p obikindex`:
- green, no regressions (180 + 12 tests).
+# Sibling annex & SNP-family pipeline — architecture
+
+## Data model
+
+A **family** is the up-to-4 k-mers sharing all flanks and differing only at
+the central base (well-defined for odd `k`). `FamilyMask` (`siblingannex.rs`)
+is a 4-bit presence mask (bit `b` set iff the family member whose own
+canonical central base is `b` is observed anywhere in the index) plus a
+minorant flag; `family_size()`/`siblings()` derive member count from it.
+`SiblingAnnex`/`SiblingAnnexBuilder` persist one `FamilyMask` per k-mer,
+**keyed by iteration order** (the physical layout of `unitigs.bin`), never by
+MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream
+against the annex file directly, no MPHF/slot indirection.
+
+## Construction
+
+`algorithms::annex::build_layer_sibling_annex` builds one layer's annex:
+seeds each k-mer's own central base, generates its up-to-3 central-base
+variants, resolves each cross-partition (`IndexCache::find_in_partition`,
+grouped and chunked per destination partition for mmap locality), writes the
+final mask (with the minorant flag) in a second streamed pass.
+`SiblingExt::build_sibling_annex` drives this for every layer of every
+partition, then invalidates and rebuilds the entropy and family-overlap
+annexes (their `family_idx` numbering depends on the sibling annex).
+
+## Shared traversal
+
+`algorithms::family_scan::scan_layer_families` is the **only** place that
+resolves a family's full per-genome presence (`genome_mask: &[u8]`,
+cross-partition) — every other consumer in this module is built on it, never
+reimplements the resolution. Two decoupled concurrency mechanisms: batch
+generation (CPU-only) runs on an `obipipeline` throttle/pipe; per-batch
+resolution (I/O-bound, mmap page faults) runs `rayon`-parallel across
+destination partitions, one generated batch at a time (never several
+concurrently, to keep one partition's pages hot instead of scattering
+access). `fast_mode` (`is_fast_mode`, `<= 7` layers per partition) picks
+between trusting a family member's recorded layer index outright or probing
+every layer of the destination partition.
+
+`Selection<'a>` bounds which families of a layer are resolved:
+
+- `All` — every minorant, monomorphic included (`SiblingExt::shannon_entropy_csv`,
+ `sibling_annex_stats`).
+- `Some(&HashSet)` — a caller-picked subset, skipping the expensive
+ cross-partition resolution entirely for excluded families
+ (`subsample::sample_layer`'s accepted-candidates batch;
+ `entropy::ensure_layer_entropy_annex`'s non-monomorphic-only restriction).
+
+`minorant_selection::eligible_bitset_layer`/`non_monomorphic_reindex_layer`
+provide the cheap, annex-bits-only (no cross-partition resolution) filters
+these restricted scans are built from: monomorphism (`family_size() < 2`) is
+knowable straight from the annex, so nothing pays `scan_layer_families`'s
+resolution cost just to discard a monomorphic result.
+
+## Masking — computed once, shared everywhere
+
+`algorithms::masking::masked_state(m, free_loss, no_ambiguity)` is the single
+function that turns a raw per-genome 4-bit state into the masked view every
+consumer needs (`None` for absence under `--free-loss` or ambiguity under
+`--no-ambiguity`). It is called **exactly once per (family, genome)**, in
+`subsample::sample_layer`, immediately after `scan_layer_families` resolves
+the raw state — never recomputed downstream. `survives_masking` (does the
+family still carry ≥2 distinct masked states among included genomes?) and
+`SurvivingFamily::masked: Vec