diff --git a/DevDoc/404.html b/DevDoc/404.html index b04c63bb..5a9ae5cb 100644 --- a/DevDoc/404.html +++ b/DevDoc/404.html @@ -1119,7 +1119,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/index_architecture.refs/index.html b/DevDoc/architecture/index_architecture.refs/index.html index 78ae7534..f19ad23b 100644 --- a/DevDoc/architecture/index_architecture.refs/index.html +++ b/DevDoc/architecture/index_architecture.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/index_architecture/index.html b/DevDoc/architecture/index_architecture/index.html index 30ea415b..09497c21 100644 --- a/DevDoc/architecture/index_architecture/index.html +++ b/DevDoc/architecture/index_architecture/index.html @@ -1370,7 +1370,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/numa_partition_runner/index.html b/DevDoc/architecture/numa_partition_runner/index.html index c79dca08..653ab74a 100644 --- a/DevDoc/architecture/numa_partition_runner/index.html +++ b/DevDoc/architecture/numa_partition_runner/index.html @@ -1128,7 +1128,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/numa_worker_pools/index.html b/DevDoc/architecture/numa_worker_pools/index.html index 97421f9a..10674165 100644 --- a/DevDoc/architecture/numa_worker_pools/index.html +++ b/DevDoc/architecture/numa_worker_pools/index.html @@ -1130,7 +1130,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/query.refs/index.html b/DevDoc/architecture/query.refs/index.html index ef859997..141c734a 100644 --- a/DevDoc/architecture/query.refs/index.html +++ b/DevDoc/architecture/query.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/query/index.html b/DevDoc/architecture/query/index.html index 2ed91fb1..a2012d4a 100644 --- a/DevDoc/architecture/query/index.html +++ b/DevDoc/architecture/query/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/rebuild_filter/index.html b/DevDoc/architecture/rebuild_filter/index.html index a21d25de..04a44fdc 100644 --- a/DevDoc/architecture/rebuild_filter/index.html +++ b/DevDoc/architecture/rebuild_filter/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/sequences/invariant.refs/index.html b/DevDoc/architecture/sequences/invariant.refs/index.html index 2d5e0338..66ddfc59 100644 --- a/DevDoc/architecture/sequences/invariant.refs/index.html +++ b/DevDoc/architecture/sequences/invariant.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/sequences/invariant/index.html b/DevDoc/architecture/sequences/invariant/index.html index 38f3fb60..7ce2bce5 100644 --- a/DevDoc/architecture/sequences/invariant/index.html +++ b/DevDoc/architecture/sequences/invariant/index.html @@ -1140,7 +1140,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/siblings/index.html b/DevDoc/architecture/siblings/index.html index 8be09b91..74189217 100644 --- a/DevDoc/architecture/siblings/index.html +++ b/DevDoc/architecture/siblings/index.html @@ -23,7 +23,7 @@ - Sibling annex (discussion) - obikmer + Sibling annex - obikmer @@ -64,7 +64,7 @@
- + Skip to content @@ -100,7 +100,7 @@
- Sibling annex (discussion) + Sibling annex
@@ -1139,7 +1139,7 @@ - Sibling annex (discussion) + Sibling annex @@ -1157,7 +1157,7 @@ - Sibling annex (discussion) + Sibling annex @@ -1183,10 +1183,10 @@
  • - + - Two index spaces, uncorrelated + Data model @@ -1194,10 +1194,10 @@
  • - + - Two pipelines, never mixed + Construction @@ -1205,10 +1205,10 @@
  • - + - Sibling annex: an iteration-pipeline artifact only + Shared traversal @@ -1216,10 +1216,10 @@
  • - + - Pending work — done + Masking — computed once, shared everywhere @@ -1227,10 +1227,10 @@
  • - + - Performance: build_sibling_annex parallelism (2026-08-14) + Precomputed annexes (entropy, family overlap) @@ -1238,10 +1238,10 @@
  • - + - Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14) + Sampling and the shared scan trunk @@ -1249,10 +1249,10 @@
  • - + - Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15) + From a tally to distances and calibration @@ -1260,10 +1260,10 @@
  • - + - --subsample / --shannon — sampling strategy (decided 2026-08-15) + Public API surface @@ -1271,109 +1271,10 @@
  • - + - Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15) - - - - -
  • - -
  • - - - - --free-loss/--tnt pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below) - - - - -
  • - -
  • - - - - Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15) - - - - -
  • - -
  • - - - - Two entropy definitions kept side by side, for comparison (2026-08-15) - - - - -
  • - -
  • - - - - PersistentSparseBitMatrix — implemented and measured (2026-08-15) - - - - -
  • - -
  • - - - - Wired into pack and the sibling-annex build path (2026-08-15) - - - - -
  • - -
  • - - - - Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15) - - - - -
  • - -
  • - - - - query never benefits from sparse row-major access (found 2026-08-19, not implemented) - - - - -
  • - -
  • - - - - Implemented (2026-08-20) - - - - -
  • - -
  • - - - - PersistentCompactIntMatrix::Sparse — implemented (2026-08-26) + Known gaps @@ -1479,10 +1380,10 @@
    • - + - Two index spaces, uncorrelated + Data model @@ -1490,10 +1391,10 @@
    • - + - Two pipelines, never mixed + Construction @@ -1501,10 +1402,10 @@
    • - + - Sibling annex: an iteration-pipeline artifact only + Shared traversal @@ -1512,10 +1413,10 @@
    • - + - Pending work — done + Masking — computed once, shared everywhere @@ -1523,10 +1424,10 @@
    • - + - Performance: build_sibling_annex parallelism (2026-08-14) + Precomputed annexes (entropy, family overlap) @@ -1534,10 +1435,10 @@
    • - + - Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14) + Sampling and the shared scan trunk @@ -1545,10 +1446,10 @@
    • - + - Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15) + From a tally to distances and calibration @@ -1556,10 +1457,10 @@
    • - + - --subsample / --shannon — sampling strategy (decided 2026-08-15) + Public API surface @@ -1567,109 +1468,10 @@
    • - + - Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15) - - - - -
    • - -
    • - - - - --free-loss/--tnt pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below) - - - - -
    • - -
    • - - - - Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15) - - - - -
    • - -
    • - - - - Two entropy definitions kept side by side, for comparison (2026-08-15) - - - - -
    • - -
    • - - - - PersistentSparseBitMatrix — implemented and measured (2026-08-15) - - - - -
    • - -
    • - - - - Wired into pack and the sibling-annex build path (2026-08-15) - - - - -
    • - -
    • - - - - Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15) - - - - -
    • - -
    • - - - - query never benefits from sparse row-major access (found 2026-08-19, not implemented) - - - - -
    • - -
    • - - - - Implemented (2026-08-20) - - - - -
    • - -
    • - - - - PersistentCompactIntMatrix::Sparse — implemented (2026-08-26) + Known gaps @@ -1693,1008 +1495,151 @@ -

      Sibling annex — architecture (discussion)

      -

      Status: architecture decided (2026-08-14). Implementation not yet mandated.

      -

      Two index spaces, uncorrelated

      -

      Every kmer stored in a Layer lives in two independent index spaces:

      +

      Sibling annex & SNP-family pipeline — architecture

      +

      Current state of obikphylo::siblings (crate obikphylo, module siblings). +Historical rationale, dated findings, and superseded designs live in jj +history, not here.

      +

      Data model

      +

      A family is the up-to-4 k-mers sharing all flanks and differing only at +the central base (well-defined for odd k). FamilyMask (siblingannex.rs) +is a 4-bit presence mask (bit b set iff the family member whose own +canonical central base is b is observed anywhere in the index) plus a +minorant flag; family_size()/siblings() derive member count from it. +SiblingAnnex/SiblingAnnexBuilder persist one FamilyMask per k-mer, +keyed by iteration order (the physical layout of unitigs.bin), never by +MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream +against the annex file directly, no MPHF/slot indirection.

      +

      Construction

      +

      algorithms::annex::build_layer_sibling_annex builds one layer's annex: +seeds each k-mer's own central base, generates its up-to-3 central-base +variants, resolves each cross-partition (IndexCache::find_in_partition, +grouped and chunked per destination partition for mmap locality), writes the +final mask (with the minorant flag) in a second streamed pass. +SiblingExt::build_sibling_annex drives this for every layer of every +partition, then invalidates and rebuilds the entropy and family-overlap +annexes (their family_idx numbering depends on the sibling annex).

      +

      Shared traversal

      +

      algorithms::family_scan::scan_layer_families is the only place that +resolves a family's full per-genome presence (genome_mask: &[u8], +cross-partition) — every other consumer in this module is built on it, never +reimplements the resolution. Two decoupled concurrency mechanisms: batch +generation (CPU-only) runs on an obipipeline throttle/pipe; per-batch +resolution (I/O-bound, mmap page faults) runs rayon-parallel across +destination partitions, one generated batch at a time (never several +concurrently, to keep one partition's pages hot instead of scattering +access). fast_mode (is_fast_mode, <= 7 layers per partition) picks +between trusting a family member's recorded layer index outright or probing +every layer of the destination partition.

      +

      Selection<'a> bounds which families of a layer are resolved:

        -
      • Iteration order: its position when enumerating unitigs.bin (the - superkmer file), deterministic but arbitrary with respect to slot.
      • -
      • MPHF slot: MphfLayer::index(kmer), the number the MPHF assigns.
      • +
      • All — every minorant, monomorphic included (SiblingExt::shannon_entropy_csv, + sibling_annex_stats).
      • +
      • Some(&HashSet<usize>) — a caller-picked subset, skipping the expensive + cross-partition resolution entirely for excluded families + (subsample::sample_layer's accepted-candidates batch; + entropy::ensure_layer_entropy_annex's non-monomorphic-only restriction).
      -

      The two are not correlated by any formula. Converting from one to the other -requires either recomputing the MPHF (kmer → slot) or scanning the iteration -stream (kmer → order). There is no slot → kmer operation: the MPHF is a -one-way function, not an invertible bijection with a stored inverse. Any -method that reconstructs a kmer from a bare slot number is wrong by -construction, regardless of the mechanism used (MPHF re-hash, or evidence -decode + direct unitig read). See MphfLayer::kmer_at -(obikindex/src/layer/mphf_layer.rs) — flagged for removal, currently called -from obikphylo/siblings/build.rs and family_scan.rs (since removed — see -"Pending work" status below).

      -

      Two pipelines, never mixed

      - - - - - - - - - - - - - - - - - - - - - - - -
      origin of the kmermembership known?correct mapping
      query pipelineexternal (caller-supplied)noquery/find/find_strict — MPHF + evidence check
      iteration pipelineenumerated from this layer's own unitigs.binyes, by constructionindex/index_batch — MPHF only, no evidence
      -

      Evidence exists solely to answer "is this external kmer a member of the -layer" for the query pipeline. Using it (or the MPHF) to go the other way — -recover a kmer from a slot, or re-verify a kmer that was just produced by -iterating the layer — is a conceptual error: evidence can be probabilistic -(Approx mode), so any slot→kmer attempt is unsound in general, and -pointless even in Exact/Hybrid mode since the kmer was already known.

      -

      Sibling annex: an iteration-pipeline artifact only

      -

      The sibling annex (FamilyMask/SiblingAnnex, .psib, -obicompactvec/src/siblingannex.rs) records, per kmer, whether it is a -family minorant and which family members are present in the index. Its only -consumers (obikphylo/siblings/stats.rs, family_scan.rs) enumerate it -exhaustively (0..annex.len()); no query-pipeline code path touches it.

      -

      Decision: the annex must be persisted in iteration order, not slot -order. This lets readers zip-iterate Layer::iter_kmers() and the annex -file directly — one linear, cache-friendly pass, no MPHF/slot indirection, -no kmer_at. It also enables specialized iterators building on this zip: -minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.

      -

      Today the annex is built and stored in slot order -(build_layer_sibling_annex, siblings/build.rs): slot_kmer is populated -via (0..n_slots).map(|slot| mphf.kmer_at(slot)), and the origin slot is -threaded through the whole cross-partition reconciliation pipeline (variant -generation, query_partition_with, final mask[slot].fetch_or(...)). This -must change to iterating iter_kmers()/enumerate_kmers() and threading -the iteration index instead of the slot end to end — eliminating -kmer_at from the build path entirely, not just the read path. No -slot-indexed intermediate is needed even during construction; the -iteration-order id is sufficient throughout.

      -

      The cross-partition side of the same pipeline is unaffected: checking -whether a generated family-variant kmer exists in another partition is a -genuine query-pipeline operation (the variant's membership in the target -partition is unknown) and must keep going through -KmerPartition::query_partition_with (MPHF + evidence), never a raw -index().

      -

      Pending work — done

      -

      The plan above shipped: obikphylo (a new crate — phylo-domain extension -traits over obikindex::KmerIndex/obikindex::layer::Layer<D>, replacing the -old obikindex::siblings module) builds and reads the annex purely in -iteration order (SiblingLayerExt::iter_siblings/iter_minorants, both with -batch variants, mirroring Layer<D>'s own KmerIter/KmerBatchIter -shape). MphfLayer::kmer_at has no remaining callers.

      -

      A separate, unrelated bug surfaced during this work and was fixed -(2026-08-14): MphfLayer::enumerate_kmers_batch computed its -batch_start_index via the stdlib .enumerate() adapter, which counts -batches (0, 1, 2…), not the cumulative k-mer offset the annex is actually -keyed on — every batch past the first wrote its mask/annex entries at the -wrong iteration-order position. Fixed by tracking a running offset instead; -regression tests added (sibling_annex_no_empty_masks_after_build, -sibling_histogram_does_not_panic_on_partial_last_batch).

      -

      Performance: build_sibling_annex parallelism (2026-08-14)

      -

      Investigated on a real multi-genome run (phyloskims_sal_vac, k=31/m=11). -Baseline: mostly one active core, with short multi-core bursts — average -~3 cores.

      -

      Fixes that helped, kept:

      +

      minorant_selection::eligible_bitset_layer/non_monomorphic_reindex_layer +provide the cheap, annex-bits-only (no cross-partition resolution) filters +these restricted scans are built from: monomorphism (family_size() < 2) is +knowable straight from the annex, so nothing pays scan_layer_families's +resolution cost just to discard a monomorphic result.

      +

      Masking — computed once, shared everywhere

      +

      algorithms::masking::masked_state(m, free_loss, no_ambiguity) is the single +function that turns a raw per-genome 4-bit state into the masked view every +consumer needs (None for absence under --free-loss or ambiguity under +--no-ambiguity). It is called exactly once per (family, genome), in +subsample::sample_layer, immediately after scan_layer_families resolves +the raw state — never recomputed downstream. survives_masking (does the +family still carry ≥2 distinct masked states among included genomes?) and +SurvivingFamily::masked: Vec<Option<u8>> both consume this same +precomputed array, which is also what AlignmentReducer/PairwiseReducer +read (see below) — this is what guarantees the alignment and the pairwise +tally can never disagree about a genome's state at a given site.

      +

      Precomputed annexes (entropy, family overlap)

      +

      algorithms::entropy::ensure_layer_entropy_annex builds EntropyAnnex +(entropy_annex.rs, one f32 Shannon-entropy value per non-monomorphic +minorant, mmap-backed) and/or feeds FamilyOverlapAccumulator +(family_overlap.rs) from one shared Selection::Some scan — both need the +same genome_mask resolution, so a missing one never pays for it twice. +family_entropy (15 non-empty states of {A,C,G,T}'s powerset, ∅ +excluded from the denominator — the project's settled informativeness +definition, matching the calibrated 16-state Sankoff cost matrix) and +family_entropy_4 (plain 4-symbol reduction, kept only for comparison) are +pure functions of genome_mask. extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes +drives this for every cached layer, lazily (only what's missing).

      +

      Sampling and the shared scan trunk

      +

      algorithms::subsample::sample_index draws up to n non-monomorphic +families index-wide, proportionally per layer: for each layer, a target +quota sized from that layer's share of the index-wide eligible count, filled +by a circular walk with a Bernoulli(p) draw per candidate — p optionally +weighted by an unnormalized Gaussian kernel over the candidate's entropy +(EntropyBias { mu, sigma }, read from the entropy annex). Accepted +candidates are resolved in batches (scan_layer_families) and checked +against survives_masking; rejected-after-masking candidates don't count +toward quota, triggering further "topup" rounds (bounded, MAX_TOPUP_ROUNDS += 5) rather than silently under-filling. Every SurvivingFamily reaching a +caller has already survived both the draw and post-masking informativeness — +no second filter pass needed downstream.

      +

      algorithms::scan::sampled_scan is the single entry point every sampled +computation goes through: one sample_index pass, fanned out to whichever +LayerReducer implementations the caller actually needs, each receiving the +exact same per-layer SurvivingFamily batch (an Arc, cloned not copied). +This is what guarantees several outputs requested together +(--pseudo-alignment and --sankoff and a snp-* --distance in one CLI +invocation) come from one random sample, not independently-drawn ones — +there is no other driver of sample_index anywhere in this crate.

      +

      Two LayerReducer implementations exist today:

        -
      • CanonicalKmerOf::minimizer() (obikseq/src/kmer.rs) — a direct O(k) - bit-arithmetic minimiser for a single isolated k-mer, replacing a - RollingStat instance fed byte-by-byte through an ASCII round-trip (used - by helpers::partition_of, called for every generated family variant). - ~3x wall-clock improvement on its own, confirmed by sampling - (obiskbuilder::rolling_stat/obikentropy frames disappeared from the - hot path). CanonicalKmerOf::partition() added alongside it (wraps - minimizer().seq_hash() & mask, the same routing rule - KmerPartition/RoutableSuperKmer use).
      • -
      • Cross-partition resolution (outgoing.par_iter() in - build_layer_sibling_annex) parallelised at the partition level — one - Rayon task per non-empty outgoing[dest] bucket. For k=31/m=11, a - central-base substitution changes the winning minimiser (and thus the - destination partition) only when that window overlaps the central base: - ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants - route right back to the partition already being built. That self bucket - ends up far larger than any other, so the per-partition split pinned one - thread to it alone while the rest of the pool finished instantly — - confirmed by sampling: one thread solid in MphfLayer::find, everyone - else idle. Fixed by splitting each non-empty bucket into - total_queries / n_workers (capped 4096) chunks before par_iter(), - preserving per-partition mmap locality (each chunk stays contiguous - within one partition) while letting Rayon spread an oversized bucket - across several threads. Net effect of both fixes together: ~3 cores - average → ~10-13 cores average on the same run, and a projected total - build time of ~1h15 down to ~30min on the real phyloskims_sal_vac run - this was measured against.
      • -
      • TracedBar's ETA (obisys/src/progress.rs) was silently starved: the - custom progress message and the self-computed ETA text used to share one - pb.set_message() slot, with the ETA holding off for 2s after any custom - message — fine when custom messages are rare, broken once - build_sibling_annex's per-partition callback fires more often than - that. Fixed by keeping the two texts in separate fields, composed - together on every render instead of one overwriting the other.
      • +
      • AlignmentReducer (alignment.rs) — accumulates SnpAlignment (one + IUPAC-coded row per included genome), reading family.masked directly.
      • +
      • PairwiseReducer (pairwise.rs) — accumulates PairwiseTally/ + PartitionDispersion, also reading family.masked directly (substitution/ + agreement counts and cardinality both derive from the same masked state + the alignment uses).
      -

      Tried and reverted — do not repeat blindly:

      +

      From a tally to distances and calibration

      +

      PairwiseTally stores per-genome-pair substitution/agreement/cardinality +counts, unfiltered, from which every downstream output is cheap O(n²) +post-processing (never a second scan):

        -
      • Parallelising the outer partition loop in build_sibling_annex with - obikindex::PartitionRunner (already used by merge/build_layers), - splitting a fixed core budget between outer (partition) and inner - (pipeline + resolution) concurrency so their product wouldn't exceed the - budget. Measured worse: throughput dropped over time (26 - partitions/5min → 38/11-12min) and peak resolution concurrency fell from - ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which - scales very well on its own — to make room for outer concurrency, and - running several partitions' resolution at once scatters access across - multiple partitions' mmap regions at once, working against the - locality outgoing's per-partition grouping exists for. PartitionRunner - stayed exported from obikindex (new_capped too) since it's - general-purpose, but nothing in obikphylo calls it.
      • -
      • Splitting resolution chunks even finer (/(n_workers*8), cap 1024, - instead of /n_workers, cap 4096) to smooth the residual sawtooth. - Measured ~10% slower, wider dips, not narrower. Reverted to the - original chunk sizing.
      • +
      • algorithms::snp_distance::distance_matrix — closed-form snp-* + corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions- + only), optionally gamma-corrected (GammaShape, PartitionDispersion::estimate_alpha + for Auto).
      • +
      • algorithms::sankoff::SankoffBundle::finish — assembles the + --sankoff/--tnt/--phyg/--iqtree pipeline's inputs + (RawSnpDistanceOutput, BasePairTally, CardinalityTally) from an + already-scanned AlignmentReducer/PairwiseReducer pair, plus + SankoffBundle::snp_distance to reuse that exact sample for a snp-* + --distance in the same invocation without a second scan.
      • +
      • algorithms::cardcomp — pure functions turning CardinalityTally/ + BasePairTally into row-stochastic transition probability matrices, then + the calibrated 16-state Sankoff cost matrix (pairwise_cost_matrix).
      -

      Known remaining limitation, not yet worth fixing: within one layer, the -four stages (sequential unitigs.bin read → parallel generation → -parallel resolution → sequential annex write) never overlap — confirmed by -1s-interval sampling: generation alone occupies ~17 threads evenly, but the -next layer's read/generation never starts until the current layer's -resolution and write are both done. This produces a real, periodic (~layer -duration) alternation between "many cores" and "few cores" that neither of -the fixes above touches, since both operate within one layer's resolution -step. The only remaining lever is overlapping consecutive layers (e.g. a -depth-2 pipeline: start layer N+1's read/generation while layer N's -resolution/write is still running) — a real restructuring, not a parameter -tweak, and explicitly not to be combined with the reverted -budget-capping idea above (let each phase use however many cores it -naturally wants; only the scheduling needs to overlap). Deferred, not -started.

      -

      Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)

      -

      family_scan.rs::scan_layer_families (shared by snp_pseudo_alignment, -sibling_annex_stats, cardinality_tally, scan_family_pairs) already -implements most of a dispatch/accumulate/resolve pipeline: generation -(cheap, CPU-only — builds outgoing[dest_partition] from FamilyMask and -buckets cross-partition queries) runs on an obipipeline::throttle + -make_pipe! stage, decoupled from resolution (I/O-bound, rayon::par_iter -across partitions, one generated batch resolved at a time, never several -concurrently — this ordering is deliberate, see the module's own docs on a -reverted concurrent-batch-resolution attempt that scattered mmap access). -The fast/slow mode gate (PartitionCache::fast_mode, cache.rs:162-163) -already exists: n_layers <= 7 (checked once from the first non-empty -partition's PartitionMeta::n_layers, documented as identical across every -partition of an index — a structural, build-time property, never a -per-partition state) decides whether FamilyMask's recorded layer_value -can be trusted to skip straight to the right layer -(find_presence_batch_fast) or must fall back to scanning every layer of -the destination partition (find_presence_batch).

      -

      Real gap, confirmed not implemented: resolution is triggered by the -source batch finishing (FAMILY_BATCH = 65536 minorants read from the -scanned layer), not by an output accumulator filling up. Since most -central-base variants of a family route back to the same partition being -scanned (~48% per the k=31/m=11 measurement above), a FAMILY_BATCH's -outgoing[dest] is large for the local/self partition and thin for the -other ~255 (or however many) destination partitions — each of those gets -resolved at low query density every batch instead of being accumulated -across several source batches until resolving it is worthwhile. This is -distinct from, and not fixed by, the fast/slow layer gate above.

      -

      Redesign sketched (not built): per-destination accumulators decoupled from -FAMILY_BATCH, flushed on reaching a size threshold instead of on source-batch -completion — a "hot" accumulator for the partition being scanned (sharded -one-per-generation-worker, no lock, since all n_workers pipeline workers -write to it concurrently — this differs from an earlier, simpler mental -model of "one thread owns one layer's local collector," which doesn't hold -here since n_workers threads cooperate on scanning one layer at a time, -not one thread per layer) and "cold" mutex-per-partition accumulators for -the rest, low contention expected since traffic to any single cold -destination is a small fraction of total.

      -

      This breaks the current strict-iteration-order delivery of on_family -(today: a reorder buffer keyed by batch, since a whole FAMILY_BATCH -resolves atomically). With cross-batch accumulation, a family only becomes -complete once every accumulator holding one of its outgoing queries has -flushed, at unpredictable, independent times — no longer streamable -strictly in order without a large, unbounded pending buffer. Resolution -sketched: replace order-dependent consumers with coordinate-addressed -writes instead of order-dependent appends (see PseudoAlignment idea -below) wherever possible, since sibling_annex_stats's reduction (plain -counts) is already order-independent and needs nothing here.

      -

      Superseded 2026-08-15 by the --subsample/--shannon design below, -which sidesteps the accumulator redesign for now: bounding the number of -families actually resolved per layer (via sampling) keeps per-layer -resolution volume small enough that the batch-density problem above stops -mattering in practice for these two consumers. The accumulator redesign -remains relevant for a future unsampled, full-index run, but is not -required to ship --subsample/--shannon.

      -

      Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15)

      -

      The reference run (phyloskims_sal_vac-scale bacterial test set, -iqtree.fasta) produced a dense alignment for 13 genomes × 383,965 sites -(4.8 MB) — trivially small. The in-progress plant index is expected to -carry on the order of 9 billion minorant families; a dense byte-per-cell -alignment at that column count is unbuildable regardless of genome count -(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000– -8,000 genomes on a large machine, which makes the per-cell cost dominant in -the other dimension too. Pruning the retained family set before -materializing anything is mandatory, not an optimization.

      -

      Already free: family_size() < 2 (no sibling variant registered at -all) is a zero-cost structural filter, read directly off FamilyMask bits, -already applied in snp_pseudo_alignment. Insufficient alone — per the -~80-85% mono-family estimate from earlier discussion, this only brings 9 -billion down to roughly 1.3-1.8 billion, still unusable.

      -

      Entropy definition — settled 2026-08-15, correcting an earlier wrong -turn. The project does not encode families as IUPAC ambiguity codes -interpreted the classical way (Fitch-parsimony subset-compatibility, or -ML's "one true state, uncertain which"); see -docmd/theory/evolutionary_distances.md ("Why the IUPAC/DNA encoding used -for the first --snp test was wrong") and the Sankoff resolution that -followed it. The real model is a genuine 16-state alphabet (the powerset of -{A,C,G,T}, ∅ included as a real state) scored with a calibrated -pairwise cost matrix (obikphylo::cardcomp::pairwise_cost_matrix, -cmd/phylo/sankoff.rs), not a compatibility/subset relation between -states. Under that model, each of the 16 states — including multi-bit ones -like AC — is a first-class, independently-costed state, not an -uncertainty encoding of a single true base. So: entropy over the 15 -non-empty states (∅ excluded, matching the earlier decision to exclude -genomes where the family is absent) is the correct informativeness -measure for this project — not a 4-symbol reduction, which would discard -exactly the cardinality/composition information the calibrated cost matrix -is built to exploit.

      -

      --subsample / --shannon — sampling strategy (decided 2026-08-15)

      -

      Goal: make both the pseudo-alignment (--snp) and a Shannon-entropy -diagnostic usable at any index scale, from the 13-genome bacterial -reference run up to the 9-billion-family plant index, without requiring the -batched-accumulator redesign above.

      -

      --subsample N (integer, families to retain): bounds the pseudo-alignment -to N minorant families, sampled proportionally per layer among -non-monomorphic minorants (family_size >= 2) — this sidesteps the need -for a true global reservoir merge across layers while still approximating a -uniform sample over the whole index, and directly answers the earlier open -question of global-vs-per-layer selection scope.

      -

      Three passes, in order:

      -
        -
      1. Global count (cheap, structural, parallel across layers — same shape - as the existing sibling_family_size_histogram, extended to report a - per-layer breakdown rather than one index-wide aggregate): for each - layer, count_layer = number of non-monomorphic minorants. Gives - total_count = Σ count_layer.
      2. -
      3. Per-layer proportional reservoir sampling (cheap, structural, one - pass per layer, no cross-partition resolution): N_layer = round(N × - count_layer / total_count). Since N_layer is a proportion of - count_layer, it can never exceed it as long as N <= total_count — the - one edge case is total_count <= N, in which case sampling is skipped - entirely and every non-monomorphic minorant of every layer is kept - (no reservoir needed, N was never a real constraint). Otherwise: - Algorithm-R reservoir sampling over the layer's non-monomorphic minorant - indices, producing N_layer iteration-order indices directly, no - intermediate full list ever materialized.
      4. -
      5. Filtered resolution (the expensive step, the existing - scan_layer_families engine, unchanged): re-scan the layer, generating - and resolving cross-partition queries only for the indices selected - in step 2 (cheap membership test against a small per-layer index set) — - this is what keeps --subsample cheap even on an unsampled-scale index, - since the cross-partition resolution volume is bounded by N, not by - the layer's true size.
      6. -
      -

      Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir -sampling would waste step-3's expensive resolution work on candidates later -evicted by the reservoir. Step 1 must fully complete (every layer) before -step 2 can start for any layer, since total_count is a global quantity.

      -

      --shannon (no argument): emits a CSV of per-family Shannon entropy -(15 non-empty states, ∅/absent genomes excluded from the denominator, per -the settled definition above). Independent of --subsample — entropy is -computed and written per family as soon as its genome_mask resolves, -O(1) memory per family, so it streams fine even unsampled at full index -scale (a time cost, not a memory one). Combined with --subsample N, it -delivers the original exploratory diagnostic (e.g. --subsample 1000000 ---shannon) directly from this general machinery, rather than a -purpose-built one-off script.

      -

      Validated end-to-end (2026-08-15) against real data: --sibling-hist on -phyloskims_sal_vac (91 real genomes, k=31/m=11, 256 partitions × 2 -layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238 -total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier -guess, corrected here). --subsample/--shannon on the smaller 20-genome -bacterial reference (benchmark/global_index_presence) produced a sample -size within rounding of the request (99,742/100,000) and a -[0.8,1.2)-bucket share (40.7%) matching the full unsampled population -(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a -real bug in reservoir_sample_layer (see next section), now fixed.

      -

      Bug found and fixed (2026-08-15): family_idx numbering mismatch. -scan_layer_families's family_idx counts every minorant of a layer -(monomorphic ones included, since iter_minorants_batch filters only on -is_minorant()), not the raw annex slot (SiblingAnnex::get(slot) spans -every k-mer, minorant or not) and not a counter over non-monomorphic -minorants alone. subsample.rs's reservoir_sample_layer originally -stored raw slot numbers in its HashSet<usize> selection, which drifts -away from family_idx as soon as any monomorphic minorant is seen — -i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by -tracking two separate counters: family_idx (every minorant, matching -scan_layer_families) and seen (non-monomorphic minorants only, what -Algorithm R actually samples over) — only family_idx values are ever -stored in the selection set. The existing unit test never exercised this -(its fixture has exactly one non-monomorphic family, always hitting the -"keep everything" shortcut) — a stronger fixture with several interleaved -monomorphic/non-monomorphic families would be needed to catch a regression -here automatically; not yet written.

      -

      Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15)

      -

      Motivation: on real data (bacterial reference, full unsampled run), only -~5‰ of non-monomorphic minorants fall in the [0.5, 1.5] bit band judged -phylogenetically interesting (entropy too low = uninformative near-invariant -site; too high = saturated/noisy, see family_entropy's 15-state -discussion) — roughly 1 in 10,000 minorants overall. Computing exact -entropy for every candidate just to discard 99.99% of them is wasteful at -the full 9-billion scale.

      -

      The idea: PersistentBitMatrix::col_view(c) gives a genome's whole -presence column as a BitSliceView (sequential, no MPHF, no cross-partition -routing — purely local to one layer's own matrix). Accumulating -TempCompactIntVecBuilder::inc_present(col) (already exists in -obicompactvec/src/builder.rs:121, along with add/min/max/diff on -IntSliceView — no new low-level API needed) over every column of a layer -produces coverage[slot]: how many genomes carry each exact k-mer, in one -sequential per-layer pass, entirely decoupled from family/sibling -structure. Persisted once per layer, a family's members' coverage could -then be looked up via a plain local MPHF index() (cheap) instead of a -full cross-partition presence resolution (find_presence_batch) — the -expensive part today is specifically the cross-partition/cross-layer -routing to a sibling's own matrix, not the bit-reading itself, and -coverage[slot] sidesteps that routing entirely by moving the cost into a -one-time, purely local, embarrassingly-parallel build step.

      -

      Why not implemented: coverage[slot] is a per-member marginal — -summing members' coverages to approximate a family's entropy silently -assumes no genome carries more than one member at once. It cannot -represent or detect joint co-occurrence (a genome carrying both A and -C at once, i.e. a combined 15-symbol state) at all, which is exactly the -phenomenon family_entropy's 15-state definition exists to capture (see -CardinalityTally/cardinality_transition_probs, the project's own -existing machinery for this same co-occurrence structure, built for the -Sankoff matrix calibration). A family that is in truth uniformly AC -across every carrying genome would look like a well-balanced 2-state split -under the marginal approximation (entropy ≈ 1) while its true 15-state -entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the -"saturated, uninformative" tail this pre-filter would need to catch, -undermining the point. It stays plausible as a coarse filter for the -low tail only (a dominant single member's marginal share reliably -predicts low true entropy too), but not as a stand-in for the high tail — -not pursued further for now.

      -

      --free-loss/--tnt pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below)

      -

      Measured on phyloskims_sal_vac (91 genomes): obikmer phylo --subsample 500000 ---free-loss --tnt logs four sequential stages — -raw_snp_distance (1413s), base_pair_tally (1456s), -cardinality_tally (2078s), snp_pseudo_alignment (143s). Reading the -code (obikmer/src/cmd/phylo/mod.rs:210-242, -obikphylo/src/siblings/distance.rs, cardinality.rs, alignment.rs) -surfaced two compounding problems, not one:

      -
        -
      1. Four separate full scans of the annex, each opening its own - KmerPartition/PartitionCache and calling scan_family_pairs/ - scan_layer_families independently — nothing computed in one stage is - reused by another. base_pair_tally is explicitly documented - (distance.rs:138-143) as a second full pass over the same - traversal raw_snp_distance already did, needed only because - raw_snp_distance doesn't keep the resolved bases, only aggregate - counts. cardinality_tally and snp_pseudo_alignment are each a - third and fourth independent full pass. Per the module's own earlier - profiling note (family_scan.rs:26-28, cited already above), this - traversal is page-fault/mmap-bound, not compute-bound — the ~10-12% - CPU efficiency ("contention" status) observed on these three slow - stages is consistent with I/O stalls scaled by repeated full scans, - not lock contention (there are no Mutex/RwLock anywhere in - siblings/*.rs; shared writes use per-slot AtomicU8::fetch_or).
      2. -
      3. Worse: raw_snp_distance and cardinality_tally don't honor - --subsample at all — both call scan_layer_families with - Selection::All hardcoded, and args.subsample isn't even threaded - into their function signatures (mod.rs:212, mod.rs:226). Only - snp_pseudo_alignment(args.subsample) builds a real reservoir-sampled - Selection::Some(set) (via compute_selections, alignment.rs:89). - So today, --subsample 500000 only bounds the pseudo-alignment step — - the SNP-distance matrix, the Sankoff base-pair calibration, and the - cardinality histogram are always computed over the full, unsampled - index regardless of the flag. This is not merely "different subsamples - per stage" (which would already be a problem worth fixing) — it's that - three of the four stages never subsample, which explains most of the - ~10x runtime gap against snp_pseudo_alignment on its own.
      4. -
      -

      Decided requirement: all four stages consume one shared selection, -computed once, not each stage either scanning everything or drawing its -own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair -calibration, the cardinality histogram, and the pseudo-alignment all -describe the same set of families — the calibration and the alignment it -calibrates are now guaranteed to agree on which sites exist.

      -

      Correction to the "single pass" framing above: base_pair_tally/ -cardinality_tally both need raw_snp_distance's complete aggregate -SNP/shared counts before they can derive included[i,j] (the -ratio_ceiling filter) — a genuine sequential dependency (included -can't be known until every pair's aggregate count is final), not an -artifact of the old code's structure. So the fix is two passes over -the shared selection, not one: pass A computes the aggregate counts (and -derives included); pass B fuses base_pair_tally + cardinality_tally -+ the pseudo-alignment (mutually independent once included is known) -into a single scan. Still a 4→2 reduction, and — per the clarification -that settled this — pass A itself now runs over the same shared -selection pass B uses (not the full unsampled index): "les ratios, on -les fait sur les sites sélectionnés, c'est tout, les autres sites -n'existent pas" — once a selection is chosen, both passes are bounded by -it, so on a real --subsample/--entropy run pass A is cheap too, not -just pass B.

      -

      Entropy-biased selection's own resolution to the "forward-looking -complication" (entropy must be known before selection, but selection -happens during the same scan that would resolve it): see "Entropy-biased -selection" below — resolved via a persisted per-layer entropy annex, not -by restructuring the scan into an inline pre-pass.

      -

      Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15)

      -

      Refines the "forward-looking complication" above with a concrete -mechanism. Instead of a hard [low, high] entropy band (or any other -exact cutoff) deciding which non-monomorphic minorant families are -eligible, selection is weighted by an unnormalized Gaussian kernel -centered on a target entropy: w(entropy) = exp(-(entropy - μ)² / (2σ²)) -— deliberately not the normalized Gaussian density (which would peak -below 1 and complicate the "probability" reading) — this kernel form -equals 1 exactly at entropy = μ and decays smoothly to 0 away from it, -so it reads directly as an acceptance weight: the further a family's -entropy from the target, the less likely it is picked, with no hard -in/out boundary — a few "bad" sites can still get in, by design. μ -(default ~1.0) and σ (default ~0.5) are meant to be user-tunable.

      -

      Mechanism: joint probability, not a weighted reservoir. Not -Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex -proposal, superseded before implementation) — instead, a single -independent accept/reject draw per qualifying candidate: draw -u ~ Uniform(0,1), accept iff u < p₀ · w(entropy). p₀ is a single -index-wide rate, N / total_count (total_count = the sum of -non_monomorphic_counts across every layer, N = --subsample's -target), applied identically at every layer — this alone already gives -each layer its proportional share (the same effect the old uniform -reservoir's explicit per-layer n_layer = N · count_layer / total_count -computation achieved, but without needing to compute it: applying one -rate uniformly is mathematically the same as proportioning per layer). -p₀ = 1.0 when there is no --subsample at all — --entropy alone is a -pure soft entropy filter over the whole index, no size target. Properties: -(1) strictly generalizes the existing uniform sampler — with σ large -enough that w ≈ 1 everywhere, this reduces to the old uniform N/total_count -draw; (2) yields "approximately N", not exactly N — expected accepted -count is N · mean(w), always ≤ N — an intentional relaxation, matching -"sous-échantillonnage à environ n" rather than the old reservoir's exact-N -guarantee; (3) one streaming pass, one random draw per candidate, no -reservoir state.

      -

      Resolving "entropy must be known before selection, but selection -happens during the same resolving scan": solved with a persisted, -per-layer entropy annex (obikphylo/src/siblings/entropy_annex.rs, -EntropyAnnex/EntropyAnnexBuilder), not by restructuring the scan. -Mirrors SiblingAnnex's mmap-backed, read-only-after-build convention, -but indexed by family_idx (every minorant of the layer, monomorphic -included — the same numbering Selection/scan_layer_families already -use), one f32 entropy15 value per entry, -1.0 sentinel for monomorphic/ -not-yet-computed. First use of --entropy/--entropy-sd on an index -pays a one-time cost (ensure_entropy_annexes in entropy.rs) — every -later run (any μ/σ, any command) reads the file positionally, no -re-scan, restoring the usual Selection::Some "skip resolving excluded -families" speedup that a naive "weigh during the resolving scan" design -would have permanently forfeited.

      -

      Bug found and fixed (2026-08-15): ensure_entropy_annexes scanned with -Selection::All instead of bounding to non-monomorphic minorants. -Monomorphism (family_size() < 2) is knowable directly from the annex -bits alone, no per-genome resolution needed — but the original -implementation called scan_layer_families with Selection::All -anyway, so fill_sub_matrix_carries (the expensive per-genome -resolution) ran for every minorant, ~98% of which are monomorphic -(measured elsewhere in this doc) and had their genome_mask immediately -discarded once the callback checked family_size() < 2. Fixed by adding -[subsample::non_monomorphic_selection_layer] — a cheap, annex-only, -non-sampling pass (same shape as reservoir_sample_layer, but keeping -every non-monomorphic minorant's family_idx instead of a bounded -reservoir) — and passing Selection::Some(&eligible) instead of -Selection::All, so the expensive resolution now runs only for the ~2% -of minorants that can actually produce a real entropy value. A -debug_assert!(mask.family_size() >= 2, ...) inside the -scan_layer_families callback guards the invariant.

      -

      Resolved: the existing hard "non-monomorphic minorant" eligibility -filter stays a hard gate upstream of the Gaussian weighting — only -qualifying families ever get a stored entropy value or a weighted draw.

      -

      CLI, implemented: two phylo options, --entropy <μ> and ---entropy-sd <σ> (obikmer/src/cmd/phylo/args.rs). The entropic filter -activates as soon as either is given (mod.rs, computed once into an -Option<EntropyBias> threaded through --snp/--family-overlap/ ---shannon/the fused sankoff pipeline below). If active but one or both -are unset, defaults are μ = 1.0, σ = 0.5.

      -

      Two entropy definitions kept side by side, for comparison (2026-08-15)

      -

      --shannon's CSV carries both entropy15 (family_entropy — the settled -15-non-empty-state definition, see above) and entropy4 (family_entropy_4 -— plain nucleotide reduction), computed from the same already-resolved -genome_mask, not from the marginal approximation above. A genome carrying -several bases at once contributes to each base's count (counted once per -base present, not fractionally split, not folded into one combined state) -— a genome polymorphic for the family is present at more than one base by -construction, so it is expected to count more than once; the denominator is -the total base-occurrence count, not the genome count (the two coincide -only when no genome carries more than one base). Kept side by side -specifically to measure, on real data, how much the two diverge — not yet -analyzed.

      -

      PersistentSparseBitMatrix — implemented and measured (2026-08-15)

      -

      A row-major (k-mer-major), deduplicated sparse alternative to -obicompactvec::PersistentBitMatrix, motivated by the same sparsity that -drove --subsample/--shannon above, but pursued as a foundational -storage-layer change rather than an index-level workaround. Full design -history, rationale, and rejected alternatives (external Elias-Fano crates, -cacheline-ef, a single unsplit dict_id array) are in the dedicated -implementation plan (vivid-mapping-tiger.md at the time of writing — the -content below is the durable summary, not a pointer to a session-scoped -file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi, -"Compact Data Structures for Collections of Sets" (2025, -biblio/Alanko et al. - Compact Data Structures for Collections of -Sets.pdf) — this design implements only their exact-duplicate special -case (a plain dedup dictionary), not their full subset-containment -hierarchy.

      -

      Design: four on-disk components, each mmap-backed, built once per -layer (matching how the rest of the build pipeline already works — never -the whole multi-billion-row index at once): an is_multi rank-capable -flag per row (singleton vs. multi-genome), a fixed-bit-width array for -singleton rows (genome index directly, ceil(log2(n_cols)) bits), a -separate fixed-bit-width array for multi-genome rows (dict_id, -ceil(log2(n_distinct_multi_sets)) bits — kept apart from the singleton -array specifically because n_distinct_multi_sets can be large in -absolute terms even when multi-genome rows are a small fraction of all -rows, and a single shared array would force every row, singletons -included, to pay the wider width), and a deduplicated dictionary of -distinct multi-genome sets (Elias-Fano-encoded byte offsets + a -varint-encoded values blob). New low-level primitives added to -obicompactvec to build this: PersistentFixedIntVec (arbitrary, -runtime-parameterized bit width, width 0 included — needed once a real -bug surfaced, see below), PersistentRankSelectBitVec (rank1/rank0/select1 -on top of the crate's existing count_ones, using -common_traits::SelectInWord), EliasFano (composes the two). A new -BinaryMatrix trait (n, n_cols, row/fill_row, fill_sub_matrix, -count_ones) unifies dense and sparse at the one call site that needs -both interchangeably (obikphylo::siblings::cache::Mat) — column-oriented -methods (col, col_view, the partial_*_dist_matrix family) stay -dense-only.

      -

      Two real bugs caught by tests, not by inspection: (1) EliasFano::open -re-derived its low-bits width from the persisted low-vector file's own -width byte; the zero-width case was built with a dummy 1-bit placeholder -(the builder rejected true width 0), so every reopened value silently -doubled. Fixed by making PersistentFixedIntVec genuinely support width 0 -(no storage, get always 0) instead of working around the limitation in -EliasFano. (2) An empty row (cardinality 0 — not expected on a real -built index, but not guarded against either) was recorded as a singleton -at genome 0, indistinguishable on read-back from a real singleton at -genome 0. Fixed by routing cardinality-0 rows through the dictionary path -(a genuine empty entry) instead of the singleton shortcut. Both caught by -obicompactvec's test suite (142 tests, including disk-reopen round-trips -that drop every builder/mmap before reopening fresh), not by manual -review — worth remembering next time a "this edge case can't happen in -practice" shortcut is tempting.

      -

      Measured on real data (layer_1 of phyloskims_sal_vac's -part_00018, 30,246,774 rows, 91 genomes — #[ignore]d benchmarks in -obikphylo/src/siblings/tests.rs):

      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
      densesparseratio
      on-disk size328.1MB43.7MB7.5x smaller
      build time / peak RSS—4.26s / 628MB(per-layer, in-memory construction — comfortable)
      row access, sequential (2M reads)43ns/row32ns/rowsparse faster (smaller structure, better cache fit)
      row access, random (2M reads)409ns/row85ns/rowsparse ~4.8x faster (the real --shannon/family-lookup shape)
      column access, one full column (30.2M rows)11.5ms993mssparse 86x slower (no native column method — every read decodes a full row to keep one bit)
      -

      The row-access wins (both directions) weren't the design's stated goal — -compactness was — but turn out real: dense's genome-major layout scatters -a single row read across a much bigger file, which costs more than -sparse's rank/select/varint decode once the file is this much smaller. -The column-access cost is the flip side of the same layout choice, and is -exactly what the next item below exists to fix.

      -

      Next, not yet planned: rewrite partial_jaccard_dist_matrix/ -partial_hamming_dist_matrix/etc. (obicompactvec/src/bitmatrix/pairwise.rs) -as a row-major co-occurrence accumulation (O(Σ_rows k²), per-row -increments into an NxN genome-pair counter — the known alternative to -today's column-fold, plausibly cheaper on data this sparse, not just a -fallback) so obikindex's --metric/distance-matrix path can use the -sparse type without the measured 86x column-access penalty. Needs its own -design pass (in particular how it plugs into the BitPartials/ -ColumnWeights traits so both matrix types keep serving --metric) -before implementation — not just "port the loop", a genuinely different -algorithm.

      -

      Also still deferred, unchanged from the implementation plan: full Alanko -et al. subset-hierarchy compression (only the exact-duplicate special case -is built), a sparse PersistentCompactIntMatrix (count matrices), and -BRWT-style column-correlation exploitation.

      -

      Wired into pack and the sibling-annex build path (2026-08-15)

      -

      PersistentSparseBitMatrix went from a validated but unused type to a -real, selectable on-disk format:

      +

      Public API surface

      +

      extensions::SiblingExt (CLI-facing trait on IndexCache) covers only the +operations with no cross-invocation sharing concern: build_sibling_annex, +shannon_entropy_csv, sibling_family_size_histogram/sibling_annex_stats, +family_overlap. The sampled SNP path (sampled_scan, AlignmentReducer, +PairwiseReducer, distance_matrix, SankoffBundle) is not wrapped in +trait methods — the CLI (obikmer::cmd::phylo) composes them directly, +because it alone knows the union of reducers a given invocation needs +(--pseudo-alignment/Sankoff group/snp-* --distance can combine in one +call). A per-output trait method would have to either duplicate that +composition logic or reintroduce the "two independent scans" problem this +design replaced.

      +

      Known gaps

        -
      • Generic Layer<D>: obikindex::layer::Layer<D>'s presence-only methods - (n_cols, sub_matrix, fill_sub_matrix) are generic over any - D: LayerData<Item = Box<[bool]>> + BinaryMatrix, not hardcoded to - PersistentBitMatrix — PersistentSparseBitMatrix implements - LayerData (open/read) the same way. find_slot/index_batch were - already generic over any D: LayerData, so they needed no change. - Verified by obikindex::layer's - presence_layer_generic_over_sparse_matches_dense test: build a dense - presence layer, convert it to sparse via build_from_dense, open both - as Layer<PersistentBitMatrix>/Layer<PersistentSparseBitMatrix> on - the same directory, assert n_cols/sub_matrix/find_slot agree. - (This test must stay at k=4 with mutually non-colliding canonical - 4-mers across its input sequences — K/M are process-wide - AtomicUsizes in test builds, not thread-local, so a test using a - different k races every other test in the same crate binary; a k=11 - version of this test passed alone but failed under the full - obikindex::layer suite for exactly that reason before being fixed.)
      • -
      • obikphylo::siblings::cache::Mat gained a third variant, - SparsePresence(Layer<PersistentSparseBitMatrix>), alongside Count - and Presence — every method (find_slot, index_batch, - iter_minorants_batch, n_cols, fill_sub_matrix_carries) dispatches - to it identically to Presence, since both go through the same generic - Layer<D> code. PartitionCache::build picks the variant per layer by - checking for presence/is_multi.prsb (the sparse format's own marker - file, see the design section above) before falling back to the dense - open path.
      • -
      • pack_sparse_bit_matrix (new, obicompactvec::bitmatrix::sparse): - pack --sparse's entry point. Idempotent (checks is_multi.prsb - first); packs to dense matrix.pbmx first if that hasn't happened yet - (the dense→sparse transpose needs random row access, which only the - packed/columnar dense forms give), then build_from_denses the sparse - form into the same directory and deletes matrix.pbmx — old-format - files are removed only after the new format is fully written, mirroring - pack_bit_matrix's own crash-safety convention.
      • -
      • CLI: obikmer pack --sparse threads a sparse: bool through - KmerIndex::pack_matrices (all other call sites — select, merge, - finalize_indexed — pass false, unchanged dense behaviour). Count - matrices are untouched by --sparse (no sparse PersistentCompactIntMatrix - — see "still deferred" above).
      • -
      • End-to-end coverage: obikphylo::siblings::tests:: - sibling_annex_works_after_pack_sparse builds a two-genome index, packs - it --sparse, asserts is_multi.prsb exists, then runs - build_sibling_annex and checks the resulting FamilyMasks match the - dense-path test (sibling_annex_one_sibling_each) exactly — proves the - sparse format round-trips through the real build pipeline - (PartitionCache sparse-detection included), not just the - obicompactvec/obikindex::layer unit layers below it.
      • -
      -

      Full workspace cargo test (all crates, unit + doc tests) green after -this change.

      -

      Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15)

      -

      Replaces the "four independent scans" problem above and implements -"Entropy-biased selection" above, end to end:

      -
        -
      • SankoffBundleExt::sankoff_bundle (new, - obikphylo/src/siblings/sankoff_bundle.rs) — the --sankoff/--tnt/ - --phyg/--iqtree block in obikmer/src/cmd/phylo/mod.rs now calls - this once instead of three separate raw_snp_distance/ - base_pair_tally/cardinality_tally calls. One PartitionCache, one - shared (possibly subsampled/entropy-biased) selection computed once via - compute_selections, two scans over it: pass A (aggregate SNP/shared - counts, --exclude-genome zeroing, then included[i,j]), pass B - (base_pair_tally + cardinality_tally + the pseudo-alignment, fused - into one scan_layer_families call per layer, all three read off the - same resolved genome_mask). snp_pseudo_alignment/ - shannon_entropy_csv (still used standalone by --snp/ - --family-overlap/--shannon) both gained an entropy_bias parameter - too, so entropy-biased selection isn't sankoff-specific.
      • -
      • Regression proof: sankoff_bundle_matches_old_separate_calls - (obikphylo/src/siblings/tests.rs) asserts sankoff_bundle's four - outputs are bit-identical to calling the old, separate - raw_snp_distance/base_pair_tally/cardinality_tally/ - snp_pseudo_alignment on the same fixture with no subsample — the - fusion is a performance change, not a behavior change.
      • -
      • EntropyAnnex/EntropyAnnexBuilder (new, - obikphylo/src/siblings/entropy_annex.rs) and ensure_entropy_annexes - (entropy.rs) implement the persisted-entropy mechanism from - "Entropy-biased selection" above. entropy_annex_builds_on_demand_and_biases_selection - (tests.rs) proves, on a fixture with one known-entropy family: the - annex file doesn't exist before any entropy-biased call; compute_selections - builds it on first use; μ set to the family's exact entropy with - p₀ = 1.0 selects it deterministically (u < 1.0 always, for - u ∈ [0,1)); μ set far away with tiny σ deterministically excludes - it (w underflows to exactly 0.0); the persisted value matches - --shannon's own family_entropy computation to 1e-6.
      • -
      • EntropyBias (pub, obikphylo::siblings::EntropyBias { mu, sigma }) - is the one new public type threading --entropy/--entropy-sd through - every Option<EntropyBias>-accepting method — resolved once in - obikmer/src/cmd/phylo/mod.rs from args.entropy/args.entropy_sd - (activation: either given; defaults 1.0/0.5 for whichever is unset).
      • -
      -

      Full workspace cargo test green after this change (167 unit tests in -obicompactvec+obikindex::layer+obikphylo alone, plus every other -crate's suite, no regressions).

      -

      Still open, not part of this change (per "Correction to the 'single -pass' framing" above): --raw-snp-distance/--raw-snp-counts (the -standalone diagnostic flags, not the --sankoff pipeline) still always -scan the full unsampled index — never threaded --subsample/--entropy, -out of scope here since the reported problem was specifically about the ---sankoff/--tnt pipeline's redundant/inconsistent scans, not these -two standalone flags.

      -

      query never benefits from sparse row-major access (found 2026-08-19, not implemented)

      -

      Benchmarked obikmer query against global_index_presence (dense-packed) -vs. global_index_presence_sparse (pack --sparse), 100k simulated reads -× 2 specimens (benchmark/, see -benchmark_query_testing.md). -Correctness: 0 mismatches — sparse and dense return bit-identical query -results. Performance: sparse consistently slower than dense (~30-50%, -reproducible across two runs with warm disk cache), the opposite of -pack --sparse's stated intent ("faster for single-row access... like -query").

      -

      Root cause, read from source, not measured in isolation: -KmerPartition::query_partition_with (obikindex/src/partition/query_layer.rs:155-220) -is architecturally column-major: stage 2 walks for g in 0..n_cols { for -slot in hit_slots { layer.col_value(g, slot) } }, documented (correctly) -as the right locality strategy for the packed/columnar formats, where -col_value → PersistentBitMatrix::get is a genuine O(1) mmap'd column -read (persistent.rs:110-113).

      -

      For Self::Sparse, that same get(c, slot) (persistent.rs:114-118) -allocates a full n_cols-wide buffer and calls fill_row — materializing -the entire row — just to return one cell. Called from inside the -column-major double loop, this reconstructs the same row once per genome -column touched: O(hits × n_cols) full-row rebuilds instead of O(hits). -PersistentSparseBitMatrix's own native row-major decode -(for_each_genome_in_row, sparse.rs:164-177, used correctly by its own -row/fill_row/fill_sub_matrix) is never reached from the query path -at all.

      -

      fill_sub_matrix (the existing BinaryMatrix trait primitive, -traits.rs:13-37) is not the right replacement for query either, even -once its own dispatch bug is fixed (see next section) — its output shape -is inherently column-dense: out[col] gets an entry for every column, -including columns with zero hits among the requested slots. On real -sparse data (a hit typically touching a handful of genomes out of dozens) -that's still O(n_cols) output regardless of true sparsity. What query -actually wants is the sparse triple stream (slot, col, value) it already -consumes as QueryHit::Value — not a materialized sub-matrix.

      -

      Proposed primitive (design only, not implemented — explicit ask: keep -count matrices not excluded, even though effort right now is -presence/absence only):

      -

      Not a closure-driven for_each — a real Iterator, one concrete struct -per matrix format, so the traversal state (current position in the sorted -slot list, current column, permutation, sparse-row decode cursor…) lives -in named struct fields instead of being threaded implicitly through -recursion or a captured closure. RPITIT (stable since 1.75, and this -workspace is edition 2024) means the trait method can return it without -naming or boxing the concrete type:

      -
      /// Yields every nonzero cell among `slots`, in implementation-defined order.
      -fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator<Item = (usize, usize, u32)> + 'a;
      -// item: (idx into `slots`, col, value)
      -
      -

      This is the one foundational primitive per format — both fill_sub_matrix -and a for_each-style callback become trivial, free consumers of it -(.for_each(f) is already Iterator::for_each from std; fill_sub_matrix -becomes "drain the iterator, scatter into out[][]"), instead of two -independently-maintained traversals that can silently diverge (see the bug -below — this is exactly how it happened).

      -
        -
      • On PersistentSparseBitMatrix: the struct is nearly free to write — - it wraps the existing (currently private) for_each_genome_in_row - per-row decode, advancing to the next slots entry on exhaustion. O(Σ - row nnz), zero n_cols-wide allocation.
      • -
      • On PersistentBitMatrix::{Packed,Columnar}: revised — cheaper than - first thought, by reusing the same split already used for - fill_matrix's own implementation instead of hand-writing a resumable - state machine at the matrix level. The base-vector layer - (BitSliceView, views.rs) already separates the two concerns: - fill_batch_sorted (views.rs:55-60, sorted-slot batch lookup) and a - genuine per-bit Iterator (BitSliceIter, views.rs:94+) sit side by - side there, one level below the matrix. Adding a - "positions among sorted_slots where the bit is set" iterator at that - same vector level is a filter over the existing get() — no new - state machine, since std::iter::Filter already is one, generated by - the compiler: -
        fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator<Item = usize> + 's {
        -    sorted_slots.iter().copied().filter(move |&slot| self.get(slot))
        -}
        -
        - The matrix-level nonzero_iter then composes these per column with - flat_map over 0..n_cols (each column's hits, tagged with c, - slot mapped back through the sort permutation fill_batch/ - fill_batch_sorted already carry) — again a combinator chain, not a - hand-rolled struct. Same algorithm, same mmap/sort locality as today's - fill_sub_matrix; just assembled from std iterator adaptors instead - of a loop body writing into a buffer, mirroring the vector/matrix split - the codebase already uses for fill_batch_sorted rather than - introducing a new shape.
      • -
      • Implicit: trivial (slots.iter().map(|&i| (i, 0, 1)), one column, - always present).
      • -
      • On PersistentCompactIntMatrix (counts): same treatment as - Packed/Columnar — no sparse count format exists yet ("Explicitly - deferred" per traits.rs:9-12), so no native low-effort case the way - Sparse has one, but not excluded either: the iterator's Item is - already (usize, usize, u32), not bool, specifically so presence - (0/1) and counts (arbitrary u32) share one primitive instead of a - bool/u32 split forcing counts out of the design. Ready for a native - sparse-count struct later without a signature change.
      • -
      -

      Would let query_partition_with's stage 2 collapse to one -for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) } -per layer, format-agnostic, each backend's struct deciding the actual -traversal.

      -

      This also closes the existing dispatch bug for free, by construction: -PersistentBitMatrix::fill_sub_matrix (persistent.rs:190-215, the enum -wrapper backing BinaryMatrix's trait impl) today does not delegate to -PersistentSparseBitMatrix::fill_sub_matrix for Self::Sparse — it -reimplements the same naive per-(column, slot) fill_row_bool loop -instead, bypassing the efficient native method one file over -(sparse.rs:249-258). obikphylo::siblings::cache::Mat -(cache.rs:138-145) independently built its own parallel enum wrapper -that dispatches correctly — evidence this was worked around rather than -fixed at the source: two hand-written traversals for the same format, -free to drift apart, and they did. If fill_sub_matrix itself is -rewritten as "drain nonzero_iter, scatter into out[][]", there is only -one traversal per format left to get right — the bug class doesn't just -get fixed once, it stops being possible to reintroduce.

      -

      Implemented (2026-08-20)

      -

      Built as designed above, with one deviation from the original sketch: -nonzero_iter ended up Box<dyn Iterator<...>>, not a bare impl -Iterator, because Columnar/Packed/Sparse/Implicit are genuinely -different concrete types and this method isn't on a trait (kept off -BinaryMatrix deliberately — that trait is used as dyn BinaryMatrix in -tests/sparse.rs, and RPITIT methods aren't dyn-compatible). One Box -per nonzero_iter call, not per cell — negligible next to what it -replaces.

      -
        -
      • BitSliceView::nonzero_among_sorted / IntSliceView::nonzero_among_sorted - (obicompactvec/src/views.rs): the vector-level filter/filter_map - primitive, exactly as sketched — no new state machine, std's own.
      • -
      • PersistentSparseBitMatrix::nonzero_iter (bitmatrix/sparse.rs): native, - std::iter::from_fn over one buffered row at a time via the existing - for_each_genome_in_row — no n_cols-wide allocation, ever.
      • -
      • PersistentBitMatrix::nonzero_iter (bitmatrix/persistent.rs): dispatches - to the above for Sparse; for Columnar/Packed, loops columns, - collects each column's nonzero_among_sorted hits via .extend() (not - flat_map — a flat_map closure can't lazily return something - borrowing its own captured sort permutation across separate calls - without either boxing per-column or fighting the borrow checker; eager - collection into one Vec sidesteps it, at zero cost since - fill_sub_matrix already fully materialized anyway). Implicit trivial.
      • -
      • PersistentBitMatrix::fill_sub_matrix and sub_matrix rewritten to - drain nonzero_iter — the dispatch bug is gone because there is now - only one traversal per format, not because the old one was patched. - PersistentCompactIntMatrix::nonzero_iter added the same way (counts - not excluded, per the earlier ask) — no native low-effort case, since no - sparse count format exists, but on the same primitive, ready for one.
      • -
      • KmerPartition::query_partition_with (obikindex/src/partition/query_layer.rs): - stage 2's column-major for g { for slot { col_value } } replaced by one - layer.nonzero_iter(&slot_list) call per layer, format-agnostic.
      • -
      • Tests: nonzero_iter_matches_dense, nonzero_iter_matches_row, and — - the one that actually targets the dispatch bug rather than each type's - own correctness — enum_wrapper_dispatches_to_native_sparse (builds - PersistentBitMatrix::Sparse(...) directly, not through open, since - open only auto-detects Sparse from a presence/ dir layout). - cargo test --workspace: green, no regressions.
      • -
      -

      Measured: re-ran the benchmark/ query branch (100k reads × 2 -specimens, same setup as the original finding). Correctness still 0 -mismatches. The dense/sparse performance gap is gone — previously sparse -~30-50% slower than dense, reproducibly; now within ~1-3% either way -(7.42s dense vs 7.60s sparse for Escherichia_coli--K-12_MG1655; 5.25s vs -5.30s for Saccharolobus_islandicus--M.16.4) — noise-level, not a -systematic gap. pack --sparse's claimed query win isn't confirmed -outright by this (sparse should arguably now beat dense on truly sparse -real data, not just tie), but the pathological regression is fixed.

      -

      PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)

      -

      Closes the gap flagged throughout this document ("no sparse count format -exists yet", traits.rs:9-12's "Explicitly deferred"): obicompactvec -already had PersistentSparseCompactIntMatrix (row-major, built on top of -PersistentSparseBitMatrix as its "which columns are non-zero" support, -values not deduplicated — see that struct's own doc comment), but it was -never wired into PersistentCompactIntMatrix, the dense-dispatching enum -every real consumer (TypedLayer<PersistentCompactIntMatrix>, -KmerLayer::Count) actually holds. Concretely: kmer_index.rs:: -pack_matrices(sparse=true) already called pack_sparse_compact_int_matrix -on every layer's counts/ — but PersistentCompactIntMatrix::open had no -code path back to what that just wrote, so a Count layer became -unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment -anyone ran pack --sparse on an index with count layers. Root cause, not a -workaround: add the missing Sparse variant.

      -
        -
      • -

        Enum + dispatch (intmatrix.rs): PersistentCompactIntMatrix::Sparse - (PersistentSparseCompactIntMatrix), detected in open/detect_storage - via a singleton_values.pciv marker (mirrors PersistentBitMatrix's own - sparse_meta.json check), reported via storage_kind(). col/ - col_view/col_persist panic/Unsupported on Sparse, same convention - as the bit side. sub_matrix/fill_sub_matrix and nonzero_iter - unified the same way PersistentBitMatrix's already are (drain - nonzero_iter, one traversal per format — see "Implemented - (2026-08-20)" above); nonzero_iter had to become Box<dyn Iterator<...>> - for the same reason (Columnar/Packed/Sparse are different concrete - types). No change needed in obikindex at all — KmerLayer::Count - already only ever holds TypedLayer<PersistentCompactIntMatrix>, so the - enum absorbing Sparse fixes the unreadable-layer bug for free, same as - PersistentBitMatrix::Sparse already did on the presence side.

        -
      • -
      • -

        CountPartials, non-naive (sparse_intmatrix.rs): unlike - PersistentSparseBitMatrix's dict-driven col_weights_and_pair_counts, - values here aren't deduplicated (two rows can share the same non-zero - column set via the same dict_id while carrying different counts), so - the "weight by how many rows share a dict entry" shortcut doesn't carry - over. What does: a single row-major pass (row_major_pairwise, decodes - each row once via for_each_cell_in_row, nests over that row's own - co-present columns) — O(Σ k̄²) over populated rows instead of the naive - O(n_cols² × n) column-pair rescan, same complexity class as the bit - side minus the dict multiplicity discount. Kernels used: min(a,b) - (bray, relfreq-bray — both vanish when either side is absent, so no - correction needed), a·b and √(a·b) (euclidean/relfreq-euclidean and - hellinger — these do need a correction, reconstructed from per-column - marginals via Σ(a-b)² = Σa²+Σb²-2Σab, since (a-0)² = a² ≠ 0 unlike - the min-based formulas). threshold_jaccard(1) shortcuts straight to - support's own BitPartials::partial_jaccard (threshold 1 is exactly - presence); threshold_jaccard(0) is closed-form (every u32 is ≥ 0).

        -
      • -
      • -

        Two pre-existing bugs found and fixed while wiring the threshold==1 - shortcut (bitmatrix/sparse.rs, BitPartials for - PersistentSparseBitMatrix, present since the 2026-08-15 implementation - above, never caught because no test compared Sparse's raw partial_* - output against dense on real data — only the diagonal-blind - jaccard_dist_matrix/hamming_dist_matrix finalisations were tested):

        -
      • -
      • partial_jaccard's diagonal was (0, 2×col_weights[i]) instead of a - genuine self-comparison (col_weights[i], col_weights[i]) — - col_weights_and_pair_counts's inter never pairs a column with - itself by construction.
      • -
      • partial_hamming's off-diagonal formula itself was wrong: total - - union (count of rows where neither column is present) instead of - the actual Hamming distance col_weights[i] + col_weights[j] - - 2×inter[i,j] (symmetric-difference size). Only coincides with the - correct value when col_weights[i] + col_weights[j] == total, so - small/synthetic test data could easily have hidden it.
      • -
      -

      Neither surfaced through jaccard_dist_matrix/hamming_dist_matrix - (both explicitly zero their own diagonal at finalisation, and the - off-diagonal partial_hamming bug had gone untested against dense - entirely) — only visible to a caller of the raw partial_* methods - directly, which is exactly what partial_threshold_jaccard(1)'s new - shortcut became. Fixed at the source, not patched around at the call - site; regression test added: - tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal.

      -
        -
      • Tests: tests::intmatrix::sparse_roundtrip_matches_columnar/ - sparse_roundtrip_from_packed (the open-dispatch fix, both build - paths); tests::intmatrix::sparse_count_partials_match_dense (all six - CountPartials formulas, thresholds 0/1/2/3, against Columnar on - asymmetric-presence data — this is what caught the diagonal gap in the - int side's own new code before it shipped, the same way it exposed the - two pre-existing bit-side bugs above); obikindex's - count_layer_transparently_reads_sparse_after_pack — the actual - end-to-end regression test for the original "layer unreadable after - pack --sparse" bug, built → packed sparse → reopened, compared against - the pre-pack dense read. cargo test -p obicompactvec -p obikindex: - green, no regressions (180 + 12 tests).
      • +
      • sample_index's skip_layers parameter (checkpointed-resume support) is + never called with a non-zero value by any current caller — --session + persistence was removed pending a redesign on top of sampled_scan + (single shared trunk, not per-driver checkpoint plumbing). The parameter + stays as the seam a future resume mechanism would use.
      • +
      • No cheap entropy pre-filter exists yet for the very high (saturated) tail — + a row-marginal coverage approximation was considered and rejected (cannot + represent joint multi-base co-occurrence, which is exactly what the + 15-state entropy definition exists to capture).
      diff --git a/DevDoc/implementation/benchmark_query_testing/index.html b/DevDoc/implementation/benchmark_query_testing/index.html index c8e7c979..14817889 100644 --- a/DevDoc/implementation/benchmark_query_testing/index.html +++ b/DevDoc/implementation/benchmark_query_testing/index.html @@ -1265,7 +1265,7 @@ - Sibling annex (discussion) + Sibling annex @@ -1485,8 +1485,9 @@ this pipeline.

      by verify_presence/verify_merge_presence, which only check index content against the .npz truth, never the query API.
    • pack --sparse produces a presence-matrix format documented (see - siblings.md) as faster for single-row - access (query) and slower for column-oriented access (phylo --metric). + obicompactvec.md, "Sparse matrix formats") as faster + for single-row access (query) and slower for column-oriented access + (phylo --metric). global_index_presence/ built by merge_presence.sh is always packed dense (packing is a stage inside merge, not a separate pack invocation) — there was no dense/sparse regression check.
    • diff --git a/DevDoc/implementation/chunkreader.refs/index.html b/DevDoc/implementation/chunkreader.refs/index.html index d19f2081..7bb38ea2 100644 --- a/DevDoc/implementation/chunkreader.refs/index.html +++ b/DevDoc/implementation/chunkreader.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/chunkreader/index.html b/DevDoc/implementation/chunkreader/index.html index b1ed11a9..6c386b97 100644 --- a/DevDoc/implementation/chunkreader/index.html +++ b/DevDoc/implementation/chunkreader/index.html @@ -579,7 +579,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/evidence_elimination.refs/index.html b/DevDoc/implementation/evidence_elimination.refs/index.html index 1fa176b3..b29cbf5d 100644 --- a/DevDoc/implementation/evidence_elimination.refs/index.html +++ b/DevDoc/implementation/evidence_elimination.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/evidence_elimination/index.html b/DevDoc/implementation/evidence_elimination/index.html index 70621d90..7db92264 100644 --- a/DevDoc/implementation/evidence_elimination/index.html +++ b/DevDoc/implementation/evidence_elimination/index.html @@ -1265,7 +1265,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/filtering/index.html b/DevDoc/implementation/filtering/index.html index 6fdab235..781161fe 100644 --- a/DevDoc/implementation/filtering/index.html +++ b/DevDoc/implementation/filtering/index.html @@ -1388,7 +1388,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/kmer.refs/index.html b/DevDoc/implementation/kmer.refs/index.html index b73ef4d2..898475b5 100644 --- a/DevDoc/implementation/kmer.refs/index.html +++ b/DevDoc/implementation/kmer.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/kmer/index.html b/DevDoc/implementation/kmer/index.html index f367ae72..ca7782a3 100644 --- a/DevDoc/implementation/kmer/index.html +++ b/DevDoc/implementation/kmer/index.html @@ -1265,7 +1265,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/layer_tier.refs/index.html b/DevDoc/implementation/layer_tier.refs/index.html index 92d01fcf..aff4e5af 100644 --- a/DevDoc/implementation/layer_tier.refs/index.html +++ b/DevDoc/implementation/layer_tier.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/layer_tier/index.html b/DevDoc/implementation/layer_tier/index.html index 2dc88b7a..048b4831 100644 --- a/DevDoc/implementation/layer_tier/index.html +++ b/DevDoc/implementation/layer_tier/index.html @@ -1504,7 +1504,7 @@ - Sibling annex (discussion) + Sibling annex @@ -2208,7 +2208,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>> pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
-

Column-first to match the on-disk column-major layout. fill_sub_matrix sorts slots once, then calls each column's fill_batch in turn — no redundant per-column sort. On PersistentSparseBitMatrix (k-mer-major, no column method) this degrades to a row-by-row decode; see siblings.md.

+

Column-first to match the on-disk column-major layout. fill_sub_matrix sorts slots once, then calls each column's fill_batch in turn — no redundant per-column sort. On PersistentSparseBitMatrix (k-mer-major, no column method) this goes through nonzero_iter instead, decoding one row at a time; see obicompactvec.md, "Sparse matrix formats".


LayeredStore\<S> and aggregation traits

LayeredStore<S> is a generic aggregation wrapper over Vec<S>. It propagates three traits from obicompactvec::traits up the hierarchy via blanket impls:

diff --git a/DevDoc/implementation/merge.refs/index.html b/DevDoc/implementation/merge.refs/index.html index 8f4e288c..22d6d050 100644 --- a/DevDoc/implementation/merge.refs/index.html +++ b/DevDoc/implementation/merge.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/merge/index.html b/DevDoc/implementation/merge/index.html index 9f91a463..b1050f4e 100644 --- a/DevDoc/implementation/merge/index.html +++ b/DevDoc/implementation/merge/index.html @@ -1348,7 +1348,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/merge_parallelism/index.html b/DevDoc/implementation/merge_parallelism/index.html index b7395e98..d78249bb 100644 --- a/DevDoc/implementation/merge_parallelism/index.html +++ b/DevDoc/implementation/merge_parallelism/index.html @@ -1332,7 +1332,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/mphf.refs/index.html b/DevDoc/implementation/mphf.refs/index.html index 331707e7..b48c8bda 100644 --- a/DevDoc/implementation/mphf.refs/index.html +++ b/DevDoc/implementation/mphf.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/mphf/index.html b/DevDoc/implementation/mphf/index.html index 010a5215..67587caa 100644 --- a/DevDoc/implementation/mphf/index.html +++ b/DevDoc/implementation/mphf/index.html @@ -1343,7 +1343,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/obicompactvec/index.html b/DevDoc/implementation/obicompactvec/index.html index 9f6c2fff..d1f0eea5 100644 --- a/DevDoc/implementation/obicompactvec/index.html +++ b/DevDoc/implementation/obicompactvec/index.html @@ -494,7 +494,7 @@ - Sibling annex (discussion) + Sibling annex @@ -632,6 +632,46 @@
  • + + + + Sparse matrix formats + + + + +
  • +
  • @@ -1191,7 +1231,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n) Sparse bit: sparse_meta.json + PFIV/Elias-Fano component files, row-major. Int: same support files (built on PersistentSparseBitMatrix internally) plus singleton_values.pciv/multi_values.pciv/multi_offsets for the per-row, non-deduplicated values -pack --sparse; see siblings.md for the sparse-vs-dense access-pattern trade-off +pack --sparse; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off Implicit (bit only) @@ -1201,7 +1241,6 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)

    PersistentBitMatrix::open(layer_dir) auto-detects the variant, in order: matrix.pbmx → Packed, presence/meta.json → Columnar, presence/sparse_meta.json → Sparse, layer_meta.json (no presence dir at all) → Implicit. PersistentCompactIntMatrix::open(layer_dir) mirrors the same priority order minus Implicit (there's no implicit count matrix — counts always have at least one on-disk column): matrix.pcmx → Packed, counts/meta.json → Columnar, counts/singleton_values.pciv → Sparse. col_view/col/sub_matrix panic on Sparse/Implicit where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through row/fill_row.

    -

    Unlike the bit side, PersistentSparseCompactIntMatrix's values are not deduplicated across rows — two rows can share the same non-zero column set (same dict_id in the shared support) while carrying different counts — so its CountPartials impl can't reuse the support's dict-multiplicity shortcut the way BitPartials for PersistentSparseBitMatrix does. It still avoids the naive O(n_cols² × n) column-pair scan via a single row-major pass (row_major_pairwise in sparse_intmatrix.rs), reconstructing the squared-difference formulas (euclidean/relfreq_euclidean/hellinger) from per-column marginals via Σ(a-b)² = Σa²+Σb²-2Σab — see siblings.md's "PersistentCompactIntMatrix::Sparse — implemented" entry for the full derivation.

    col_view(c) returns the appropriate view directly:

    // PersistentBitMatrix
     pub fn col_view(&self, c: usize) -> BitSliceView<'_>
    @@ -1212,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
     

    No wrapper enums (BitColView, IntColView): the caller receives a Copy view struct immediately usable with any view method or bulk builder method.

    pack_compact_int_matrix and pack_bit_matrix convert columnar → packed format.


    +

    Sparse matrix formats

    +

    Motivated by real k-mer-family data being extremely row-sparse (most rows — +k-mers — present in only a handful of genome columns out of dozens to +thousands): a row-major, deduplicated alternative to the dense column-major +formats above, built once per layer like the rest of the pipeline (never the +whole multi-billion-row index at once). Implements the exact-duplicate +special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data +Structures for Collections of Sets" (2025, +biblio/Alanko et al. - Compact Data Structures for Collections of +Sets.pdf) — a plain dedup dictionary, not their full subset-containment +hierarchy.

    +

    PersistentSparseBitMatrix

    +

    Four mmap-backed on-disk components: an is_multi rank-capable flag per row +(singleton vs. multi-genome), a fixed-bit-width array for singleton rows +(genome index directly, ceil(log2(n_cols)) bits), a separate fixed-width +array for multi-genome rows (dict_id, ceil(log2(n_distinct_multi_sets)) +bits — kept apart from the singleton array because n_distinct_multi_sets +can be large in absolute terms even when multi-genome rows are a small +fraction of all rows, and a shared array would force every row, singletons +included, to pay the wider width), and a deduplicated dictionary of distinct +multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values +blob). Supporting primitives: PersistentFixedIntVec (arbitrary, +runtime-parameterised bit width, width 0 included — needed once an +EliasFano reopen bug surfaced, see below), PersistentRankSelectBitVec +(rank1/rank0/select1 on top of the crate's existing count_ones, via +common_traits::SelectInWord), EliasFano (composes the two). BinaryMatrix +(n, n_cols, row/fill_row, fill_sub_matrix, count_ones) is the +trait unifying dense and sparse at the call sites that need both +interchangeably; column-oriented methods (col, col_view, the +partial_*_dist_matrix family) stay dense-only — Sparse/Implicit panic +there (see the PersistentBitMatrix::open note above).

    +

    Measured on real data (one layer, 30.2M rows, 91 genomes):

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    densesparseratio
    on-disk size328.1MB43.7MB7.5x smaller
    build time / peak RSS—4.26s / 628MBper-layer, in-memory construction
    row access, sequential (2M reads)43ns/row32ns/rowsparse faster (smaller structure, better cache fit)
    row access, random (2M reads)409ns/row85ns/rowsparse ~4.8x faster — the --shannon/family-lookup access shape
    column access, one full column (30.2M rows)11.5ms993mssparse 86x slower — no native column method, every read decodes a full row to keep one bit
    +

    The row-access wins (compactness was the stated design goal, not speed) come +from dense's genome-major layout scattering a single row read across a much +bigger file — costlier than sparse's rank/select/varint decode once the file +is this much smaller. Column access is the flip side of the same layout +choice; nonzero_iter (below) is what removes it from the hot path that +actually matters (query).

    +

    Two reopen-correctness bugs, now fixed at the source: (1) EliasFano::open +re-derived its low-bits width from the persisted low-vector's own width +byte; the width-0 case was built with a dummy 1-bit placeholder (the +builder used to reject true width 0), silently doubling every reopened +value. Fixed by making PersistentFixedIntVec genuinely support width 0 +(no storage, get always 0). (2) An empty row (cardinality 0) used to be +recorded as a singleton at genome 0, indistinguishable on read-back from a +real singleton at genome 0. Fixed by routing cardinality-0 rows through +the dictionary path (a genuine empty entry).

    +

    Still deferred: full Alanko et al. subset-hierarchy compression (only the +exact-duplicate case is built), a sparse PersistentCompactIntMatrix +variant beyond what's described below, BRWT-style column-correlation +exploitation, and a row-major rewrite of +partial_jaccard_dist_matrix/partial_hamming_dist_matrix/etc. as a +per-row co-occurrence accumulation (O(Σ_rows k̄²)) so obikindex's +--metric distance-matrix path stops paying the column-access penalty on +Sparse — needs its own design pass (how it plugs into +BitPartials/ColumnWeights so both matrix types keep serving --metric), +not just "port the loop".

    +

    nonzero_iter — the one traversal per format

    +

    fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a> +(item: index into slots, column, value) is the single foundational +per-format traversal primitive every consumer builds on — fill_sub_matrix +and a for_each-style callback are both trivial, free consumers of it +(.for_each(f) is Iterator::for_each; fill_sub_matrix drains the +iterator and scatters into out[][]), instead of two independently +hand-maintained traversals that can silently diverge. Box<dyn Iterator>, +not a bare impl Iterator: Columnar/Packed/Sparse/Implicit are +genuinely different concrete types and the method isn't on a +dyn-compatible trait (BinaryMatrix is used as dyn BinaryMatrix +elsewhere, and RPITIT methods aren't dyn-compatible) — one Box per +nonzero_iter call, not per cell.

    +

    Per-format implementation: PersistentSparseBitMatrix decodes natively, +buffered one row at a time (std::iter::from_fn over the existing +for_each_genome_in_row, no n_cols-wide allocation ever). Columnar/ +Packed loop columns, collecting each column's hits via +BitSliceView::nonzero_among_sorted/IntSliceView::nonzero_among_sorted +(a filter/filter_map over the vector-level get(), no new state +machine — std's own Filter iterator already is one) into one Vec via +.extend() (not flat_map: a flat_map closure can't lazily return +something borrowing its own captured sort permutation across separate calls +without either boxing per-column or fighting the borrow checker — eager +collection sidesteps it at zero extra cost, since fill_sub_matrix already +fully materialises anyway). Implicit is trivial (slots.iter().map(|&i| (i, 0, 1))).

    +

    This closed a real dispatch bug by construction, not by patching it: +PersistentBitMatrix::fill_sub_matrix/sub_matrix used to reimplement a +naive per-(column, slot) loop instead of delegating to +PersistentSparseBitMatrix's own efficient native method for Self::Sparse +— two hand-written traversals for the same format, free to drift apart, and +they had (obikphylo's own cache layer had independently built a parallel +enum wrapper that did dispatch correctly, evidence the bug was worked +around rather than fixed at the source). With fill_sub_matrix/sub_matrix +rewritten to drain nonzero_iter, there is only one traversal per format +left to get right. KmerPartition::query_partition_with +(obikindex/src/partition/query_layer.rs) — previously column-major +(for g { for slot { col_value } }, which for Sparse rebuilt a full row +per single cell touched, O(hits × n_cols) instead of O(hits) — the original +motivation for this primitive) — now issues one layer.nonzero_iter(&slot_list) +call per layer, format-agnostic. Dense/sparse query performance parity +confirmed on real data (previously sparse 30-50% slower than dense, +reproducibly; now within ~1-3% either way — noise level).

    +

    PersistentSparseCompactIntMatrix

    +

    The count-matrix analogue, built on PersistentSparseBitMatrix as its +"which columns are non-zero" support, values not deduplicated (unlike +the bit side, two rows can share the same non-zero column set — same +dict_id — while carrying different counts, so the dict-multiplicity +shortcut BitPartials for PersistentSparseBitMatrix uses doesn't carry +over). CountPartials here: a single row-major pass (row_major_pairwise +in sparse_intmatrix.rs, decodes each row once via +for_each_cell_in_row, nests over that row's own co-present columns) — +O(Σ k̄²) over populated rows, avoiding the naive O(n_cols² × n) +column-pair rescan. min(a,b)-based kernels (bray, relfreq_bray) need +no correction — they vanish when either side is absent. a·b and √(a·b) +kernels (euclidean/relfreq_euclidean/hellinger) do need one, since +(a-0)² = a² ≠ 0 unlike the min-based formulas — reconstructed from +per-column marginals via Σ(a-b)² = Σa²+Σb²-2Σab. +threshold_jaccard(1) shortcuts to BitPartials::partial_jaccard +(threshold 1 is exactly presence); threshold_jaccard(0) is closed-form +(every u32 is ≥ 0).

    +

    Wiring the threshold==1 shortcut surfaced two pre-existing formula bugs in +BitPartials for PersistentSparseBitMatrix (present since that type's own +introduction, never caught because no test had compared Sparse's raw +partial_* output against dense on real data — only the diagonal-blind +jaccard_dist_matrix/hamming_dist_matrix finalisations were tested), +fixed at the source:

    +
      +
    1. partial_jaccard's diagonal used to be (0, 2×col_weights[i]) instead + of a genuine self-comparison (col_weights[i], col_weights[i]) — + col_weights_and_pair_counts's inter never pairs a column with itself + by construction, so the diagonal needs its own case.
    2. +
    3. partial_hamming's off-diagonal formula itself was wrong: it computed + total - union (rows where neither column is present) instead of the + actual Hamming distance col_weights[i] + col_weights[j] - 2×inter[i,j] + (symmetric-difference size) — the two only coincide when + col_weights[i] + col_weights[j] == total, which small/synthetic test + data can easily satisfy by accident.
    4. +

    Aggregation traits (matrix level)

    ColumnWeights

    trait ColumnWeights: Send + Sync {
    diff --git a/DevDoc/implementation/obipipeline.refs/index.html b/DevDoc/implementation/obipipeline.refs/index.html
    index 46dace87..ba0cda60 100644
    --- a/DevDoc/implementation/obipipeline.refs/index.html
    +++ b/DevDoc/implementation/obipipeline.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/obipipeline/index.html b/DevDoc/implementation/obipipeline/index.html
    index edebe6b6..8c864f3f 100644
    --- a/DevDoc/implementation/obipipeline/index.html
    +++ b/DevDoc/implementation/obipipeline/index.html
    @@ -1293,7 +1293,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/obitaxonomy/index.html b/DevDoc/implementation/obitaxonomy/index.html
    index b1886a24..a9d00b9e 100644
    --- a/DevDoc/implementation/obitaxonomy/index.html
    +++ b/DevDoc/implementation/obitaxonomy/index.html
    @@ -1305,7 +1305,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/partition_layer_cache/index.html b/DevDoc/implementation/partition_layer_cache/index.html
    index 1cce3095..07dd753c 100644
    --- a/DevDoc/implementation/partition_layer_cache/index.html
    +++ b/DevDoc/implementation/partition_layer_cache/index.html
    @@ -1563,7 +1563,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_bit_vec.refs/index.html b/DevDoc/implementation/persistent_bit_vec.refs/index.html
    index 452c7db1..bccfdd30 100644
    --- a/DevDoc/implementation/persistent_bit_vec.refs/index.html
    +++ b/DevDoc/implementation/persistent_bit_vec.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_bit_vec/index.html b/DevDoc/implementation/persistent_bit_vec/index.html
    index 3e40d825..a8701785 100644
    --- a/DevDoc/implementation/persistent_bit_vec/index.html
    +++ b/DevDoc/implementation/persistent_bit_vec/index.html
    @@ -1405,7 +1405,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
    index 9e306acb..7d6b5deb 100644
    --- a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
    +++ b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_compact_int_vec/index.html b/DevDoc/implementation/persistent_compact_int_vec/index.html
    index 05ea4d9f..ab5d8aa2 100644
    --- a/DevDoc/implementation/persistent_compact_int_vec/index.html
    +++ b/DevDoc/implementation/persistent_compact_int_vec/index.html
    @@ -1388,7 +1388,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/pipeline.refs/index.html b/DevDoc/implementation/pipeline.refs/index.html
    index 3ab74967..985e28c7 100644
    --- a/DevDoc/implementation/pipeline.refs/index.html
    +++ b/DevDoc/implementation/pipeline.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/pipeline/index.html b/DevDoc/implementation/pipeline/index.html
    index a697eee7..fc08d198 100644
    --- a/DevDoc/implementation/pipeline/index.html
    +++ b/DevDoc/implementation/pipeline/index.html
    @@ -1254,7 +1254,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/select/index.html b/DevDoc/implementation/select/index.html
    index c98f84e1..c3cb4add 100644
    --- a/DevDoc/implementation/select/index.html
    +++ b/DevDoc/implementation/select/index.html
    @@ -1382,7 +1382,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/storage.refs/index.html b/DevDoc/implementation/storage.refs/index.html
    index 03bfe3b3..06d67528 100644
    --- a/DevDoc/implementation/storage.refs/index.html
    +++ b/DevDoc/implementation/storage.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/storage/index.html b/DevDoc/implementation/storage/index.html
    index 7fd77097..ce4b99a7 100644
    --- a/DevDoc/implementation/storage/index.html
    +++ b/DevDoc/implementation/storage/index.html
    @@ -1326,7 +1326,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/superkmer.refs/index.html b/DevDoc/implementation/superkmer.refs/index.html
    index 946e4da1..6622b610 100644
    --- a/DevDoc/implementation/superkmer.refs/index.html
    +++ b/DevDoc/implementation/superkmer.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/superkmer/index.html b/DevDoc/implementation/superkmer/index.html
    index d5267a43..6d6069fe 100644
    --- a/DevDoc/implementation/superkmer/index.html
    +++ b/DevDoc/implementation/superkmer/index.html
    @@ -1232,7 +1232,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/unitig_evidence.refs/index.html b/DevDoc/implementation/unitig_evidence.refs/index.html
    index 1dfbaa9c..4118486a 100644
    --- a/DevDoc/implementation/unitig_evidence.refs/index.html
    +++ b/DevDoc/implementation/unitig_evidence.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/unitig_evidence/index.html b/DevDoc/implementation/unitig_evidence/index.html
    index a66adbd3..32ef44a3 100644
    --- a/DevDoc/implementation/unitig_evidence/index.html
    +++ b/DevDoc/implementation/unitig_evidence/index.html
    @@ -1354,7 +1354,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/index.html b/DevDoc/index.html
    index e6629e0a..06cc62b0 100644
    --- a/DevDoc/index.html
    +++ b/DevDoc/index.html
    @@ -1228,7 +1228,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/index.refs/index.html b/DevDoc/index.refs/index.html
    index 5f0a7f61..b229367c 100644
    --- a/DevDoc/index.refs/index.html
    +++ b/DevDoc/index.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/installation/index.html b/DevDoc/installation/index.html
    index 42e8a762..d044a12d 100644
    --- a/DevDoc/installation/index.html
    +++ b/DevDoc/installation/index.html
    @@ -1314,7 +1314,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/kmers.refs/index.html b/DevDoc/kmers.refs/index.html
    index b12faff2..f789c1f2 100644
    --- a/DevDoc/kmers.refs/index.html
    +++ b/DevDoc/kmers.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/kmers/index.html b/DevDoc/kmers/index.html
    index a707331c..89637ba8 100644
    --- a/DevDoc/kmers/index.html
    +++ b/DevDoc/kmers/index.html
    @@ -1227,7 +1227,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/sitemap.xml.gz b/DevDoc/sitemap.xml.gz
    index 03a78e93..15bde70b 100644
    Binary files a/DevDoc/sitemap.xml.gz and b/DevDoc/sitemap.xml.gz differ
    diff --git a/DevDoc/theory/encoding.refs/index.html b/DevDoc/theory/encoding.refs/index.html
    index fe650220..44c46f36 100644
    --- a/DevDoc/theory/encoding.refs/index.html
    +++ b/DevDoc/theory/encoding.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/encoding/index.html b/DevDoc/theory/encoding/index.html
    index e8a2dbb1..abfa12e4 100644
    --- a/DevDoc/theory/encoding/index.html
    +++ b/DevDoc/theory/encoding/index.html
    @@ -1199,7 +1199,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/entropy.refs/index.html b/DevDoc/theory/entropy.refs/index.html
    index 49c1b70d..1a9e9081 100644
    --- a/DevDoc/theory/entropy.refs/index.html
    +++ b/DevDoc/theory/entropy.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/entropy/index.html b/DevDoc/theory/entropy/index.html
    index 5e201572..f913aeca 100644
    --- a/DevDoc/theory/entropy/index.html
    +++ b/DevDoc/theory/entropy/index.html
    @@ -1265,7 +1265,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/evolutionary_distances/index.html b/DevDoc/theory/evolutionary_distances/index.html
    index 77c3d6cf..57634792 100644
    --- a/DevDoc/theory/evolutionary_distances/index.html
    +++ b/DevDoc/theory/evolutionary_distances/index.html
    @@ -932,6 +932,17 @@
         
       
       
    +
  • + +
  • + + + + Automatic α estimation (--gamma-shape auto) + + + +
  • @@ -943,6 +954,45 @@ +
  • + +
  • + + + + α-estimation literature survey and design discussion (2026-09-11) + + + + +
  • + +
  • + + + + Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11 + + + + + +
  • @@ -1691,7 +1741,7 @@ - Sibling annex (discussion) + Sibling annex @@ -2268,6 +2318,17 @@ +
  • + +
  • + + + + Automatic α estimation (--gamma-shape auto) + + + +
  • @@ -2279,6 +2340,45 @@ +
  • + +
  • + + + + α-estimation literature survey and design discussion (2026-09-11) + + + + +
  • + +
  • + + + + Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11 + + + + + +
  • @@ -4476,9 +4576,75 @@ F84/TN93 and is included below. snp- prefix on every CLI value.

    +Γ rate-heterogeneity modifier, applicable to snp-jc, snp-k2p, snp-k81, snp-t92, snp-f84, snp-hky85, snp-tn93 (not snp-raw, nothing to correct; not snp-logdet, no standard gamma formulation) — same -formula as the base correction, weighted by a shape parameter α supplied -by the user (--gamma-shape <alpha>), not estimated by ML. A modifier on -existing values, not a separate enum arm per distance.

    +formula as the base correction, weighted by a shape parameter α either +supplied by the user (--gamma-shape <alpha>) or estimated from the data +(--gamma-shape auto/estimate, method-of-moments — not ML; see +"Automatic α estimation" below). A modifier on existing values, not a +separate enum arm per distance.

    +

    Automatic α estimation (--gamma-shape auto)

    +

    Correction (verified against the primary source, 2026-09-11): Jin & +Nei (1990) itself ("Limitations of the Evolutionary Parsimony Method of +Phylogenetic Analysis", Mol. Biol. Evol. 7(2):82–102 — the paper this +whole +Γ correction is cited from, confirmed algebraically to match this +codebase's corrected_log/k2p exactly against their eq. A4, general, and +A8, the a = 1 case) contains no data-driven α-estimation procedure. +Their own recommendation (p. 98) is a fixed default: "we suggest that the +gamma distance with a = 1 [eq. A8] be used. However, one may choose a +different gamma distance, estimating a from data. Wilson et al. (1989) +recently used a distance with a = 1/2 for restriction-site data of +mitochondrial DNA in hominoids." — i.e. Jin & Nei explicitly punt +data-driven estimation to a different paper (Wilson et al. 1989), not +read/verified here. The estimator below is therefore not "Jin & Nei's +method" under any framing — that attribution (present in an earlier +revision of this section) was wrong, not just under-cited.

    +

    Implemented (PartitionDispersion, obikphylo/src/siblings/algorithms/pairwise.rs) +as an independent method-of-moments estimator, unrelated to any specific +published procedure: pools substitution counts by partition rather +than by genome pair, during the same reduce_pairwise pass that builds +PairwiseTally (no second scan).

    +

    For partition i: n_i = substitutions pooled over every genome pair, +L_i = eligible loci pooled over every genome pair, R_i = n_i / L_i. +Modeling among-site rate heterogeneity the same way as the +Γ correction +itself (a Gamma(α, α)-distributed, mean-1, multiplicative rate shared by +every locus in a partition — the classical Poisson-Gamma/negative-binomial +mixture, the general identity behind gamma-rate-heterogeneity corrections, +also behind Uzzell & Corbin's (1971) original observation that substitution +counts across sites/regions are over-dispersed relative to Poisson):

    +
    \[ +\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha} +\]
    +

    Weighting each partition's squared deviation by its own L_i cancels the +Poisson term before attributing what's left to α:

    +
    \[ +\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad +V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad +\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L} +\]
    +

    where \bar L is the mean partition size. Returns "no estimate" (falls +back to the uncorrected formula, warns) when fewer than 2 partitions have +data, \hat\mu \le 0, or V doesn't exceed the Poisson floor +\hat\mu/\bar L — no detectable over-dispersion, α would be unbounded.

    +

    Caveat, stated explicitly rather than left implicit: unlike every +closed-form correction in snp_distance.rs (each verified line-by-line +against ape's dist_dna.c, and now also against Jin & Nei 1990 directly +for the base +Γ formula), this estimator is derived from first +principles (the general Poisson-Gamma/negative-binomial identity) with no +primary-source procedure behind it at all — not Jin & Nei's (confirmed +above), and Wilson et al. (1989), the paper they point to instead, hasn't +been read/verified either. Mathematically self-consistent (re-derived and +checked, not guessed), but a from-scratch method, not a literature +implementation. If --gamma-shape needs a value with a literature +pedigree rather than an estimated one, Jin & Nei's own stated default, +α = 1 (--gamma-shape 1), is the better-supported choice today.

    +

    Deliberately not gated by --sankoff-ratio-ceiling the way +base_pair_tally is (same precedent as cardinality_tally — see its own +doc comment): that filter excludes individual saturated pairs from a +composition estimate computed once at the very end, from the complete +PairwiseTally; the partition axis needed here only exists transiently, +one partition at a time, while PairwiseTally is still being built — long +before any pair's final SNP ratio (and thus its ratio_ceiling eligibility) +is known. --exclude-genome isn't applied either, matching +reduce_pairwise's own raw per-pair fold.

    Implemented now: snp-raw, snp-jc, snp-k2p, snp-k81, snp-f81, snp-t92, snp-tn93, snp-tv, all with +Γ except raw/tv — see "Exact formulas" below. snp-tajima-nei, snp-f84, snp-hky85, @@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same mechanical substitution but weren't independently checked against an ape-equivalent reference for those three specifically — flagged here, not silently assumed correct.

    +

    α-estimation literature survey and design discussion (2026-09-11)

    +

    Follow-up discussion after implementing PartitionDispersion::estimate_alpha +(previous section), triggered by the user pointing out that the literature +conflates two distinct problems: estimating α from an alignment versus +using a given α in a Jin-Nei-style corrected distance. Jin & Nei (1990) +itself is squarely in the second camp — α is a user-supplied parameter +there, not something their paper estimates (confirmed by reading the paper +directly, see previous section). This section catalogs the actual +α-estimation literature and records which approaches were considered and +why each was or wasn't adopted.

    +

    Bibliography (methodological papers, not exhaustive users of +Γ):

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    MethodReferenceNeeds a tree?Estimates "true" α or a task-specific one?Cost
    ML, continuous Γ (foundational)Gu, Fu & Li (1995), MBE 12:546–557 — read in fullyes, topology fixed a prioritrue α (+ θ jointly, invariant+Γ)very high, ≤5–6 taxa in practice
    ML, discrete ΓYang (1993/94), MBE 10:1396; JME 39:306yestrue αhigh
    Parsimony-based ML on inferred changesYang & Kumar (1996), MBE 13:650yes, equal branch lengths assumedtrue α, biased when branches unequallow
    Corrected substitutions/site → NB fitGu & Zhang (1997), MBE 14:1106 — read in fullyes, + ancestral reconstructiontrue α, close to ML accuracylow (given a tree)
    Topology-optimizing grid searchGuindon & Gascuel (2002), MBE 19:534 — read in fullyes, but built from the candidate distances themselves (BIONJ)not true α — deliberately biased upward for topological accuracylow–moderate (grid × tree build)
    Free (non-parametric) rate distributionSusko et al. (2003), Syst. Biol. 52:594yestests the Γ assumption itselfhigh
    Parsimony counts/site → NB fitWakeley (1993), JME 37:613 — read in fullyes, NJ tree + Fitch parsimony ancestral reconstructiontrue α, biased upward by parsimony (conservative)low (given a tree)
    Poisson-Gamma/negative-binomial, foundationalUzzell & Corbin (1971), Science 172:1089 — not read, only seen via citationsunknown — likely also count-per-site/tree-based, unverifiedtrue α (indirect, via NB overdispersion)unknown
    Bayesian posterior over αBPP/MrBayes/BEAST2/RevBayesyesposterior, not point estimatevery high
    +

    Correction (2026-09-11, after reading Wakeley in full): an earlier +revision of this table filed Wakeley (1993) under "no tree needed" — +wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei +1987, on the real mtDNA data; coalescent-simulated for the simulation +study), reconstruct ancestral states at internal nodes by Fitch (1971) +parsimony, count the minimum number of changes per site this implies, +then fit either a two-rate Poisson mixture or a gamma-distributed-rates +model (⟹ negative binomial, parameters by the methods of Cohen 1965 and +Johnson & Kotz 1969) to the resulting per-site count distribution — the +same general "tree + ancestral reconstruction + per-site counts + NB fit" +shape as Gu & Zhang (1997), just using raw parsimony counts directly +rather than Gu & Zhang's multiple-hit-corrected k̂. Confirmed by +simulation (Wakeley's own Fig. 1) that parsimony underestimates both the +mean and (more severely) the variance of per-site change counts, which +biases α upward (toward apparent uniformity) — consistent with Gu & +Zhang's own citation of Wakeley for exactly this bias, and with the +general "parsimony overestimates α" pattern noted throughout this +literature. Real hypervariable-region-1 mtDNA data gave α̂ ≈ 0.44–0.60 +per data set, 0.47 combined — despite the conservative bias, still +solidly in the "strong heterogeneity" range, which is the paper's own +point: the method is biased but usefully conservative, not useless.

    +

    Where PartitionDispersion::estimate_alpha actually sits: nowhere in +this table's tree-dependent rows. It shares only the abstract +mathematical identity (Poisson-Gamma mixture ⟹ negative binomial +over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a +genuinely different, tree-free and ancestral-reconstruction-free route: +pairwise genome-to-genome substitution counts pooled by partition +instead of per-site counts pooled across an alignment's columns after +ancestral state reconstruction. As far as this survey has established +(Uzzell & Corbin not read, so not fully ruled out), no tree and no +ancestral reconstruction of any kind appears to be a genuinely +distinguishing property of PartitionDispersion, not something it +inherits from prior art — consistent with it being an independent +derivation rather than a literature implementation (see previous +section's caveat, now on firmer footing).

    +

    Why a tree-free estimate is even valid — the identifiability question. +Raised directly by the user after reading Gu, Fu & Li (1995), which states +explicitly: "one cannot estimate the rate heterogeneity when the number +of sequences is <3." This is a real non-identifiability result, not a +practical inconvenience: with only 2 sequences, a site's history is +summarized by one binary observation (differ / don't differ), collapsed +across all sites into two aggregate numbers (transition and transversion +proportions) — divergence time and among-site rate variance are +confounded in that pair of numbers, with no way to separate them. Every +method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3 +sequences sharing one evolutionary history (a tree) specifically to get +multiple independent looks at the same site's rate across different +lineages, which is what identifies its variance.

    +

    PartitionDispersion doesn't violate this, because it isn't estimating +the same quantity. The literature above estimates rate variation across +sites, within one shared tree. PartitionDispersion estimates rate +variation across partitions, pooled over every genome pair in the +index — each genome pair stands in for one independent draw of a +partition's relative rate, the same identifying role multiple lineages +play in the tree-based methods, just substituting "genome pair" for +"lineage" and "partition" for "site." With P genome pairs (potentially +in the hundreds here) all contributing to every partition's pooled count, +the ≥3-samples identifiability requirement is met by the pair count, not +by tree depth — closer in spirit to Jin & Nei's own original motivating +scenario (comparing several genomic regions across one fixed panel of +taxa) than to single-tree site-rate estimation.

    +

    This substitution carries its own assumption, which must be named rather +than left implicit: that a partition's relative rate (fast/slow) is +reasonably stable across genome pairs, not just across sites within +one pair — the direct analogue of the "no lineage-specific rate variation, +only site-specific" assumption every tree-based +Γ method already makes, +just moved from lineage→pair and site→partition. If this breaks — e.g. an +index mixing very closely related and very divergent genome pairs, where a +partition's saturation behavior differs qualitatively between the two — +the estimate could be misled in a way a tree-based method would at least +have the topology to detect and a tree-free, pool-everything method +cannot. Not yet tested against real data with strongly heterogeneous +pairwise divergence; worth keeping in mind as the main open validity +question for --gamma-shape auto, not the citation question (now settled) +this whole discussion started from.

    +

    Gu, Fu & Li (1995), read in full — the foundational ML method, not a +candidate to implement. Models rate variation as invariant+Γ (θ = +proportion of invariant sites, α = gamma shape among the variable sites), +and derives the exact joint likelihood over nucleotide configurations +across all n sequences on a fixed, a-priori-known tree topology — +for 3 sequences this is closed-form (Jukes-Cantor), for n sequences it's +a sum over 4^n configurations weighted by coefficients tied to the +specific topology's branching structure (their eq. 32–34), with branch +lengths, α, and θ jointly optimized by Newton-Raphson (Hessian over all +free parameters). The site-rate integral has the same closed form +E = θ + (1-θ)(1+D/α)^{-α} that underlies +Γ distance corrections +generally (the same mechanism as corrected_log, not a coincidence — Jin & +Nei's own formula is the θ=0 special case applied to a pairwise D +rather than a whole-tree one). Explicitly stated by the authors as +infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a +candidate for obikmer's typical genome counts. Notable contribution worth +keeping in mind regardless: ρ = (1+θα)/(1+α) is shown to be a more robust +summary of rate heterogeneity than α alone whenever θ isn't negligible +— α and θ are confounded (nearly-invariant sites can come from either a +small α's left tail or from a nonzero θ), so α alone can be very +unstable while ρ stays well-behaved. PartitionDispersion has no θ +term at all (no invariant-site component), so this confound doesn't arise +for it the same way, but it's worth remembering if an invariant+Γ variant +is ever considered.

    +

    Gu & Zhang (1997), read in full — rejected for now. Their procedure: +(1) a tree with least-squares branch lengths must already exist; (2) +ancestral states are reconstructed at every internal node (they use Zhang & +Nei 1997's likelihood method, but note plain parsimony would work, just +less accurately); (3) per site, branches are split into "changed"/"unchanged" +given the ancestral reconstruction, and the expected substitution count +k̂ (corrected for multiple hits) is obtained by solving their eq. (7) +(or eq. 12 for the generalized model) numerically — a root-find per site, +not a closed form; (4) the site-level k̂ values (real-valued, not +integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to +get α — again no closed form, numerical optimization. This is a real +per-site pipeline requiring a tree before distances/α can be computed — +the reverse of obikmer's current dependency order (snp_distance computes +distances, which --nj/--upgma only turn into a tree afterward). Their +own answer to this chicken-and-egg problem is an iterative +distance→tree→α→distance loop, which they themselves flag as expensive. +Adopting this would mean a new ancestral-reconstruction subsystem (even a +parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB +ML fit — a much bigger addition than PartitionDispersion, not something +to build without a concrete need beyond "closer to Gu&Zhang's own +simulated accuracy than our moment estimator."

    +

    Guindon & Gascuel (2002), read in full — plausible future addition, not implemented. +Their Q criterion: for a candidate α, build a tree (they use BIONJ) from +the α-corrected distance matrix; for every internal branch, group taxa into +the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree +distances \(\bar\delta_{AB}\), \(\bar\delta_{CD}\), etc., and let S ≤ M ≤ L +be the three pairings' sums ({AB,CD}, {AC,BD}, {AD,BC}); the branch's +reliability score is Q_branch = L − M (zero when the four points are +perfectly tree-additive). The whole-tree Q is the mean over internal +branches (negative/zero branches excluded). α* = argmin_α Q(α) over a +grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where +sensitivity is highest). Complexity: O(n²l) once for the base +frequencies, then O(n²r) for the r candidate distance matrices, then +O(n³r) worst case for building r trees and evaluating Q on each +(same order as the tree-building step itself, so "free" relative to it).

    +

    Central finding, the opposite direction from an initial misreading in +this discussion: α_opt (Guindon-Gascuel's topology-optimal value) is +always ≥ the true α, not the reverse — underestimating rate +heterogeneity (i.e., picking a larger α than reality) reduces the variance +of distance estimates and thereby improves NJ/BIONJ topological accuracy, +especially when the molecular clock roughly holds (where α_opt → ∞, i.e. +no correction at all is topologically best). This is an empirical +regularity observed across their tested conditions (η ∈ {0.5, 2.0}, true +α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard +guarantee would be another unverified-claim mistake of the kind this whole +discussion has been correcting.

    +

    Proposed integration (not yet implemented): since α_true ≤ α_opt +empirically, PartitionDispersion::estimate_alpha()'s output could seed +the lower bound of Guindon-Gascuel's grid (search [α̂, ∞) instead of +[0.1, 5000]), narrowing the grid substantially and reusing the existing +estimator rather than needing an independent "true α" source — this also +sidesteps Gu & Zhang's tree-first dependency problem entirely, since +Guindon-Gascuel's own grid already builds a tree per candidate. Still +requires: (a) a BIONJ implementation (see below), (b) the Q-criterion +subtree-grouping logic, (c) a grid-search driver. Not started.

    +

    BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found. +Searched crates.io directly (API query for "bionj": 0 results) and read the +docs.rs pages of the most plausible candidates: phylotree (builds/reads/ +manipulates trees but has no distance-matrix reconstruction — no NJ, no +BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected +after checking the actual docs), speedytree (NJ only — Canonical/RapidNJ/ +Hybrid variants, explicitly no BIONJ), nj (plain NJ only), phylo (no +distance-based reconstruction at all). obikphylo::neighbor_joining +(tree.rs) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in +how branch lengths are weighted during agglomeration (accounts for +distance-estimate variance, not just the sum), a modest delta over an +existing NJ implementation rather than a rewrite, but would need to be +written in-house if the Guindon-Gascuel integration above is ever pursued +(their own paper notes other tree-building methods gave similar results in +their simulations, so plain NJ may be an acceptable substitute if BIONJ +proper is judged not worth the effort).

    +

    Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11

    +

    Correctness bug, not just a performance one. --sankoff/--tnt/--phyg/ +--iqtree (via SiblingExt::sankoff_bundle) and a snp-* --distance +(via SiblingExt::snp_distance) always consume the exact same selection +parameters when requested together in one obikmer phylo invocation +(n/free_loss/no_ambiguity/excluded-set/entropy_bias — the CLI has +no way to give them different values in one run, see cmd/phylo/args.rs). +Before this fix, each independently called sample_index, and +sample_layer's rand::rng() (subsample.rs:235) is a thread-local +generator that advances across calls, not reseeded each time — so the +second call silently drew a different random sample of sites than the +first, even with identical parameters. Verified empirically (throwaway +8-genome index): running snp-jc alone vs. combined with --sankoff +produced visibly different distance matrices from the same index/params. +This defeated the actual point of combining these flags — running several +algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a snp-* distance) on +one identical site selection for direct comparison.

    +

    Fix: SankoffBundle (sankoff.rs) now retains its PairwiseTally/ +PartitionDispersion internally (previously consumed into raw/ +base_pair_tally/cardinality_tally and dropped) and exposes +SankoffBundle::snp_distance(kind, gamma_shape), which computes the +matrix from that same already-sampled tally — no second sample_index +call. The shared post-sampling logic (the --gamma-shape support check, +alpha resolution, final matrix build — previously all inline in the +snp_distance() free function) was factored into +pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape) in +snp_distance.rs, called by both the standalone snp_distance() (after +its own fresh sampling) and SankoffBundle::snp_distance (reusing the +bundle's). cmd/phylo/mod.rs keeps the Option<SankoffBundle> alive past +the Sankoff-family if block and, when a snp-* --distance is also +requested, calls bundle.snp_distance(...) instead of +cache.snp_distance(...) whenever a bundle was built — since both branches +are driven by the same args.* fields, the parameters trivially always +match when both fire; no runtime "do the params match" check needed. +Verified end-to-end: the reused path logs (reusing the Sankoff bundle's +sample) and the snp_distance stage timer reads 0ms (formula(kind) +is O(n²) post-processing, no I/O), vs. ~150-230ms for a fresh sample on +the same tiny test index.

    +

    Explicit --session (implemented 2026-09-12)

    +

    The in-process fix above only covered reuse within one command. +--session DIR (cmd/phylo/args.rs) names a directory (outside the +index) that persists the snp-* --distance sample — the +PairwiseTally/PartitionDispersion pair sample_index would otherwise +rebuild from scratch — across separate obikmer phylo invocations.

    +

    Crate placement, as planned: a new obiksession crate +(src/obiksession/), domain-agnostic — no PairwiseTally/"site" concept +anywhere in it. Session::open(dir, params: &[u8], force: bool) handles +directory lifecycle and the params-conflict check against an opaque byte +blob; Session::store(name, bytes)/Session::restore(name) -> Option<Mmap> +handle a checksummed artifact cache. obikphylo +(siblings/algorithms/session_cache.rs, shared by snp_distance.rs and +sankoff.rs — see below) is the consumer: it serializes its own +PairwiseTally/PartitionDispersion/SnpAlignment to bytes (via rkyv) +and hands them to obiksession, which never sees their type.

    +

    Serialization: rkyv 0.8.18, added as planned. One deviation from the +original zero-copy pitch, scoped down deliberately: artifacts are restored +via rkyv::from_bytes (full owned deserialize) rather than rkyv::access +(zero-copy over the mmap). True zero-copy would require every +PairwiseTally query method (categories, base_freq, pair, ...) to +work generically over Archived<PairwiseTally> as well as the owned type +— a separate, larger change not needed to get the actual win (skipping +sample_index's expensive re-scan of the sibling annex; deserializing an +already-in-memory-sized count array is comparatively cheap CPU, not I/O). +#[derive(Archive, Serialize, Deserialize)] was added to PairStats, +PairwiseTally, and PartitionDispersion (pairwise.rs) for this.

    +

    Startup behavior, as designed: Session::open creates DIR if +missing (fresh, params recorded); if DIR already holds different saved +params, cmd/phylo reports a hard error and exits rather than silently +preferring one side, unless --session-force is given (which discards the +directory's cached artifacts and starts over under the new params) — +implements exactly the "explicit escape hatch, not silent override" +decision from the design discussion. obiksession itself doesn't decode +either side's bytes to produce a field-by-field diff (true to "opaque +blob," left to whoever needs it); cmd/phylo's conflict message is +currently a single generic sentence listing which flags could be the +cause, not a computed diff — an honest v1 simplification, not a limitation +of obiksession itself.

    +

    Per-artifact caching, as designed: only PairwiseTally/ +PartitionDispersion are cached (snp_distance.rs's restore_tally/ +store_tally) — ratio_ceiling/gamma_shape remain pure post-processing +over whatever tally is in hand, cached or fresh, exactly as planned.

    +

    Locking, reconsidered and kept as the whole-session DirLock. First +written up as a corner cut ("per-entry locking was sketched, not +implemented"). On actually working through the concrete race, whole-session +locking turned out to be the correct choice, not a simplification: +--session-force resets a session by deleting its whole artifacts/ +directory. If the directory lock were only held briefly (during +Session::open, then released so a long-running Session doesn't block +other processes from touching different artifacts), a second process +calling open(..., force: true) could acquire the lock, see mismatched +params, and wipe artifacts/ while a first process's already-open +Session is mid-checkpoint — genuine corruption risk, not just wasted +work. Holding the lock for the Session's entire lifetime makes that +sequence impossible: a force reset must wait for every currently-open +Session on that directory to be dropped first. The concurrency this +gives up (two different processes reading/writing different artifacts +of the same session at once) isn't exercised by anything today — only one +code path (snp_distance/sankoff_bundle, sequentially per invocation) +uses sessions at all.

    +

    Per-layer chunked/resumable dumps — implemented. subsample::sample_index +gained a skip_layers: usize parameter (skip the first N (partition, +layer) pairs, in its own deterministic iteration order, entirely — no +I/O, no on_layer call) and its on_layer callback gained a raw_index +argument (that pair's 0-based position in the same order), so a caller can +tell exactly how far a run has gotten. snp_distance/sankoff_bundle +checkpoint their tally/dispersion (and, for sankoff_bundle, the +in-progress SnpAlignment) to the session every +session_cache::CHECKPOINT_INTERVAL_LAYERS (8) layers or +CHECKPOINT_INTERVAL (30s) of wall time, whichever comes first — bounding +how much completed scanning a crash can lose, without paying a full +tally-rewrite on every one of what can be thousands of layers. A progress +artifact records how many layers are reflected in the checkpoint, with a +PROGRESS_COMPLETE (u64::MAX) sentinel once every layer has been +processed (distinct from any real layer count, so "finished" is never +confused with "stopped after N layers where N happens to equal the total").

    +

    Verified with a real kill: built a 10-genome/64-partition index, ran +--distance snp-jc --session DIR under timeout 0.4, confirmed a +checkpoint at progress=8 (exactly CHECKPOINT_INTERVAL_LAYERS) was on +disk, then re-ran the same command and confirmed it resumed from layer 8 +and completed successfully — 0-effort verification that the mechanism +itself is real, not just plausible-sounding.

    +

    Important correction from that same test — resume is not bit-reproducible, +and shouldn't be. The output matrix from the resumed run differed from +an uninterrupted control run against the same index/params. Root cause: +sample_layer's random draws come from rand::rng(), a per-process, +OS-seeded generator with no continuity across a process boundary — the +layers processed before a kill and the layers processed after a resume +come from two independent random streams. An earlier revision of this doc +(and of the code comments) claimed resume "reproduces the exact same +sample an uninterrupted run would have" — that's wrong, and was corrected +after the user pointed out the actual reasoning error: true randomness has +no notion of continuity in the first place, so there was never a +correctness requirement for a resumed run to match an uninterrupted one +bit-for-bit. The sample produced after a resume is exactly as legitimate a +random draw as one from an uninterrupted run — just a different one, +same as running the command twice without --session already gives two +different samples today. That variance is explicitly wanted (see below), +not a defect to fix. Comments in subsample.rs/session_cache.rs were +corrected to state this plainly instead of the false reproducibility claim.

    +

    Explicitly declined: deterministic/seeded sampling. Raised as a +possible fix for the above (seed sample_layer's RNG from a per-session +value plus (partition, layer), making a resume reproduce bit-identical +results, and, if applied everywhere, making any two runs with identical +parameters produce identical samples). The user explicitly declined this +for now: repeated runs are relied on to measure sampling variance, and a +--seed flag would need its own explicit design if wanted later — this is +not an accidental gap, it's a stated preference to revisit only if asked.

    +

    SankoffBundle integration — implemented. sankoff_bundle accepts +the same session: Option<&Session> and uses the same artifact names +(session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}) +as snp_distance, plus its own ALIGNMENT_ARTIFACT (sankoff.rs) folded +in lockstep with the tally at every checkpoint (same on_layer call +produces both, so they can never drift out of sync). Consequence verified +directly: a tally checkpointed by --sankoff --session DIR is restored by +a later, separate --distance snp-k2p --session DIR invocation with no +--sankoff at all, and vice versa — cross-consumer reuse, not just +same-command reuse, exactly the synergy this design aimed for.

    +

    Per-artifact caching, as designed: only the tally/dispersion (plus, +for sankoff_bundle, the alignment) are cached — ratio_ceiling/ +gamma_shape remain pure post-processing over whatever tally is in hand, +cached or fresh, exactly as planned.

    +

    Still not done — genuine follow-up, not a justified tradeoff: +exhaustive-sample caching for --subsample-varying exploration. The +design discussion's suggestion — cache the exhaustive (n = None) sample +once and subsample from it on each run instead of one session per exact +n — wasn't implemented; a session today is still scoped to one exact +parameter tuple, --subsample included.

    Output format: PHYLIP-relaxed by default for the distance matrix

    Implemented. The primary distance-matrix output (_dist.csv today) gains multiple formats: PHYLIP-relaxed becomes the diff --git a/DevDoc/theory/indexing.refs/index.html b/DevDoc/theory/indexing.refs/index.html index 890844ab..8dcb615c 100644 --- a/DevDoc/theory/indexing.refs/index.html +++ b/DevDoc/theory/indexing.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/theory/indexing/index.html b/DevDoc/theory/indexing/index.html index afbd21e6..be5ea79a 100644 --- a/DevDoc/theory/indexing/index.html +++ b/DevDoc/theory/indexing/index.html @@ -1199,7 +1199,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/theory/minimizer.refs/index.html b/DevDoc/theory/minimizer.refs/index.html index efa795a8..26fbf103 100644 --- a/DevDoc/theory/minimizer.refs/index.html +++ b/DevDoc/theory/minimizer.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/theory/minimizer/index.html b/DevDoc/theory/minimizer/index.html index f7d4cdc0..b789b4f2 100644 --- a/DevDoc/theory/minimizer/index.html +++ b/DevDoc/theory/minimizer/index.html @@ -1243,7 +1243,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDocMD/architecture/siblings.md b/DevDocMD/architecture/siblings.md index 352956f4..a9e32377 100644 --- a/DevDocMD/architecture/siblings.md +++ b/DevDocMD/architecture/siblings.md @@ -1,1024 +1,160 @@ -# Sibling annex — architecture (discussion) - -Status: architecture decided (2026-08-14). Implementation not yet mandated. - -## Two index spaces, uncorrelated - -Every kmer stored in a `Layer` lives in two independent index spaces: - -- **Iteration order**: its position when enumerating `unitigs.bin` (the - superkmer file), deterministic but arbitrary with respect to slot. -- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns. - -The two are not correlated by any formula. Converting from one to the other -requires either recomputing the MPHF (kmer → slot) or scanning the iteration -stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a -one-way function, not an invertible bijection with a stored inverse. Any -method that reconstructs a kmer from a bare slot number is wrong by -construction, regardless of the mechanism used (MPHF re-hash, or evidence -decode + direct unitig read). See `MphfLayer::kmer_at` -(`obikindex/src/layer/mphf_layer.rs`) — flagged for removal, currently called -from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see -"Pending work" status below). - -## Two pipelines, never mixed - -| | origin of the kmer | membership known? | correct mapping | -|---|---|---|---| -| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check | -| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence | - -Evidence exists solely to answer "is this external kmer a member of the -layer" for the query pipeline. Using it (or the MPHF) to go the other way — -recover a kmer from a slot, or re-verify a kmer that was just produced by -iterating the layer — is a conceptual error: evidence can be probabilistic -(`Approx` mode), so any slot→kmer attempt is unsound in general, and -pointless even in `Exact`/`Hybrid` mode since the kmer was already known. - -## Sibling annex: an iteration-pipeline artifact only - -The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`, -`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a -family minorant and which family members are present in the index. Its only -consumers (`obikphylo/siblings/stats.rs`, `family_scan.rs`) enumerate it -exhaustively (`0..annex.len()`); no query-pipeline code path touches it. - -**Decision**: the annex must be persisted in iteration order, not slot -order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex -file directly — one linear, cache-friendly pass, no MPHF/slot indirection, -no `kmer_at`. It also enables specialized iterators building on this zip: -minorants-only iteration, batch-of-kmers → batch-of-family-members, etc. - -Today the annex is built and stored in **slot** order -(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated -via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is -threaded through the whole cross-partition reconciliation pipeline (variant -generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This -must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading -the **iteration index** instead of the slot end to end — eliminating -`kmer_at` from the build path entirely, not just the read path. No -slot-indexed intermediate is needed even during construction; the -iteration-order id is sufficient throughout. - -The cross-partition side of the same pipeline is unaffected: checking -whether a generated family-variant kmer exists in another partition is a -genuine query-pipeline operation (the variant's membership in the *target* -partition is unknown) and must keep going through -`KmerPartition::query_partition_with` (MPHF + evidence), never a raw -`index()`. - -## Pending work — done - -The plan above shipped: `obikphylo` (a new crate — phylo-domain extension -traits over `obikindex::KmerIndex`/`obikindex::layer::Layer`, replacing the -old `obikindex::siblings` module) builds and reads the annex purely in -iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with -batch variants, mirroring `Layer`'s own `KmerIter`/`KmerBatchIter` -shape). `MphfLayer::kmer_at` has no remaining callers. - -A separate, unrelated bug surfaced during this work and was fixed -(2026-08-14): `MphfLayer::enumerate_kmers_batch` computed its -`batch_start_index` via the stdlib `.enumerate()` adapter, which counts -*batches* (0, 1, 2…), not the cumulative k-mer offset the annex is actually -keyed on — every batch past the first wrote its mask/annex entries at the -wrong iteration-order position. Fixed by tracking a running offset instead; -regression tests added (`sibling_annex_no_empty_masks_after_build`, -`sibling_histogram_does_not_panic_on_partial_last_batch`). - -## Performance: `build_sibling_annex` parallelism (2026-08-14) - -Investigated on a real multi-genome run (`phyloskims_sal_vac`, k=31/m=11). -Baseline: mostly one active core, with short multi-core bursts — average -~3 cores. - -**Fixes that helped, kept:** - -- `CanonicalKmerOf::minimizer()` (`obikseq/src/kmer.rs`) — a direct O(k) - bit-arithmetic minimiser for a single isolated k-mer, replacing a - `RollingStat` instance fed byte-by-byte through an ASCII round-trip (used - by `helpers::partition_of`, called for every generated family variant). - ~3x wall-clock improvement on its own, confirmed by sampling - (`obiskbuilder::rolling_stat`/`obikentropy` frames disappeared from the - hot path). `CanonicalKmerOf::partition()` added alongside it (wraps - `minimizer().seq_hash() & mask`, the same routing rule - `KmerPartition`/`RoutableSuperKmer` use). -- Cross-partition resolution (`outgoing.par_iter()` in - `build_layer_sibling_annex`) parallelised at the *partition* level — one - Rayon task per non-empty `outgoing[dest]` bucket. For k=31/m=11, a - central-base substitution changes the winning minimiser (and thus the - destination partition) only when that window overlaps the central base: - ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants - route right back to the partition already being built. That self bucket - ends up far larger than any other, so the per-partition split pinned one - thread to it alone while the rest of the pool finished instantly — - confirmed by sampling: one thread solid in `MphfLayer::find`, everyone - else idle. Fixed by splitting each non-empty bucket into - `total_queries / n_workers` (capped 4096) chunks *before* `par_iter()`, - preserving per-partition mmap locality (each chunk stays contiguous - within one partition) while letting Rayon spread an oversized bucket - across several threads. Net effect of both fixes together: ~3 cores - average → ~10-13 cores average on the same run, and a projected total - build time of ~1h15 down to ~30min on the real `phyloskims_sal_vac` run - this was measured against. -- `TracedBar`'s ETA (`obisys/src/progress.rs`) was silently starved: the - custom progress message and the self-computed ETA text used to share one - `pb.set_message()` slot, with the ETA holding off for 2s after any custom - message — fine when custom messages are rare, broken once - `build_sibling_annex`'s per-partition callback fires more often than - that. Fixed by keeping the two texts in separate fields, composed - together on every render instead of one overwriting the other. - -**Tried and reverted — do not repeat blindly:** - -- Parallelising the *outer* partition loop in `build_sibling_annex` with - `obikindex::PartitionRunner` (already used by `merge`/`build_layers`), - splitting a fixed core budget between outer (partition) and inner - (pipeline + resolution) concurrency so their product wouldn't exceed the - budget. Measured *worse*: throughput dropped over time (26 - partitions/5min → 38/11-12min) and peak resolution concurrency fell from - ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which - scales very well on its own — to make room for outer concurrency, and - running several partitions' resolution at once scatters access across - multiple partitions' mmap regions at once, working against the - locality `outgoing`'s per-partition grouping exists for. `PartitionRunner` - stayed exported from `obikindex` (`new_capped` too) since it's - general-purpose, but nothing in `obikphylo` calls it. -- Splitting resolution chunks even finer (`/(n_workers*8)`, cap 1024, - instead of `/n_workers`, cap 4096) to smooth the residual sawtooth. - Measured ~10% *slower*, wider dips, not narrower. Reverted to the - original chunk sizing. - -**Known remaining limitation, not yet worth fixing:** within one layer, the -four stages (sequential `unitigs.bin` read → parallel generation → -parallel resolution → sequential annex write) never overlap — confirmed by -1s-interval sampling: generation alone occupies ~17 threads evenly, but the -next layer's read/generation never starts until the current layer's -resolution and write are both done. This produces a real, periodic (~layer -duration) alternation between "many cores" and "few cores" that neither of -the fixes above touches, since both operate *within* one layer's resolution -step. The only remaining lever is overlapping consecutive layers (e.g. a -depth-2 pipeline: start layer N+1's read/generation while layer N's -resolution/write is still running) — a real restructuring, not a parameter -tweak, and explicitly *not* to be combined with the reverted -budget-capping idea above (let each phase use however many cores it -naturally wants; only the *scheduling* needs to overlap). Deferred, not -started. - -## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14) - -`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`, -`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already -implements most of a dispatch/accumulate/resolve pipeline: generation -(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and -buckets cross-partition queries) runs on an `obipipeline::throttle` + -`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter` -*across partitions*, one generated batch resolved at a time, never several -concurrently — this ordering is deliberate, see the module's own docs on a -reverted concurrent-batch-resolution attempt that scattered mmap access). -The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`) -already exists: `n_layers <= 7` (checked once from the first non-empty -partition's `PartitionMeta::n_layers`, documented as identical across every -partition of an index — a structural, build-time property, never a -per-partition state) decides whether `FamilyMask`'s recorded `layer_value` -can be trusted to skip straight to the right layer -(`find_presence_batch_fast`) or must fall back to scanning every layer of -the destination partition (`find_presence_batch`). - -**Real gap, confirmed not implemented**: resolution is triggered by the -*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the -scanned layer), not by an *output* accumulator filling up. Since most -central-base variants of a family route back to the same partition being -scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s -`outgoing[dest]` is large for the local/self partition and thin for the -other ~255 (or however many) destination partitions — each of those gets -resolved at low query density every batch instead of being accumulated -across several source batches until resolving it is worthwhile. This is -distinct from, and not fixed by, the fast/slow layer gate above. - -Redesign sketched (not built): per-destination accumulators decoupled from -`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch -completion — a "hot" accumulator for the partition being scanned (sharded -one-per-generation-worker, no lock, since all `n_workers` pipeline workers -write to it concurrently — this differs from an earlier, simpler mental -model of "one thread owns one layer's local collector," which doesn't hold -here since `n_workers` threads cooperate on scanning *one* layer at a time, -not one thread per layer) and "cold" mutex-per-partition accumulators for -the rest, low contention expected since traffic to any single cold -destination is a small fraction of total. - -This breaks the current strict-iteration-order delivery of `on_family` -(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH` -resolves atomically). With cross-batch accumulation, a family only becomes -complete once *every* accumulator holding one of its outgoing queries has -flushed, at unpredictable, independent times — no longer streamable -strictly in order without a large, unbounded pending buffer. Resolution -sketched: replace order-dependent consumers with coordinate-addressed -writes instead of order-dependent appends (see `PseudoAlignment` idea -below) wherever possible, since `sibling_annex_stats`'s reduction (plain -counts) is already order-independent and needs nothing here. - -**Superseded 2026-08-15** by the `--subsample`/`--shannon` design below, -which sidesteps the accumulator redesign for now: bounding the number of -families actually resolved per layer (via sampling) keeps per-layer -resolution volume small enough that the batch-density problem above stops -mattering in practice for these two consumers. The accumulator redesign -remains relevant for a future *unsampled, full-index* run, but is not -required to ship `--subsample`/`--shannon`. - -## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15) - -The reference run (`phyloskims_sal_vac`-scale bacterial test set, -`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites -(4.8 MB) — trivially small. The in-progress plant index is expected to -carry on the order of 9 billion minorant families; a dense byte-per-cell -alignment at that column count is unbuildable regardless of genome count -(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000– -8,000 genomes on a large machine, which makes the per-cell cost dominant in -the other dimension too. Pruning the retained family set before -materializing anything is mandatory, not an optimization. - -**Already free**: `family_size() < 2` (no sibling variant registered at -all) is a zero-cost structural filter, read directly off `FamilyMask` bits, -already applied in `snp_pseudo_alignment`. Insufficient alone — per the -~80-85% mono-family estimate from earlier discussion, this only brings 9 -billion down to roughly 1.3-1.8 billion, still unusable. - -**Entropy definition — settled 2026-08-15, correcting an earlier wrong -turn.** The project does **not** encode families as IUPAC ambiguity codes -interpreted the classical way (Fitch-parsimony subset-compatibility, or -ML's "one true state, uncertain which"); see -`docmd/theory/evolutionary_distances.md` ("Why the IUPAC/DNA encoding used -for the first `--snp` test was wrong") and the Sankoff resolution that -followed it. The real model is a genuine 16-state alphabet (the powerset of -`{A,C,G,T}`, `∅` included as a real state) scored with a *calibrated -pairwise cost matrix* (`obikphylo::cardcomp::pairwise_cost_matrix`, -`cmd/phylo/sankoff.rs`), not a compatibility/subset relation between -states. Under that model, each of the 16 states — including multi-bit ones -like `AC` — is a first-class, independently-costed state, not an -uncertainty encoding of a single true base. So: **entropy over the 15 -non-empty states (`∅` excluded, matching the earlier decision to exclude -genomes where the family is absent) is the correct informativeness -measure** for this project — not a 4-symbol reduction, which would discard -exactly the cardinality/composition information the calibrated cost matrix -is built to exploit. - -## `--subsample` / `--shannon` — sampling strategy (decided 2026-08-15) - -Goal: make both the pseudo-alignment (`--snp`) and a Shannon-entropy -diagnostic usable at any index scale, from the 13-genome bacterial -reference run up to the 9-billion-family plant index, without requiring the -batched-accumulator redesign above. - -**`--subsample N`** (integer, families to retain): bounds the pseudo-alignment -to `N` minorant families, sampled **proportionally per layer** among -non-monomorphic minorants (`family_size >= 2`) — this sidesteps the need -for a true global reservoir merge across layers while still approximating a -uniform sample over the whole index, and directly answers the earlier open -question of global-vs-per-layer selection scope. - -Three passes, in order: - -1. **Global count** (cheap, structural, parallel across layers — same shape - as the existing `sibling_family_size_histogram`, extended to report a - **per-layer** breakdown rather than one index-wide aggregate): for each - layer, `count_layer` = number of non-monomorphic minorants. Gives - `total_count = Σ count_layer`. -2. **Per-layer proportional reservoir sampling** (cheap, structural, one - pass per layer, no cross-partition resolution): `N_layer = round(N × - count_layer / total_count)`. Since `N_layer` is a proportion of - `count_layer`, it can never exceed it as long as `N <= total_count` — the - one edge case is `total_count <= N`, in which case sampling is skipped - entirely and *every* non-monomorphic minorant of every layer is kept - (no reservoir needed, `N` was never a real constraint). Otherwise: - Algorithm-R reservoir sampling over the layer's non-monomorphic minorant - indices, producing `N_layer` iteration-order indices directly, no - intermediate full list ever materialized. -3. **Filtered resolution** (the expensive step, the existing - `scan_layer_families` engine, unchanged): re-scan the layer, generating - and resolving cross-partition queries **only** for the indices selected - in step 2 (cheap membership test against a small per-layer index set) — - this is what keeps `--subsample` cheap even on an unsampled-scale index, - since the cross-partition resolution volume is bounded by `N`, not by - the layer's true size. - -Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir -sampling would waste step-3's expensive resolution work on candidates later -evicted by the reservoir. Step 1 must fully complete (every layer) before -step 2 can start for any layer, since `total_count` is a global quantity. - -**`--shannon`** (no argument): emits a CSV of per-family Shannon entropy -(15 non-empty states, `∅`/absent genomes excluded from the denominator, per -the settled definition above). Independent of `--subsample` — entropy is -computed and written per family as soon as its `genome_mask` resolves, -O(1) memory per family, so it streams fine even unsampled at full index -scale (a time cost, not a memory one). Combined with `--subsample N`, it -delivers the original exploratory diagnostic (e.g. `--subsample 1000000 ---shannon`) directly from this general machinery, rather than a -purpose-built one-off script. - -Validated end-to-end (2026-08-15) against real data: `--sibling-hist` on -`phyloskims_sal_vac` (91 real genomes, k=31/m=11, 256 partitions × 2 -layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238 -total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier -guess, corrected here). `--subsample`/`--shannon` on the smaller 20-genome -bacterial reference (`benchmark/global_index_presence`) produced a sample -size within rounding of the request (99,742/100,000) and a -[0.8,1.2)-bucket share (40.7%) matching the full unsampled population -(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a -real bug in `reservoir_sample_layer` (see next section), now fixed. - -**Bug found and fixed (2026-08-15): `family_idx` numbering mismatch.** -`scan_layer_families`'s `family_idx` counts *every minorant* of a layer -(monomorphic ones included, since `iter_minorants_batch` filters only on -`is_minorant()`), not the raw annex slot (`SiblingAnnex::get(slot)` spans -every k-mer, minorant or not) and not a counter over non-monomorphic -minorants alone. `subsample.rs`'s `reservoir_sample_layer` originally -stored raw slot numbers in its `HashSet` selection, which drifts -away from `family_idx` as soon as *any* monomorphic minorant is seen — -i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by -tracking two separate counters: `family_idx` (every minorant, matching -`scan_layer_families`) and `seen` (non-monomorphic minorants only, what -Algorithm R actually samples over) — only `family_idx` values are ever -stored in the selection set. The existing unit test never exercised this -(its fixture has exactly one non-monomorphic family, always hitting the -"keep everything" shortcut) — a stronger fixture with several interleaved -monomorphic/non-monomorphic families would be needed to catch a regression -here automatically; not yet written. - -## Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15) - -Motivation: on real data (bacterial reference, full unsampled run), only -~5‰ of non-monomorphic minorants fall in the `[0.5, 1.5]` bit band judged -phylogenetically interesting (entropy too low = uninformative near-invariant -site; too high = saturated/noisy, see `family_entropy`'s 15-state -discussion) — roughly 1 in 10,000 minorants overall. Computing exact -entropy for every candidate just to discard 99.99% of them is wasteful at -the full 9-billion scale. - -**The idea**: `PersistentBitMatrix::col_view(c)` gives a genome's whole -presence column as a `BitSliceView` (sequential, no MPHF, no cross-partition -routing — purely local to one layer's own matrix). Accumulating -`TempCompactIntVecBuilder::inc_present(col)` (already exists in -`obicompactvec/src/builder.rs:121`, along with `add`/`min`/`max`/`diff` on -`IntSliceView` — no new low-level API needed) over every column of a layer -produces `coverage[slot]`: how many genomes carry each exact k-mer, in one -sequential per-layer pass, entirely decoupled from family/sibling -structure. Persisted once per layer, a family's members' coverage could -then be looked up via a plain local MPHF `index()` (cheap) instead of a -full cross-partition presence resolution (`find_presence_batch`) — the -expensive part today is specifically the cross-partition/cross-layer -routing to a sibling's own matrix, not the bit-reading itself, and -`coverage[slot]` sidesteps that routing entirely by moving the cost into a -one-time, purely local, embarrassingly-parallel build step. - -**Why not implemented**: `coverage[slot]` is a per-member marginal — -summing members' coverages to approximate a family's entropy silently -assumes no genome carries more than one member at once. It cannot -represent or detect joint co-occurrence (a genome carrying both `A` and -`C` at once, i.e. a combined 15-symbol state) at all, which is exactly the -phenomenon `family_entropy`'s 15-state definition exists to capture (see -`CardinalityTally`/`cardinality_transition_probs`, the project's own -existing machinery for this same co-occurrence structure, built for the -Sankoff matrix calibration). A family that is in truth uniformly `AC` -across every carrying genome would look like a well-balanced 2-state split -under the marginal approximation (entropy ≈ 1) while its true 15-state -entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the -"saturated, uninformative" tail this pre-filter would need to catch, -undermining the point. It stays plausible as a coarse filter for the -*low* tail only (a dominant single member's marginal share reliably -predicts low true entropy too), but not as a stand-in for the high tail — -not pursued further for now. - -## `--free-loss`/`--tnt` pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below) - -Measured on `phyloskims_sal_vac` (91 genomes): `obikmer phylo --subsample 500000 ---free-loss --tnt` logs four sequential stages — -`raw_snp_distance` (1413s), `base_pair_tally` (1456s), -`cardinality_tally` (2078s), `snp_pseudo_alignment` (143s). Reading the -code (`obikmer/src/cmd/phylo/mod.rs:210-242`, -`obikphylo/src/siblings/distance.rs`, `cardinality.rs`, `alignment.rs`) -surfaced two compounding problems, not one: - -1. **Four separate full scans of the annex**, each opening its own - `KmerPartition`/`PartitionCache` and calling `scan_family_pairs`/ - `scan_layer_families` independently — nothing computed in one stage is - reused by another. `base_pair_tally` is explicitly documented - (`distance.rs:138-143`) as a second full pass over the same - traversal `raw_snp_distance` already did, needed only because - `raw_snp_distance` doesn't keep the resolved bases, only aggregate - counts. `cardinality_tally` and `snp_pseudo_alignment` are each a - third and fourth independent full pass. Per the module's own earlier - profiling note (`family_scan.rs:26-28`, cited already above), this - traversal is page-fault/mmap-bound, not compute-bound — the ~10-12% - CPU efficiency ("contention" status) observed on these three slow - stages is consistent with I/O stalls scaled by repeated full scans, - not lock contention (there are no `Mutex`/`RwLock` anywhere in - `siblings/*.rs`; shared writes use per-slot `AtomicU8::fetch_or`). -2. **Worse: `raw_snp_distance` and `cardinality_tally` don't honor - `--subsample` at all** — both call `scan_layer_families` with - `Selection::All` hardcoded, and `args.subsample` isn't even threaded - into their function signatures (`mod.rs:212`, `mod.rs:226`). Only - `snp_pseudo_alignment(args.subsample)` builds a real reservoir-sampled - `Selection::Some(set)` (via `compute_selections`, `alignment.rs:89`). - So today, `--subsample 500000` only bounds the pseudo-alignment step — - the SNP-distance matrix, the Sankoff base-pair calibration, and the - cardinality histogram are always computed over the **full, unsampled** - index regardless of the flag. This is not merely "different subsamples - per stage" (which would already be a problem worth fixing) — it's that - three of the four stages never subsample, which explains most of the - ~10x runtime gap against `snp_pseudo_alignment` on its own. - -**Decided requirement**: all four stages consume **one shared selection**, -computed once, not each stage either scanning everything or drawing its -own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair -calibration, the cardinality histogram, and the pseudo-alignment all -describe the same set of families — the calibration and the alignment it -calibrates are now guaranteed to agree on which sites exist. - -**Correction to the "single pass" framing above**: `base_pair_tally`/ -`cardinality_tally` both need `raw_snp_distance`'s *complete* aggregate -SNP/shared counts before they can derive `included[i,j]` (the -`ratio_ceiling` filter) — a genuine sequential dependency (`included` -can't be known until every pair's aggregate count is final), not an -artifact of the old code's structure. So the fix is **two** passes over -the shared selection, not one: pass A computes the aggregate counts (and -derives `included`); pass B fuses `base_pair_tally` + `cardinality_tally` -+ the pseudo-alignment (mutually independent once `included` is known) -into a single scan. Still a 4→2 reduction, and — per the clarification -that settled this — pass A itself now runs over the *same shared -selection* pass B uses (not the full unsampled index): "les ratios, on -les fait sur les sites sélectionnés, c'est tout, les autres sites -n'existent pas" — once a selection is chosen, both passes are bounded by -it, so on a real `--subsample`/`--entropy` run pass A is cheap too, not -just pass B. - -**Entropy-biased selection's own resolution to the "forward-looking -complication"** (entropy must be known before selection, but selection -happens during the same scan that would resolve it): see "Entropy-biased -selection" below — resolved via a persisted per-layer entropy annex, not -by restructuring the scan into an inline pre-pass. - -## Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15) - -Refines the "forward-looking complication" above with a concrete -mechanism. Instead of a hard `[low, high]` entropy band (or any other -exact cutoff) deciding which non-monomorphic minorant families are -eligible, selection is weighted by an **unnormalized Gaussian kernel** -centered on a target entropy: `w(entropy) = exp(-(entropy - μ)² / (2σ²))` -— deliberately not the normalized Gaussian density (which would peak -below 1 and complicate the "probability" reading) — this kernel form -equals 1 exactly at `entropy = μ` and decays smoothly to 0 away from it, -so it reads directly as an acceptance weight: the further a family's -entropy from the target, the less likely it is picked, with no hard -in/out boundary — a few "bad" sites can still get in, by design. `μ` -(default ~1.0) and `σ` (default ~0.5) are meant to be user-tunable. - -**Mechanism: joint probability, not a weighted reservoir**. Not -Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex -proposal, superseded before implementation) — instead, a single -independent accept/reject draw per qualifying candidate: draw -`u ~ Uniform(0,1)`, accept iff `u < p₀ · w(entropy)`. `p₀` is a single -**index-wide** rate, `N / total_count` (`total_count` = the sum of -`non_monomorphic_counts` across every layer, `N` = `--subsample`'s -target), applied identically at every layer — this alone already gives -each layer its proportional share (the same effect the old uniform -reservoir's explicit per-layer `n_layer = N · count_layer / total_count` -computation achieved, but without needing to compute it: applying one -rate uniformly is mathematically the same as proportioning per layer). -`p₀ = 1.0` when there is no `--subsample` at all — `--entropy` alone is a -pure soft entropy filter over the whole index, no size target. Properties: -(1) **strictly generalizes the existing uniform sampler** — with `σ` large -enough that `w ≈ 1` everywhere, this reduces to the old uniform `N/total_count` -draw; (2) yields "approximately N", not exactly N — expected accepted -count is `N · mean(w)`, always `≤ N` — an intentional relaxation, matching -"sous-échantillonnage à environ n" rather than the old reservoir's exact-N -guarantee; (3) one streaming pass, one random draw per candidate, no -reservoir state. - -**Resolving "entropy must be known before selection, but selection -happens during the same resolving scan"**: solved with a **persisted, -per-layer entropy annex** (`obikphylo/src/siblings/entropy_annex.rs`, -`EntropyAnnex`/`EntropyAnnexBuilder`), not by restructuring the scan. -Mirrors `SiblingAnnex`'s mmap-backed, read-only-after-build convention, -but indexed by `family_idx` (every minorant of the layer, monomorphic -included — the same numbering `Selection`/`scan_layer_families` already -use), one `f32` entropy15 value per entry, `-1.0` sentinel for monomorphic/ -not-yet-computed. First use of `--entropy`/`--entropy-sd` on an index -pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`) — every -later run (any `μ`/`σ`, any command) reads the file positionally, no -re-scan, restoring the usual `Selection::Some` "skip resolving excluded -families" speedup that a naive "weigh during the resolving scan" design -would have permanently forfeited. - -**Bug found and fixed (2026-08-15): `ensure_entropy_annexes` scanned with -`Selection::All` instead of bounding to non-monomorphic minorants.** -Monomorphism (`family_size() < 2`) is knowable directly from the annex -bits alone, no per-genome resolution needed — but the original -implementation called `scan_layer_families` with `Selection::All` -anyway, so `fill_sub_matrix_carries` (the expensive per-genome -resolution) ran for *every* minorant, ~98% of which are monomorphic -(measured elsewhere in this doc) and had their `genome_mask` immediately -discarded once the callback checked `family_size() < 2`. Fixed by adding -[`subsample::non_monomorphic_selection_layer`] — a cheap, annex-only, -non-sampling pass (same shape as `reservoir_sample_layer`, but keeping -every non-monomorphic minorant's `family_idx` instead of a bounded -reservoir) — and passing `Selection::Some(&eligible)` instead of -`Selection::All`, so the expensive resolution now runs only for the ~2% -of minorants that can actually produce a real entropy value. A -`debug_assert!(mask.family_size() >= 2, ...)` inside the -`scan_layer_families` callback guards the invariant. - -**Resolved**: the existing hard "non-monomorphic minorant" eligibility -filter stays a hard gate upstream of the Gaussian weighting — only -qualifying families ever get a stored entropy value or a weighted draw. - -**CLI, implemented**: two `phylo` options, `--entropy <μ>` and -`--entropy-sd <σ>` (`obikmer/src/cmd/phylo/args.rs`). The entropic filter -activates as soon as *either* is given (`mod.rs`, computed once into an -`Option` threaded through `--snp`/`--family-overlap`/ -`--shannon`/the fused sankoff pipeline below). If active but one or both -are unset, defaults are `μ = 1.0`, `σ = 0.5`. - -## Two entropy definitions kept side by side, for comparison (2026-08-15) - -`--shannon`'s CSV carries both `entropy15` (`family_entropy` — the settled -15-non-empty-state definition, see above) and `entropy4` (`family_entropy_4` -— plain nucleotide reduction), computed from the same already-resolved -`genome_mask`, not from the marginal approximation above. A genome carrying -several bases at once contributes to *each* base's count (counted once per -base present, not fractionally split, not folded into one combined state) -— a genome polymorphic for the family is present at more than one base by -construction, so it is expected to count more than once; the denominator is -the total base-occurrence count, not the genome count (the two coincide -only when no genome carries more than one base). Kept side by side -specifically to measure, on real data, how much the two diverge — not yet -analyzed. - -## `PersistentSparseBitMatrix` — implemented and measured (2026-08-15) - -A row-major (k-mer-major), deduplicated sparse alternative to -`obicompactvec::PersistentBitMatrix`, motivated by the same sparsity that -drove `--subsample`/`--shannon` above, but pursued as a foundational -storage-layer change rather than an index-level workaround. Full design -history, rationale, and rejected alternatives (external Elias-Fano crates, -`cacheline-ef`, a single unsplit `dict_id` array) are in the dedicated -implementation plan (`vivid-mapping-tiger.md` at the time of writing — the -content below is the durable summary, not a pointer to a session-scoped -file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi, -"Compact Data Structures for Collections of Sets" (2025, -`biblio/Alanko et al. - Compact Data Structures for Collections of -Sets.pdf`) — this design implements only their exact-duplicate special -case (a plain dedup dictionary), not their full subset-containment -hierarchy. - -**Design**: four on-disk components, each mmap-backed, built once per -layer (matching how the rest of the build pipeline already works — never -the whole multi-billion-row index at once): an `is_multi` rank-capable -flag per row (singleton vs. multi-genome), a fixed-bit-width array for -singleton rows (genome index directly, `ceil(log2(n_cols))` bits), a -separate fixed-bit-width array for multi-genome rows (`dict_id`, -`ceil(log2(n_distinct_multi_sets))` bits — kept apart from the singleton -array specifically because `n_distinct_multi_sets` can be large in -absolute terms even when multi-genome rows are a small *fraction* of all -rows, and a single shared array would force every row, singletons -included, to pay the wider width), and a deduplicated dictionary of -distinct multi-genome sets (Elias-Fano-encoded byte offsets + a -varint-encoded values blob). New low-level primitives added to -`obicompactvec` to build this: `PersistentFixedIntVec` (arbitrary, -runtime-parameterized bit width, width 0 included — needed once a real -bug surfaced, see below), `PersistentRankSelectBitVec` (rank1/rank0/select1 -on top of the crate's existing `count_ones`, using -`common_traits::SelectInWord`), `EliasFano` (composes the two). A new -`BinaryMatrix` trait (`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, -`count_ones`) unifies dense and sparse at the one call site that needs -both interchangeably (`obikphylo::siblings::cache::Mat`) — column-oriented -methods (`col`, `col_view`, the `partial_*_dist_matrix` family) stay -dense-only. - -**Two real bugs caught by tests, not by inspection**: (1) `EliasFano::open` -re-derived its low-bits width from the persisted low-vector file's own -width byte; the zero-width case was built with a dummy 1-bit placeholder -(the builder rejected true width 0), so every reopened value silently -doubled. Fixed by making `PersistentFixedIntVec` genuinely support width 0 -(no storage, `get` always 0) instead of working around the limitation in -`EliasFano`. (2) An empty row (cardinality 0 — not expected on a real -built index, but not guarded against either) was recorded as a singleton -at genome 0, indistinguishable on read-back from a *real* singleton at -genome 0. Fixed by routing cardinality-0 rows through the dictionary path -(a genuine empty entry) instead of the singleton shortcut. Both caught by -`obicompactvec`'s test suite (142 tests, including disk-reopen round-trips -that drop every builder/mmap before reopening fresh), not by manual -review — worth remembering next time a "this edge case can't happen in -practice" shortcut is tempting. - -**Measured on real data** (`layer_1` of `phyloskims_sal_vac`'s -`part_00018`, 30,246,774 rows, 91 genomes — `#[ignore]`d benchmarks in -`obikphylo/src/siblings/tests.rs`): - -| | dense | sparse | ratio | -|---|---|---|---| -| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller | -| build time / peak RSS | — | 4.26s / 628MB | (per-layer, in-memory construction — comfortable) | -| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse **faster** (smaller structure, better cache fit) | -| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** (the real `--shannon`/family-lookup shape) | -| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** (no native column method — every read decodes a full row to keep one bit) | - -The row-access wins (both directions) weren't the design's stated goal — -compactness was — but turn out real: dense's genome-major layout scatters -a single row read across a much bigger file, which costs more than -sparse's rank/select/varint decode once the file is this much smaller. -The column-access cost is the flip side of the same layout choice, and is -exactly what the next item below exists to fix. - -**Next, not yet planned**: rewrite `partial_jaccard_dist_matrix`/ -`partial_hamming_dist_matrix`/etc. (`obicompactvec/src/bitmatrix/pairwise.rs`) -as a row-major co-occurrence accumulation (`O(Σ_rows k²)`, per-row -increments into an `NxN` genome-pair counter — the known alternative to -today's column-fold, plausibly cheaper on data this sparse, not just a -fallback) so `obikindex`'s `--metric`/distance-matrix path can use the -sparse type without the measured 86x column-access penalty. Needs its own -design pass (in particular how it plugs into the `BitPartials`/ -`ColumnWeights` traits so both matrix types keep serving `--metric`) -before implementation — not just "port the loop", a genuinely different -algorithm. - -Also still deferred, unchanged from the implementation plan: full Alanko -et al. subset-hierarchy compression (only the exact-duplicate special case -is built), a sparse `PersistentCompactIntMatrix` (count matrices), and -BRWT-style column-correlation exploitation. - -## Wired into `pack` and the sibling-annex build path (2026-08-15) - -`PersistentSparseBitMatrix` went from a validated but unused type to a -real, selectable on-disk format: - -- **Generic `Layer`**: `obikindex::layer::Layer`'s presence-only methods - (`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any - `D: LayerData> + BinaryMatrix`, not hardcoded to - `PersistentBitMatrix` — `PersistentSparseBitMatrix` implements - `LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were - already generic over any `D: LayerData`, so they needed no change. - Verified by `obikindex::layer`'s - `presence_layer_generic_over_sparse_matches_dense` test: build a dense - presence layer, convert it to sparse via `build_from_dense`, open both - as `Layer`/`Layer` on - the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree. - (This test must stay at `k=4` with mutually non-colliding canonical - 4-mers across its input sequences — `K`/`M` are process-wide - `AtomicUsize`s in test builds, not thread-local, so a test using a - different `k` races every other test in the same crate binary; a k=11 - version of this test passed alone but failed under the full - `obikindex::layer` suite for exactly that reason before being fixed.) -- **`obikphylo::siblings::cache::Mat`** gained a third variant, - `SparsePresence(Layer)`, alongside `Count` - and `Presence` — every method (`find_slot`, `index_batch`, - `iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches - to it identically to `Presence`, since both go through the same generic - `Layer` code. `PartitionCache::build` picks the variant per layer by - checking for `presence/is_multi.prsb` (the sparse format's own marker - file, see the design section above) before falling back to the dense - open path. -- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`): - `pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb` - first); packs to dense `matrix.pbmx` first if that hasn't happened yet - (the dense→sparse transpose needs random row access, which only the - packed/columnar dense forms give), then `build_from_dense`s the sparse - form into the same directory and deletes `matrix.pbmx` — old-format - files are removed only after the new format is fully written, mirroring - `pack_bit_matrix`'s own crash-safety convention. -- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through - `KmerIndex::pack_matrices` (all other call sites — `select`, `merge`, - `finalize_indexed` — pass `false`, unchanged dense behaviour). Count - matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix` - — see "still deferred" above). -- **End-to-end coverage**: `obikphylo::siblings::tests:: - sibling_annex_works_after_pack_sparse` builds a two-genome index, packs - it `--sparse`, asserts `is_multi.prsb` exists, then runs - `build_sibling_annex` and checks the resulting `FamilyMask`s match the - dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the - sparse format round-trips through the real build pipeline - (`PartitionCache` sparse-detection included), not just the - `obicompactvec`/`obikindex::layer` unit layers below it. - -Full workspace `cargo test` (all crates, unit + doc tests) green after -this change. - -## Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15) - -Replaces the "four independent scans" problem above and implements -"Entropy-biased selection" above, end to end: - -- **`SankoffBundleExt::sankoff_bundle`** (new, - `obikphylo/src/siblings/sankoff_bundle.rs`) — the `--sankoff`/`--tnt`/ - `--phyg`/`--iqtree` block in `obikmer/src/cmd/phylo/mod.rs` now calls - this once instead of three separate `raw_snp_distance`/ - `base_pair_tally`/`cardinality_tally` calls. One `PartitionCache`, one - shared (possibly subsampled/entropy-biased) selection computed once via - `compute_selections`, two scans over it: pass A (aggregate SNP/shared - counts, `--exclude-genome` zeroing, then `included[i,j]`), pass B - (`base_pair_tally` + `cardinality_tally` + the pseudo-alignment, fused - into one `scan_layer_families` call per layer, all three read off the - same resolved `genome_mask`). `snp_pseudo_alignment`/ - `shannon_entropy_csv` (still used standalone by `--snp`/ - `--family-overlap`/`--shannon`) both gained an `entropy_bias` parameter - too, so entropy-biased selection isn't sankoff-specific. -- **Regression proof**: `sankoff_bundle_matches_old_separate_calls` - (`obikphylo/src/siblings/tests.rs`) asserts `sankoff_bundle`'s four - outputs are bit-identical to calling the old, separate - `raw_snp_distance`/`base_pair_tally`/`cardinality_tally`/ - `snp_pseudo_alignment` on the same fixture with no subsample — the - fusion is a performance change, not a behavior change. -- **`EntropyAnnex`/`EntropyAnnexBuilder`** (new, - `obikphylo/src/siblings/entropy_annex.rs`) and **`ensure_entropy_annexes`** - (`entropy.rs`) implement the persisted-entropy mechanism from - "Entropy-biased selection" above. `entropy_annex_builds_on_demand_and_biases_selection` - (`tests.rs`) proves, on a fixture with one known-entropy family: the - annex file doesn't exist before any entropy-biased call; `compute_selections` - builds it on first use; `μ` set to the family's exact entropy with - `p₀ = 1.0` selects it deterministically (`u < 1.0` always, for - `u ∈ [0,1)`); `μ` set far away with tiny `σ` deterministically excludes - it (`w` underflows to exactly `0.0`); the persisted value matches - `--shannon`'s own `family_entropy` computation to `1e-6`. -- **`EntropyBias`** (`pub`, `obikphylo::siblings::EntropyBias { mu, sigma }`) - is the one new public type threading `--entropy`/`--entropy-sd` through - every `Option`-accepting method — resolved once in - `obikmer/src/cmd/phylo/mod.rs` from `args.entropy`/`args.entropy_sd` - (activation: either given; defaults `1.0`/`0.5` for whichever is unset). - -Full workspace `cargo test` green after this change (167 unit tests in -`obicompactvec`+`obikindex::layer`+`obikphylo` alone, plus every other -crate's suite, no regressions). - -**Still open, not part of this change** (per "Correction to the 'single -pass' framing" above): `--raw-snp-distance`/`--raw-snp-counts` (the -standalone diagnostic flags, not the `--sankoff` pipeline) still always -scan the full unsampled index — never threaded `--subsample`/`--entropy`, -out of scope here since the reported problem was specifically about the -`--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these -two standalone flags. - -## `query` never benefits from sparse row-major access (found 2026-08-19, not implemented) - -Benchmarked `obikmer query` against `global_index_presence` (dense-packed) -vs. `global_index_presence_sparse` (`pack --sparse`), 100k simulated reads -× 2 specimens (`benchmark/`, see -[benchmark_query_testing.md](../implementation/benchmark_query_testing.md)). -Correctness: 0 mismatches — sparse and dense return bit-identical query -results. Performance: sparse consistently *slower* than dense (~30-50%, -reproducible across two runs with warm disk cache), the opposite of -`pack --sparse`'s stated intent ("faster for single-row access... like -query"). - -**Root cause, read from source, not measured in isolation:** -`KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs:155-220`) -is architecturally column-major: stage 2 walks `for g in 0..n_cols { for -slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly) -as the right locality strategy for the packed/columnar formats, where -`col_value` → `PersistentBitMatrix::get` is a genuine O(1) mmap'd column -read (`persistent.rs:110-113`). - -For `Self::Sparse`, that same `get(c, slot)` (`persistent.rs:114-118`) -allocates a full `n_cols`-wide buffer and calls `fill_row` — materializing -the *entire row* — just to return one cell. Called from inside the -column-major double loop, this reconstructs the same row once per genome -column touched: O(hits × n_cols) full-row rebuilds instead of O(hits). -`PersistentSparseBitMatrix`'s own native row-major decode -(`for_each_genome_in_row`, `sparse.rs:164-177`, used correctly by its own -`row`/`fill_row`/`fill_sub_matrix`) is never reached from the query path -at all. - -**`fill_sub_matrix` (the existing `BinaryMatrix` trait primitive, -`traits.rs:13-37`) is not the right replacement for `query` either**, even -once its own dispatch bug is fixed (see next section) — its output shape -is inherently column-dense: `out[col]` gets an entry for every column, -including columns with zero hits among the requested slots. On real -sparse data (a hit typically touching a handful of genomes out of dozens) -that's still O(n_cols) output regardless of true sparsity. What `query` -actually wants is the sparse triple stream `(slot, col, value)` it already -consumes as `QueryHit::Value` — not a materialized sub-matrix. - -**Proposed primitive** (design only, not implemented — explicit ask: keep -count matrices *not excluded*, even though effort right now is -presence/absence only): - -Not a closure-driven `for_each` — a real `Iterator`, one concrete struct -per matrix format, so the traversal state (current position in the sorted -slot list, current column, permutation, sparse-row decode cursor…) lives -in named struct fields instead of being threaded implicitly through -recursion or a captured closure. RPITIT (stable since 1.75, and this -workspace is edition 2024) means the trait method can return it without -naming or boxing the concrete type: - -```rust -/// Yields every nonzero cell among `slots`, in implementation-defined order. -fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator + 'a; -// item: (idx into `slots`, col, value) -``` - -This is the one foundational primitive per format — both `fill_sub_matrix` -and a `for_each`-style callback become trivial, free consumers of it -(`.for_each(f)` is already `Iterator::for_each` from std; `fill_sub_matrix` -becomes "drain the iterator, scatter into `out[][]`"), instead of two -independently-maintained traversals that can silently diverge (see the bug -below — this is exactly how it happened). - -- **On `PersistentSparseBitMatrix`**: the struct is nearly free to write — - it wraps the existing (currently private) `for_each_genome_in_row` - per-row decode, advancing to the next `slots` entry on exhaustion. O(Σ - row nnz), zero `n_cols`-wide allocation. -- **On `PersistentBitMatrix::{Packed,Columnar}`**: revised — cheaper than - first thought, by reusing the same split already used for - `fill_matrix`'s own implementation instead of hand-writing a resumable - state machine at the matrix level. The base-vector layer - (`BitSliceView`, `views.rs`) already separates the two concerns: - `fill_batch_sorted` (`views.rs:55-60`, sorted-slot batch lookup) and a - genuine per-bit `Iterator` (`BitSliceIter`, `views.rs:94+`) sit side by - side there, one level below the matrix. Adding a - "positions among `sorted_slots` where the bit is set" iterator at that - same vector level is a `filter` over the existing `get()` — no new - state machine, since `std::iter::Filter` already *is* one, generated by - the compiler: - ```rust - fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator + 's { - sorted_slots.iter().copied().filter(move |&slot| self.get(slot)) - } - ``` - The matrix-level `nonzero_iter` then composes these per column with - `flat_map` over `0..n_cols` (each column's hits, tagged with `c`, - slot mapped back through the sort permutation `fill_batch`/ - `fill_batch_sorted` already carry) — again a combinator chain, not a - hand-rolled struct. Same algorithm, same mmap/sort locality as today's - `fill_sub_matrix`; just assembled from `std` iterator adaptors instead - of a loop body writing into a buffer, mirroring the vector/matrix split - the codebase already uses for `fill_batch_sorted` rather than - introducing a new shape. -- **`Implicit`**: trivial (`slots.iter().map(|&i| (i, 0, 1))`, one column, - always present). -- **On `PersistentCompactIntMatrix` (counts)**: same treatment as - `Packed`/`Columnar` — no sparse count format exists yet ("Explicitly - deferred" per `traits.rs:9-12`), so no native low-effort case the way - `Sparse` has one, but not excluded either: the iterator's `Item` is - already `(usize, usize, u32)`, not `bool`, specifically so presence - (`0`/`1`) and counts (arbitrary `u32`) share one primitive instead of a - bool/u32 split forcing counts out of the design. Ready for a native - sparse-count struct later without a signature change. - -Would let `query_partition_with`'s stage 2 collapse to one -`for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) }` -per layer, format-agnostic, each backend's struct deciding the actual -traversal. - -**This also closes the existing dispatch bug for free, by construction**: -`PersistentBitMatrix::fill_sub_matrix` (`persistent.rs:190-215`, the enum -wrapper backing `BinaryMatrix`'s trait impl) today does *not* delegate to -`PersistentSparseBitMatrix::fill_sub_matrix` for `Self::Sparse` — it -reimplements the same naive per-(column, slot) `fill_row_bool` loop -instead, bypassing the efficient native method one file over -(`sparse.rs:249-258`). `obikphylo::siblings::cache::Mat` -(`cache.rs:138-145`) independently built its own parallel enum wrapper -that dispatches correctly — evidence this was worked around rather than -fixed at the source: two hand-written traversals for the same format, -free to drift apart, and they did. If `fill_sub_matrix` itself is -rewritten as "drain `nonzero_iter`, scatter into `out[][]`", there is only -one traversal per format left to get right — the bug class doesn't just -get fixed once, it stops being possible to reintroduce. - -## Implemented (2026-08-20) - -Built as designed above, with one deviation from the original sketch: -`nonzero_iter` ended up `Box>`, not a bare `impl -Iterator`, because `Columnar`/`Packed`/`Sparse`/`Implicit` are genuinely -different concrete types and this method isn't on a trait (kept off -`BinaryMatrix` deliberately — that trait is used as `dyn BinaryMatrix` in -`tests/sparse.rs`, and RPITIT methods aren't dyn-compatible). One `Box` -per `nonzero_iter` call, not per cell — negligible next to what it -replaces. - -- `BitSliceView::nonzero_among_sorted` / `IntSliceView::nonzero_among_sorted` - (`obicompactvec/src/views.rs`): the vector-level `filter`/`filter_map` - primitive, exactly as sketched — no new state machine, `std`'s own. -- `PersistentSparseBitMatrix::nonzero_iter` (`bitmatrix/sparse.rs`): native, - `std::iter::from_fn` over one buffered row at a time via the existing - `for_each_genome_in_row` — no `n_cols`-wide allocation, ever. -- `PersistentBitMatrix::nonzero_iter` (`bitmatrix/persistent.rs`): dispatches - to the above for `Sparse`; for `Columnar`/`Packed`, loops columns, - collects each column's `nonzero_among_sorted` hits via `.extend()` (not - `flat_map` — a `flat_map` closure can't lazily return something - borrowing its own captured sort permutation across separate calls - without either boxing per-column or fighting the borrow checker; eager - collection into one `Vec` sidesteps it, at zero cost since - `fill_sub_matrix` already fully materialized anyway). `Implicit` trivial. -- `PersistentBitMatrix::fill_sub_matrix` and `sub_matrix` rewritten to - drain `nonzero_iter` — the dispatch bug is gone because there is now - only one traversal per format, not because the old one was patched. - `PersistentCompactIntMatrix::nonzero_iter` added the same way (counts - not excluded, per the earlier ask) — no native low-effort case, since no - sparse count format exists, but on the same primitive, ready for one. -- `KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs`): - stage 2's column-major `for g { for slot { col_value } }` replaced by one - `layer.nonzero_iter(&slot_list)` call per layer, format-agnostic. -- Tests: `nonzero_iter_matches_dense`, `nonzero_iter_matches_row`, and — - the one that actually targets the dispatch bug rather than each type's - own correctness — `enum_wrapper_dispatches_to_native_sparse` (builds - `PersistentBitMatrix::Sparse(...)` directly, not through `open`, since - `open` only auto-detects `Sparse` from a `presence/` dir layout). - `cargo test --workspace`: green, no regressions. - -**Measured**: re-ran the `benchmark/` query branch (100k reads × 2 -specimens, same setup as the original finding). Correctness still 0 -mismatches. The dense/sparse performance gap is gone — previously sparse -~30-50% slower than dense, reproducibly; now within ~1-3% either way -(7.42s dense vs 7.60s sparse for `Escherichia_coli--K-12_MG1655`; 5.25s vs -5.30s for `Saccharolobus_islandicus--M.16.4`) — noise-level, not a -systematic gap. `pack --sparse`'s claimed query win isn't confirmed -outright by this (sparse should arguably now *beat* dense on truly sparse -real data, not just tie), but the pathological regression is fixed. - -## `PersistentCompactIntMatrix::Sparse` — implemented (2026-08-26) - -Closes the gap flagged throughout this document ("no sparse count format -exists yet", `traits.rs:9-12`'s "Explicitly deferred"): `obicompactvec` -already had `PersistentSparseCompactIntMatrix` (row-major, built on top of -`PersistentSparseBitMatrix` as its "which columns are non-zero" support, -values *not* deduplicated — see that struct's own doc comment), but it was -never wired into `PersistentCompactIntMatrix`, the dense-dispatching enum -every real consumer (`TypedLayer`, -`KmerLayer::Count`) actually holds. Concretely: `kmer_index.rs:: -pack_matrices(sparse=true)` already called `pack_sparse_compact_int_matrix` -on every layer's `counts/` — but `PersistentCompactIntMatrix::open` had no -code path back to what that just wrote, so a `Count` layer became -unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment -anyone ran `pack --sparse` on an index with count layers. Root cause, not a -workaround: add the missing `Sparse` variant. - -- **Enum + dispatch** (`intmatrix.rs`): `PersistentCompactIntMatrix::Sparse - (PersistentSparseCompactIntMatrix)`, detected in `open`/`detect_storage` - via a `singleton_values.pciv` marker (mirrors `PersistentBitMatrix`'s own - `sparse_meta.json` check), reported via `storage_kind()`. `col`/ - `col_view`/`col_persist` panic/`Unsupported` on `Sparse`, same convention - as the bit side. `sub_matrix`/`fill_sub_matrix` and `nonzero_iter` - unified the same way `PersistentBitMatrix`'s already are (drain - `nonzero_iter`, one traversal per format — see "Implemented - (2026-08-20)" above); `nonzero_iter` had to become `Box>` - for the same reason (`Columnar`/`Packed`/`Sparse` are different concrete - types). No change needed in `obikindex` at all — `KmerLayer::Count` - already only ever holds `TypedLayer`, so the - enum absorbing `Sparse` fixes the unreadable-layer bug for free, same as - `PersistentBitMatrix::Sparse` already did on the presence side. - -- **`CountPartials`, non-naive** (`sparse_intmatrix.rs`): unlike - `PersistentSparseBitMatrix`'s dict-driven `col_weights_and_pair_counts`, - values here aren't deduplicated (two rows can share the same non-zero - column set via the same `dict_id` while carrying different counts), so - the "weight by how many rows share a dict entry" shortcut doesn't carry - over. What does: a single row-major pass (`row_major_pairwise`, decodes - each row once via `for_each_cell_in_row`, nests over that row's own - co-present columns) — `O(Σ k̄²)` over populated rows instead of the naive - `O(n_cols² × n)` column-pair rescan, same complexity class as the bit - side minus the dict multiplicity discount. Kernels used: `min(a,b)` - (bray, relfreq-bray — both vanish when either side is absent, so no - correction needed), `a·b` and `√(a·b)` (euclidean/relfreq-euclidean and - hellinger — these *do* need a correction, reconstructed from per-column - marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`, since `(a-0)² = a² ≠ 0` unlike - the `min`-based formulas). `threshold_jaccard(1)` shortcuts straight to - `support`'s own `BitPartials::partial_jaccard` (threshold 1 is exactly - presence); `threshold_jaccard(0)` is closed-form (every `u32` is `≥ 0`). - -- **Two pre-existing bugs found and fixed while wiring the `threshold==1` - shortcut** (`bitmatrix/sparse.rs`, `BitPartials for - PersistentSparseBitMatrix`, present since the 2026-08-15 implementation - above, never caught because no test compared `Sparse`'s raw `partial_*` - output against dense on real data — only the diagonal-blind - `jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested): - 1. `partial_jaccard`'s diagonal was `(0, 2×col_weights[i])` instead of a - genuine self-comparison `(col_weights[i], col_weights[i])` — - `col_weights_and_pair_counts`'s `inter` never pairs a column with - itself by construction. - 2. `partial_hamming`'s off-diagonal formula itself was wrong: `total - - union` (count of rows where *neither* column is present) instead of - the actual Hamming distance `col_weights[i] + col_weights[j] - - 2×inter[i,j]` (symmetric-difference size). Only coincides with the - correct value when `col_weights[i] + col_weights[j] == total`, so - small/synthetic test data could easily have hidden it. - - Neither surfaced through `jaccard_dist_matrix`/`hamming_dist_matrix` - (both explicitly zero their own diagonal at finalisation, and the - off-diagonal `partial_hamming` bug had gone untested against dense - entirely) — only visible to a caller of the raw `partial_*` methods - directly, which is exactly what `partial_threshold_jaccard(1)`'s new - shortcut became. Fixed at the source, not patched around at the call - site; regression test added: - `tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal`. - -- **Tests**: `tests::intmatrix::sparse_roundtrip_matches_columnar`/ - `sparse_roundtrip_from_packed` (the `open`-dispatch fix, both build - paths); `tests::intmatrix::sparse_count_partials_match_dense` (all six - `CountPartials` formulas, thresholds 0/1/2/3, against `Columnar` on - asymmetric-presence data — this is what caught the diagonal gap in the - int side's own new code before it shipped, the same way it exposed the - two pre-existing bit-side bugs above); `obikindex`'s - `count_layer_transparently_reads_sparse_after_pack` — the actual - end-to-end regression test for the original "layer unreadable after - `pack --sparse`" bug, built → packed sparse → reopened, compared against - the pre-pack dense read. `cargo test -p obicompactvec -p obikindex`: - green, no regressions (180 + 12 tests). +# Sibling annex & SNP-family pipeline — architecture + +## Data model + +A **family** is the up-to-4 k-mers sharing all flanks and differing only at +the central base (well-defined for odd `k`). `FamilyMask` (`siblingannex.rs`) +is a 4-bit presence mask (bit `b` set iff the family member whose own +canonical central base is `b` is observed anywhere in the index) plus a +minorant flag; `family_size()`/`siblings()` derive member count from it. +`SiblingAnnex`/`SiblingAnnexBuilder` persist one `FamilyMask` per k-mer, +**keyed by iteration order** (the physical layout of `unitigs.bin`), never by +MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream +against the annex file directly, no MPHF/slot indirection. + +## Construction + +`algorithms::annex::build_layer_sibling_annex` builds one layer's annex: +seeds each k-mer's own central base, generates its up-to-3 central-base +variants, resolves each cross-partition (`IndexCache::find_in_partition`, +grouped and chunked per destination partition for mmap locality), writes the +final mask (with the minorant flag) in a second streamed pass. +`SiblingExt::build_sibling_annex` drives this for every layer of every +partition, then invalidates and rebuilds the entropy and family-overlap +annexes (their `family_idx` numbering depends on the sibling annex). + +## Shared traversal + +`algorithms::family_scan::scan_layer_families` is the **only** place that +resolves a family's full per-genome presence (`genome_mask: &[u8]`, +cross-partition) — every other consumer in this module is built on it, never +reimplements the resolution. Two decoupled concurrency mechanisms: batch +generation (CPU-only) runs on an `obipipeline` throttle/pipe; per-batch +resolution (I/O-bound, mmap page faults) runs `rayon`-parallel across +destination partitions, one generated batch at a time (never several +concurrently, to keep one partition's pages hot instead of scattering +access). `fast_mode` (`is_fast_mode`, `<= 7` layers per partition) picks +between trusting a family member's recorded layer index outright or probing +every layer of the destination partition. + +`Selection<'a>` bounds which families of a layer are resolved: + +- `All` — every minorant, monomorphic included (`SiblingExt::shannon_entropy_csv`, + `sibling_annex_stats`). +- `Some(&HashSet)` — a caller-picked subset, skipping the expensive + cross-partition resolution entirely for excluded families + (`subsample::sample_layer`'s accepted-candidates batch; + `entropy::ensure_layer_entropy_annex`'s non-monomorphic-only restriction). + +`minorant_selection::eligible_bitset_layer`/`non_monomorphic_reindex_layer` +provide the cheap, annex-bits-only (no cross-partition resolution) filters +these restricted scans are built from: monomorphism (`family_size() < 2`) is +knowable straight from the annex, so nothing pays `scan_layer_families`'s +resolution cost just to discard a monomorphic result. + +## Masking — computed once, shared everywhere + +`algorithms::masking::masked_state(m, free_loss, no_ambiguity)` is the single +function that turns a raw per-genome 4-bit state into the masked view every +consumer needs (`None` for absence under `--free-loss` or ambiguity under +`--no-ambiguity`). It is called **exactly once per (family, genome)**, in +`subsample::sample_layer`, immediately after `scan_layer_families` resolves +the raw state — never recomputed downstream. `survives_masking` (does the +family still carry ≥2 distinct masked states among included genomes?) and +`SurvivingFamily::masked: Vec>` both consume this same +precomputed array, which is also what `AlignmentReducer`/`PairwiseReducer` +read (see below) — this is what guarantees the alignment and the pairwise +tally can never disagree about a genome's state at a given site. + +## Precomputed annexes (entropy, family overlap) + +`algorithms::entropy::ensure_layer_entropy_annex` builds `EntropyAnnex` +(`entropy_annex.rs`, one `f32` Shannon-entropy value per non-monomorphic +minorant, mmap-backed) and/or feeds `FamilyOverlapAccumulator` +(`family_overlap.rs`) from one shared `Selection::Some` scan — both need the +same `genome_mask` resolution, so a missing one never pays for it twice. +`family_entropy` (15 non-empty states of `{A,C,G,T}`'s powerset, `∅` +excluded from the denominator — the project's settled informativeness +definition, matching the calibrated 16-state Sankoff cost matrix) and +`family_entropy_4` (plain 4-symbol reduction, kept only for comparison) are +pure functions of `genome_mask`. `extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes` +drives this for every cached layer, lazily (only what's missing). + +## Sampling and the shared scan trunk + +`algorithms::subsample::sample_index` draws up to `n` non-monomorphic +families index-wide, proportionally per layer: for each layer, a target +quota sized from that layer's share of the index-wide eligible count, filled +by a circular walk with a Bernoulli(`p`) draw per candidate — `p` optionally +weighted by an unnormalized Gaussian kernel over the candidate's entropy +(`EntropyBias { mu, sigma }`, read from the entropy annex). Accepted +candidates are resolved in batches (`scan_layer_families`) and checked +against `survives_masking`; rejected-after-masking candidates don't count +toward quota, triggering further "topup" rounds (bounded, `MAX_TOPUP_ROUNDS += 5`) rather than silently under-filling. Every `SurvivingFamily` reaching a +caller has already survived both the draw and post-masking informativeness — +no second filter pass needed downstream. + +`algorithms::scan::sampled_scan` is the single entry point every sampled +computation goes through: one `sample_index` pass, fanned out to whichever +`LayerReducer` implementations the caller actually needs, each receiving the +exact same per-layer `SurvivingFamily` batch (an `Arc`, cloned not copied). +This is what guarantees several outputs requested together +(`--pseudo-alignment` and `--sankoff` and a `snp-*` `--distance` in one CLI +invocation) come from **one** random sample, not independently-drawn ones — +there is no other driver of `sample_index` anywhere in this crate. + +Two `LayerReducer` implementations exist today: + +- `AlignmentReducer` (`alignment.rs`) — accumulates `SnpAlignment` (one + IUPAC-coded row per included genome), reading `family.masked` directly. +- `PairwiseReducer` (`pairwise.rs`) — accumulates `PairwiseTally`/ + `PartitionDispersion`, also reading `family.masked` directly (substitution/ + agreement counts and cardinality both derive from the same masked state + the alignment uses). + +## From a tally to distances and calibration + +`PairwiseTally` stores per-genome-pair substitution/agreement/cardinality +counts, unfiltered, from which every downstream output is cheap `O(n²)` +post-processing (never a second scan): + +- `algorithms::snp_distance::distance_matrix` — closed-form `snp-*` + corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions- + only), optionally gamma-corrected (`GammaShape`, `PartitionDispersion::estimate_alpha` + for `Auto`). +- `algorithms::sankoff::SankoffBundle::finish` — assembles the + `--sankoff`/`--tnt`/`--phyg`/`--iqtree` pipeline's inputs + (`RawSnpDistanceOutput`, `BasePairTally`, `CardinalityTally`) from an + already-scanned `AlignmentReducer`/`PairwiseReducer` pair, plus + `SankoffBundle::snp_distance` to reuse that exact sample for a `snp-*` + `--distance` in the same invocation without a second scan. +- `algorithms::cardcomp` — pure functions turning `CardinalityTally`/ + `BasePairTally` into row-stochastic transition probability matrices, then + the calibrated 16-state Sankoff cost matrix (`pairwise_cost_matrix`). + +## Public API surface + +`extensions::SiblingExt` (CLI-facing trait on `IndexCache`) covers only the +operations with no cross-invocation sharing concern: `build_sibling_annex`, +`shannon_entropy_csv`, `sibling_family_size_histogram`/`sibling_annex_stats`, +`family_overlap`. The sampled SNP path (`sampled_scan`, `AlignmentReducer`, +`PairwiseReducer`, `distance_matrix`, `SankoffBundle`) is **not** wrapped in +trait methods — the CLI (`obikmer::cmd::phylo`) composes them directly, +because it alone knows the union of reducers a given invocation needs +(`--pseudo-alignment`/Sankoff group/`snp-*` `--distance` can combine in one +call). A per-output trait method would have to either duplicate that +composition logic or reintroduce the "two independent scans" problem this +design replaced. + +## Known gaps + +- `sample_index`'s `skip_layers` parameter (checkpointed-resume support) is + never called with a non-zero value by any current caller — `--session` + persistence was removed pending a redesign on top of `sampled_scan` + (single shared trunk, not per-driver checkpoint plumbing). The parameter + stays as the seam a future resume mechanism would use. +- No cheap entropy pre-filter exists yet for the very high (saturated) tail — + a row-marginal coverage approximation was considered and rejected (cannot + represent joint multi-base co-occurrence, which is exactly what the + 15-state entropy definition exists to capture). diff --git a/DevDocMD/implementation/benchmark_query_testing.md b/DevDocMD/implementation/benchmark_query_testing.md index e234e984..cb9c358c 100644 --- a/DevDocMD/implementation/benchmark_query_testing.md +++ b/DevDocMD/implementation/benchmark_query_testing.md @@ -12,8 +12,9 @@ this pipeline. by `verify_presence`/`verify_merge_presence`, which only check index *content* against the `.npz` truth, never the query API. - `pack --sparse` produces a presence-matrix format documented (see - [siblings.md](../architecture/siblings.md)) as faster for single-row - access (query) and slower for column-oriented access (phylo `--metric`). + [obicompactvec.md](obicompactvec.md), "Sparse matrix formats") as faster + for single-row access (query) and slower for column-oriented access + (phylo `--metric`). `global_index_presence/` built by `merge_presence.sh` is always packed dense (packing is a stage inside `merge`, not a separate `pack` invocation) — there was no dense/sparse regression check. diff --git a/DevDocMD/implementation/layer_tier.md b/DevDocMD/implementation/layer_tier.md index 57499b99..e61c34a7 100644 --- a/DevDocMD/implementation/layer_tier.md +++ b/DevDocMD/implementation/layer_tier.md @@ -299,7 +299,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec> pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec]) ``` -Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md). +Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this goes through `nonzero_iter` instead, decoding one row at a time; see [obicompactvec.md](obicompactvec.md), "Sparse matrix formats". --- diff --git a/DevDocMD/implementation/obicompactvec.md b/DevDocMD/implementation/obicompactvec.md index b999793d..fd29112d 100644 --- a/DevDocMD/implementation/obicompactvec.md +++ b/DevDocMD/implementation/obicompactvec.md @@ -298,13 +298,11 @@ Both matrix types are enums behind a transparent API — the caller never matche |---|---|---| | `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) | | `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` | -| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off | +| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off | | `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` | `PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`. -Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation. - `col_view(c)` returns the appropriate view directly: ```rust @@ -321,6 +319,163 @@ No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view --- +## Sparse matrix formats + +Motivated by real k-mer-family data being extremely row-sparse (most rows — +k-mers — present in only a handful of genome columns out of dozens to +thousands): a row-major, deduplicated alternative to the dense column-major +formats above, built once per layer like the rest of the pipeline (never the +whole multi-billion-row index at once). Implements the exact-duplicate +special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data +Structures for Collections of Sets" (2025, +`biblio/Alanko et al. - Compact Data Structures for Collections of +Sets.pdf`) — a plain dedup dictionary, not their full subset-containment +hierarchy. + +### `PersistentSparseBitMatrix` + +Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row +(singleton vs. multi-genome), a fixed-bit-width array for singleton rows +(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width +array for multi-genome rows (`dict_id`, `ceil(log2(n_distinct_multi_sets))` +bits — kept apart from the singleton array because `n_distinct_multi_sets` +can be large in absolute terms even when multi-genome rows are a small +*fraction* of all rows, and a shared array would force every row, singletons +included, to pay the wider width), and a deduplicated dictionary of distinct +multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values +blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary, +runtime-parameterised bit width, width `0` included — needed once an +`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec` +(rank1/rank0/select1 on top of the crate's existing `count_ones`, via +`common_traits::SelectInWord`), `EliasFano` (composes the two). `BinaryMatrix` +(`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, `count_ones`) is the +trait unifying dense and sparse at the call sites that need both +interchangeably; column-oriented methods (`col`, `col_view`, the +`partial_*_dist_matrix` family) stay dense-only — `Sparse`/`Implicit` panic +there (see the `PersistentBitMatrix::open` note above). + +Measured on real data (one layer, 30.2M rows, 91 genomes): + +| | dense | sparse | ratio | +|---|---|---|---| +| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller | +| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction | +| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) | +| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** — the `--shannon`/family-lookup access shape | +| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit | + +The row-access wins (compactness was the stated design goal, not speed) come +from dense's genome-major layout scattering a single row read across a much +bigger file — costlier than sparse's rank/select/varint decode once the file +is this much smaller. Column access is the flip side of the same layout +choice; `nonzero_iter` (below) is what removes it from the hot path that +actually matters (`query`). + +Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open` +re-derived its low-bits width from the persisted low-vector's own width +byte; the width-`0` case was built with a dummy 1-bit placeholder (the +builder used to reject true width `0`), silently doubling every reopened +value. Fixed by making `PersistentFixedIntVec` genuinely support width `0` +(no storage, `get` always `0`). (2) An empty row (cardinality 0) used to be +recorded as a singleton at genome `0`, indistinguishable on read-back from a +*real* singleton at genome `0`. Fixed by routing cardinality-0 rows through +the dictionary path (a genuine empty entry). + +Still deferred: full Alanko et al. subset-hierarchy compression (only the +exact-duplicate case is built), a sparse `PersistentCompactIntMatrix` +variant beyond what's described below, BRWT-style column-correlation +exploitation, and a row-major rewrite of +`partial_jaccard_dist_matrix`/`partial_hamming_dist_matrix`/etc. as a +per-row co-occurrence accumulation (`O(Σ_rows k̄²)`) so `obikindex`'s +`--metric` distance-matrix path stops paying the column-access penalty on +`Sparse` — needs its own design pass (how it plugs into +`BitPartials`/`ColumnWeights` so both matrix types keep serving `--metric`), +not just "port the loop". + +### `nonzero_iter` — the one traversal per format + +`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box + 'a>` +(item: index into `slots`, column, value) is the single foundational +per-format traversal primitive every consumer builds on — `fill_sub_matrix` +and a `for_each`-style callback are both trivial, free consumers of it +(`.for_each(f)` is `Iterator::for_each`; `fill_sub_matrix` drains the +iterator and scatters into `out[][]`), instead of two independently +hand-maintained traversals that can silently diverge. `Box`, +not a bare `impl Iterator`: `Columnar`/`Packed`/`Sparse`/`Implicit` are +genuinely different concrete types and the method isn't on a +dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix` +elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per +`nonzero_iter` call, not per cell. + +Per-format implementation: `PersistentSparseBitMatrix` decodes natively, +buffered one row at a time (`std::iter::from_fn` over the existing +`for_each_genome_in_row`, no `n_cols`-wide allocation ever). `Columnar`/ +`Packed` loop columns, collecting each column's hits via +`BitSliceView::nonzero_among_sorted`/`IntSliceView::nonzero_among_sorted` +(a `filter`/`filter_map` over the vector-level `get()`, no new state +machine — `std`'s own `Filter` iterator already is one) into one `Vec` via +`.extend()` (not `flat_map`: a `flat_map` closure can't lazily return +something borrowing its own captured sort permutation across separate calls +without either boxing per-column or fighting the borrow checker — eager +collection sidesteps it at zero extra cost, since `fill_sub_matrix` already +fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))`). + +This closed a real dispatch bug by construction, not by patching it: +`PersistentBitMatrix::fill_sub_matrix`/`sub_matrix` used to reimplement a +naive per-`(column, slot)` loop instead of delegating to +`PersistentSparseBitMatrix`'s own efficient native method for `Self::Sparse` +— two hand-written traversals for the same format, free to drift apart, and +they had (`obikphylo`'s own cache layer had independently built a parallel +enum wrapper that *did* dispatch correctly, evidence the bug was worked +around rather than fixed at the source). With `fill_sub_matrix`/`sub_matrix` +rewritten to drain `nonzero_iter`, there is only one traversal per format +left to get right. `KmerPartition::query_partition_with` +(`obikindex/src/partition/query_layer.rs`) — previously column-major +(`for g { for slot { col_value } }`, which for `Sparse` rebuilt a full row +per single cell touched, O(hits × n_cols) instead of O(hits) — the original +motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)` +call per layer, format-agnostic. Dense/sparse query performance parity +confirmed on real data (previously sparse 30-50% slower than dense, +reproducibly; now within ~1-3% either way — noise level). + +### `PersistentSparseCompactIntMatrix` + +The count-matrix analogue, built on `PersistentSparseBitMatrix` as its +"which columns are non-zero" support, values **not** deduplicated (unlike +the bit side, two rows can share the same non-zero column set — same +`dict_id` — while carrying different counts, so the dict-multiplicity +shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry +over). `CountPartials` here: a single row-major pass (`row_major_pairwise` +in `sparse_intmatrix.rs`, decodes each row once via +`for_each_cell_in_row`, nests over that row's own co-present columns) — +`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)` +column-pair rescan. `min(a,b)`-based kernels (`bray`, `relfreq_bray`) need +no correction — they vanish when either side is absent. `a·b` and `√(a·b)` +kernels (`euclidean`/`relfreq_euclidean`/`hellinger`) do need one, since +`(a-0)² = a² ≠ 0` unlike the `min`-based formulas — reconstructed from +per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`. +`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard` +(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form +(every `u32` is `≥ 0`). + +Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in +`BitPartials for PersistentSparseBitMatrix` (present since that type's own +introduction, never caught because no test had compared `Sparse`'s raw +`partial_*` output against dense on real data — only the diagonal-blind +`jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested), +fixed at the source: + +1. `partial_jaccard`'s diagonal used to be `(0, 2×col_weights[i])` instead + of a genuine self-comparison `(col_weights[i], col_weights[i])` — + `col_weights_and_pair_counts`'s `inter` never pairs a column with itself + by construction, so the diagonal needs its own case. +2. `partial_hamming`'s off-diagonal formula itself was wrong: it computed + `total - union` (rows where *neither* column is present) instead of the + actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]` + (symmetric-difference size) — the two only coincide when + `col_weights[i] + col_weights[j] == total`, which small/synthetic test + data can easily satisfy by accident. + ## Aggregation traits (matrix level) ### ColumnWeights diff --git a/mkdocs.yml b/mkdocs.yml index f1aa9f13..6f493d68 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -60,7 +60,7 @@ nav: - Architecture: - Sequences: architecture/sequences/invariant.md - Kmer index: architecture/index_architecture.md - - Sibling annex (discussion): architecture/siblings.md + - Sibling annex: architecture/siblings.md - NUMA-aware worker pools: architecture/numa_worker_pools.md - NUMA-aware partition runner: architecture/numa_partition_runner.md