From 95bbc0b2ae4764001cb4017e0870a7f6d14193e4 Mon Sep 17 00:00:00 2001 From: Eric Coissac Date: Fri, 18 Sep 2026 19:17:59 +0200 Subject: [PATCH] Implement sparse matrix optimizations and update architecture documentation This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion. --- DevDoc/404.html | 2 +- .../index_architecture.refs/index.html | 2 +- .../index_architecture/index.html | 2 +- .../numa_partition_runner/index.html | 2 +- .../architecture/numa_worker_pools/index.html | 2 +- DevDoc/architecture/query.refs/index.html | 2 +- DevDoc/architecture/query/index.html | 2 +- DevDoc/architecture/rebuild_filter/index.html | 2 +- .../sequences/invariant.refs/index.html | 2 +- .../sequences/invariant/index.html | 2 +- DevDoc/architecture/siblings/index.html | 1413 +++-------------- .../benchmark_query_testing/index.html | 7 +- .../chunkreader.refs/index.html | 2 +- DevDoc/implementation/chunkreader/index.html | 2 +- .../evidence_elimination.refs/index.html | 2 +- .../evidence_elimination/index.html | 2 +- DevDoc/implementation/filtering/index.html | 2 +- DevDoc/implementation/kmer.refs/index.html | 2 +- DevDoc/implementation/kmer/index.html | 2 +- .../implementation/layer_tier.refs/index.html | 2 +- DevDoc/implementation/layer_tier/index.html | 4 +- DevDoc/implementation/merge.refs/index.html | 2 +- DevDoc/implementation/merge/index.html | 2 +- .../merge_parallelism/index.html | 2 +- DevDoc/implementation/mphf.refs/index.html | 2 +- DevDoc/implementation/mphf/index.html | 2 +- .../implementation/obicompactvec/index.html | 222 ++- .../obipipeline.refs/index.html | 2 +- DevDoc/implementation/obipipeline/index.html | 2 +- DevDoc/implementation/obitaxonomy/index.html | 2 +- .../partition_layer_cache/index.html | 2 +- .../persistent_bit_vec.refs/index.html | 2 +- .../persistent_bit_vec/index.html | 2 +- .../index.html | 2 +- .../persistent_compact_int_vec/index.html | 2 +- .../implementation/pipeline.refs/index.html | 2 +- DevDoc/implementation/pipeline/index.html | 2 +- DevDoc/implementation/select/index.html | 2 +- DevDoc/implementation/storage.refs/index.html | 2 +- DevDoc/implementation/storage/index.html | 2 +- .../implementation/superkmer.refs/index.html | 2 +- DevDoc/implementation/superkmer/index.html | 2 +- .../unitig_evidence.refs/index.html | 2 +- .../implementation/unitig_evidence/index.html | 2 +- DevDoc/index.html | 2 +- DevDoc/index.refs/index.html | 2 +- DevDoc/installation/index.html | 2 +- DevDoc/kmers.refs/index.html | 2 +- DevDoc/kmers/index.html | 2 +- DevDoc/sitemap.xml.gz | Bin 127 -> 127 bytes DevDoc/theory/encoding.refs/index.html | 2 +- DevDoc/theory/encoding/index.html | 2 +- DevDoc/theory/entropy.refs/index.html | 2 +- DevDoc/theory/entropy/index.html | 2 +- .../theory/evolutionary_distances/index.html | 598 ++++++- DevDoc/theory/indexing.refs/index.html | 2 +- DevDoc/theory/indexing/index.html | 2 +- DevDoc/theory/minimizer.refs/index.html | 2 +- DevDoc/theory/minimizer/index.html | 2 +- DevDocMD/architecture/siblings.md | 1184 ++------------ .../implementation/benchmark_query_testing.md | 5 +- DevDocMD/implementation/layer_tier.md | 2 +- DevDocMD/implementation/obicompactvec.md | 161 +- mkdocs.yml | 2 +- 64 files changed, 1374 insertions(+), 2330 deletions(-) diff --git a/DevDoc/404.html b/DevDoc/404.html index b04c63bb..5a9ae5cb 100644 --- a/DevDoc/404.html +++ b/DevDoc/404.html @@ -1119,7 +1119,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/index_architecture.refs/index.html b/DevDoc/architecture/index_architecture.refs/index.html index 78ae7534..f19ad23b 100644 --- a/DevDoc/architecture/index_architecture.refs/index.html +++ b/DevDoc/architecture/index_architecture.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/index_architecture/index.html b/DevDoc/architecture/index_architecture/index.html index 30ea415b..09497c21 100644 --- a/DevDoc/architecture/index_architecture/index.html +++ b/DevDoc/architecture/index_architecture/index.html @@ -1370,7 +1370,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/numa_partition_runner/index.html b/DevDoc/architecture/numa_partition_runner/index.html index c79dca08..653ab74a 100644 --- a/DevDoc/architecture/numa_partition_runner/index.html +++ b/DevDoc/architecture/numa_partition_runner/index.html @@ -1128,7 +1128,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/numa_worker_pools/index.html b/DevDoc/architecture/numa_worker_pools/index.html index 97421f9a..10674165 100644 --- a/DevDoc/architecture/numa_worker_pools/index.html +++ b/DevDoc/architecture/numa_worker_pools/index.html @@ -1130,7 +1130,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/query.refs/index.html b/DevDoc/architecture/query.refs/index.html index ef859997..141c734a 100644 --- a/DevDoc/architecture/query.refs/index.html +++ b/DevDoc/architecture/query.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/query/index.html b/DevDoc/architecture/query/index.html index 2ed91fb1..a2012d4a 100644 --- a/DevDoc/architecture/query/index.html +++ b/DevDoc/architecture/query/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/rebuild_filter/index.html b/DevDoc/architecture/rebuild_filter/index.html index a21d25de..04a44fdc 100644 --- a/DevDoc/architecture/rebuild_filter/index.html +++ b/DevDoc/architecture/rebuild_filter/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/sequences/invariant.refs/index.html b/DevDoc/architecture/sequences/invariant.refs/index.html index 2d5e0338..66ddfc59 100644 --- a/DevDoc/architecture/sequences/invariant.refs/index.html +++ b/DevDoc/architecture/sequences/invariant.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/sequences/invariant/index.html b/DevDoc/architecture/sequences/invariant/index.html index 38f3fb60..7ce2bce5 100644 --- a/DevDoc/architecture/sequences/invariant/index.html +++ b/DevDoc/architecture/sequences/invariant/index.html @@ -1140,7 +1140,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/architecture/siblings/index.html b/DevDoc/architecture/siblings/index.html index 8be09b91..74189217 100644 --- a/DevDoc/architecture/siblings/index.html +++ b/DevDoc/architecture/siblings/index.html @@ -23,7 +23,7 @@ - Sibling annex (discussion) - obikmer + Sibling annex - obikmer @@ -64,7 +64,7 @@
- + Skip to content @@ -100,7 +100,7 @@
- Sibling annex (discussion) + Sibling annex
@@ -1139,7 +1139,7 @@ - Sibling annex (discussion) + Sibling annex @@ -1157,7 +1157,7 @@ - Sibling annex (discussion) + Sibling annex @@ -1183,10 +1183,10 @@
  • - + - Two index spaces, uncorrelated + Data model @@ -1194,10 +1194,10 @@
  • - + - Two pipelines, never mixed + Construction @@ -1205,10 +1205,10 @@
  • - + - Sibling annex: an iteration-pipeline artifact only + Shared traversal @@ -1216,10 +1216,10 @@
  • - + - Pending work — done + Masking — computed once, shared everywhere @@ -1227,10 +1227,10 @@
  • - + - Performance: build_sibling_annex parallelism (2026-08-14) + Precomputed annexes (entropy, family overlap) @@ -1238,10 +1238,10 @@
  • - + - Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14) + Sampling and the shared scan trunk @@ -1249,10 +1249,10 @@
  • - + - Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15) + From a tally to distances and calibration @@ -1260,10 +1260,10 @@
  • - + - --subsample / --shannon — sampling strategy (decided 2026-08-15) + Public API surface @@ -1271,109 +1271,10 @@
  • - + - Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15) - - - - -
  • - -
  • - - - - --free-loss/--tnt pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below) - - - - -
  • - -
  • - - - - Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15) - - - - -
  • - -
  • - - - - Two entropy definitions kept side by side, for comparison (2026-08-15) - - - - -
  • - -
  • - - - - PersistentSparseBitMatrix — implemented and measured (2026-08-15) - - - - -
  • - -
  • - - - - Wired into pack and the sibling-annex build path (2026-08-15) - - - - -
  • - -
  • - - - - Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15) - - - - -
  • - -
  • - - - - query never benefits from sparse row-major access (found 2026-08-19, not implemented) - - - - -
  • - -
  • - - - - Implemented (2026-08-20) - - - - -
  • - -
  • - - - - PersistentCompactIntMatrix::Sparse — implemented (2026-08-26) + Known gaps @@ -1479,10 +1380,10 @@
    • - + - Two index spaces, uncorrelated + Data model @@ -1490,10 +1391,10 @@
    • - + - Two pipelines, never mixed + Construction @@ -1501,10 +1402,10 @@
    • - + - Sibling annex: an iteration-pipeline artifact only + Shared traversal @@ -1512,10 +1413,10 @@
    • - + - Pending work — done + Masking — computed once, shared everywhere @@ -1523,10 +1424,10 @@
    • - + - Performance: build_sibling_annex parallelism (2026-08-14) + Precomputed annexes (entropy, family overlap) @@ -1534,10 +1435,10 @@
    • - + - Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14) + Sampling and the shared scan trunk @@ -1545,10 +1446,10 @@
    • - + - Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15) + From a tally to distances and calibration @@ -1556,10 +1457,10 @@
    • - + - --subsample / --shannon — sampling strategy (decided 2026-08-15) + Public API surface @@ -1567,109 +1468,10 @@
    • - + - Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15) - - - - -
    • - -
    • - - - - --free-loss/--tnt pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below) - - - - -
    • - -
    • - - - - Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15) - - - - -
    • - -
    • - - - - Two entropy definitions kept side by side, for comparison (2026-08-15) - - - - -
    • - -
    • - - - - PersistentSparseBitMatrix — implemented and measured (2026-08-15) - - - - -
    • - -
    • - - - - Wired into pack and the sibling-annex build path (2026-08-15) - - - - -
    • - -
    • - - - - Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15) - - - - -
    • - -
    • - - - - query never benefits from sparse row-major access (found 2026-08-19, not implemented) - - - - -
    • - -
    • - - - - Implemented (2026-08-20) - - - - -
    • - -
    • - - - - PersistentCompactIntMatrix::Sparse — implemented (2026-08-26) + Known gaps @@ -1693,1008 +1495,151 @@ -

      Sibling annex — architecture (discussion)

      -

      Status: architecture decided (2026-08-14). Implementation not yet mandated.

      -

      Two index spaces, uncorrelated

      -

      Every kmer stored in a Layer lives in two independent index spaces:

      +

      Sibling annex & SNP-family pipeline — architecture

      +

      Current state of obikphylo::siblings (crate obikphylo, module siblings). +Historical rationale, dated findings, and superseded designs live in jj +history, not here.

      +

      Data model

      +

      A family is the up-to-4 k-mers sharing all flanks and differing only at +the central base (well-defined for odd k). FamilyMask (siblingannex.rs) +is a 4-bit presence mask (bit b set iff the family member whose own +canonical central base is b is observed anywhere in the index) plus a +minorant flag; family_size()/siblings() derive member count from it. +SiblingAnnex/SiblingAnnexBuilder persist one FamilyMask per k-mer, +keyed by iteration order (the physical layout of unitigs.bin), never by +MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream +against the annex file directly, no MPHF/slot indirection.

      +

      Construction

      +

      algorithms::annex::build_layer_sibling_annex builds one layer's annex: +seeds each k-mer's own central base, generates its up-to-3 central-base +variants, resolves each cross-partition (IndexCache::find_in_partition, +grouped and chunked per destination partition for mmap locality), writes the +final mask (with the minorant flag) in a second streamed pass. +SiblingExt::build_sibling_annex drives this for every layer of every +partition, then invalidates and rebuilds the entropy and family-overlap +annexes (their family_idx numbering depends on the sibling annex).

      +

      Shared traversal

      +

      algorithms::family_scan::scan_layer_families is the only place that +resolves a family's full per-genome presence (genome_mask: &[u8], +cross-partition) — every other consumer in this module is built on it, never +reimplements the resolution. Two decoupled concurrency mechanisms: batch +generation (CPU-only) runs on an obipipeline throttle/pipe; per-batch +resolution (I/O-bound, mmap page faults) runs rayon-parallel across +destination partitions, one generated batch at a time (never several +concurrently, to keep one partition's pages hot instead of scattering +access). fast_mode (is_fast_mode, <= 7 layers per partition) picks +between trusting a family member's recorded layer index outright or probing +every layer of the destination partition.

      +

      Selection<'a> bounds which families of a layer are resolved:

        -
      • Iteration order: its position when enumerating unitigs.bin (the - superkmer file), deterministic but arbitrary with respect to slot.
      • -
      • MPHF slot: MphfLayer::index(kmer), the number the MPHF assigns.
      • +
      • All — every minorant, monomorphic included (SiblingExt::shannon_entropy_csv, + sibling_annex_stats).
      • +
      • Some(&HashSet<usize>) — a caller-picked subset, skipping the expensive + cross-partition resolution entirely for excluded families + (subsample::sample_layer's accepted-candidates batch; + entropy::ensure_layer_entropy_annex's non-monomorphic-only restriction).
      -

      The two are not correlated by any formula. Converting from one to the other -requires either recomputing the MPHF (kmer → slot) or scanning the iteration -stream (kmer → order). There is no slot → kmer operation: the MPHF is a -one-way function, not an invertible bijection with a stored inverse. Any -method that reconstructs a kmer from a bare slot number is wrong by -construction, regardless of the mechanism used (MPHF re-hash, or evidence -decode + direct unitig read). See MphfLayer::kmer_at -(obikindex/src/layer/mphf_layer.rs) — flagged for removal, currently called -from obikphylo/siblings/build.rs and family_scan.rs (since removed — see -"Pending work" status below).

      -

      Two pipelines, never mixed

      - - - - - - - - - - - - - - - - - - - - - - - -
      origin of the kmermembership known?correct mapping
      query pipelineexternal (caller-supplied)noquery/find/find_strict — MPHF + evidence check
      iteration pipelineenumerated from this layer's own unitigs.binyes, by constructionindex/index_batch — MPHF only, no evidence
      -

      Evidence exists solely to answer "is this external kmer a member of the -layer" for the query pipeline. Using it (or the MPHF) to go the other way — -recover a kmer from a slot, or re-verify a kmer that was just produced by -iterating the layer — is a conceptual error: evidence can be probabilistic -(Approx mode), so any slot→kmer attempt is unsound in general, and -pointless even in Exact/Hybrid mode since the kmer was already known.

      -

      Sibling annex: an iteration-pipeline artifact only

      -

      The sibling annex (FamilyMask/SiblingAnnex, .psib, -obicompactvec/src/siblingannex.rs) records, per kmer, whether it is a -family minorant and which family members are present in the index. Its only -consumers (obikphylo/siblings/stats.rs, family_scan.rs) enumerate it -exhaustively (0..annex.len()); no query-pipeline code path touches it.

      -

      Decision: the annex must be persisted in iteration order, not slot -order. This lets readers zip-iterate Layer::iter_kmers() and the annex -file directly — one linear, cache-friendly pass, no MPHF/slot indirection, -no kmer_at. It also enables specialized iterators building on this zip: -minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.

      -

      Today the annex is built and stored in slot order -(build_layer_sibling_annex, siblings/build.rs): slot_kmer is populated -via (0..n_slots).map(|slot| mphf.kmer_at(slot)), and the origin slot is -threaded through the whole cross-partition reconciliation pipeline (variant -generation, query_partition_with, final mask[slot].fetch_or(...)). This -must change to iterating iter_kmers()/enumerate_kmers() and threading -the iteration index instead of the slot end to end — eliminating -kmer_at from the build path entirely, not just the read path. No -slot-indexed intermediate is needed even during construction; the -iteration-order id is sufficient throughout.

      -

      The cross-partition side of the same pipeline is unaffected: checking -whether a generated family-variant kmer exists in another partition is a -genuine query-pipeline operation (the variant's membership in the target -partition is unknown) and must keep going through -KmerPartition::query_partition_with (MPHF + evidence), never a raw -index().

      -

      Pending work — done

      -

      The plan above shipped: obikphylo (a new crate — phylo-domain extension -traits over obikindex::KmerIndex/obikindex::layer::Layer<D>, replacing the -old obikindex::siblings module) builds and reads the annex purely in -iteration order (SiblingLayerExt::iter_siblings/iter_minorants, both with -batch variants, mirroring Layer<D>'s own KmerIter/KmerBatchIter -shape). MphfLayer::kmer_at has no remaining callers.

      -

      A separate, unrelated bug surfaced during this work and was fixed -(2026-08-14): MphfLayer::enumerate_kmers_batch computed its -batch_start_index via the stdlib .enumerate() adapter, which counts -batches (0, 1, 2…), not the cumulative k-mer offset the annex is actually -keyed on — every batch past the first wrote its mask/annex entries at the -wrong iteration-order position. Fixed by tracking a running offset instead; -regression tests added (sibling_annex_no_empty_masks_after_build, -sibling_histogram_does_not_panic_on_partial_last_batch).

      -

      Performance: build_sibling_annex parallelism (2026-08-14)

      -

      Investigated on a real multi-genome run (phyloskims_sal_vac, k=31/m=11). -Baseline: mostly one active core, with short multi-core bursts — average -~3 cores.

      -

      Fixes that helped, kept:

      +

      minorant_selection::eligible_bitset_layer/non_monomorphic_reindex_layer +provide the cheap, annex-bits-only (no cross-partition resolution) filters +these restricted scans are built from: monomorphism (family_size() < 2) is +knowable straight from the annex, so nothing pays scan_layer_families's +resolution cost just to discard a monomorphic result.

      +

      Masking — computed once, shared everywhere

      +

      algorithms::masking::masked_state(m, free_loss, no_ambiguity) is the single +function that turns a raw per-genome 4-bit state into the masked view every +consumer needs (None for absence under --free-loss or ambiguity under +--no-ambiguity). It is called exactly once per (family, genome), in +subsample::sample_layer, immediately after scan_layer_families resolves +the raw state — never recomputed downstream. survives_masking (does the +family still carry ≥2 distinct masked states among included genomes?) and +SurvivingFamily::masked: Vec<Option<u8>> both consume this same +precomputed array, which is also what AlignmentReducer/PairwiseReducer +read (see below) — this is what guarantees the alignment and the pairwise +tally can never disagree about a genome's state at a given site.

      +

      Precomputed annexes (entropy, family overlap)

      +

      algorithms::entropy::ensure_layer_entropy_annex builds EntropyAnnex +(entropy_annex.rs, one f32 Shannon-entropy value per non-monomorphic +minorant, mmap-backed) and/or feeds FamilyOverlapAccumulator +(family_overlap.rs) from one shared Selection::Some scan — both need the +same genome_mask resolution, so a missing one never pays for it twice. +family_entropy (15 non-empty states of {A,C,G,T}'s powerset, ∅ +excluded from the denominator — the project's settled informativeness +definition, matching the calibrated 16-state Sankoff cost matrix) and +family_entropy_4 (plain 4-symbol reduction, kept only for comparison) are +pure functions of genome_mask. extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes +drives this for every cached layer, lazily (only what's missing).

      +

      Sampling and the shared scan trunk

      +

      algorithms::subsample::sample_index draws up to n non-monomorphic +families index-wide, proportionally per layer: for each layer, a target +quota sized from that layer's share of the index-wide eligible count, filled +by a circular walk with a Bernoulli(p) draw per candidate — p optionally +weighted by an unnormalized Gaussian kernel over the candidate's entropy +(EntropyBias { mu, sigma }, read from the entropy annex). Accepted +candidates are resolved in batches (scan_layer_families) and checked +against survives_masking; rejected-after-masking candidates don't count +toward quota, triggering further "topup" rounds (bounded, MAX_TOPUP_ROUNDS += 5) rather than silently under-filling. Every SurvivingFamily reaching a +caller has already survived both the draw and post-masking informativeness — +no second filter pass needed downstream.

      +

      algorithms::scan::sampled_scan is the single entry point every sampled +computation goes through: one sample_index pass, fanned out to whichever +LayerReducer implementations the caller actually needs, each receiving the +exact same per-layer SurvivingFamily batch (an Arc, cloned not copied). +This is what guarantees several outputs requested together +(--pseudo-alignment and --sankoff and a snp-* --distance in one CLI +invocation) come from one random sample, not independently-drawn ones — +there is no other driver of sample_index anywhere in this crate.

      +

      Two LayerReducer implementations exist today:

        -
      • CanonicalKmerOf::minimizer() (obikseq/src/kmer.rs) — a direct O(k) - bit-arithmetic minimiser for a single isolated k-mer, replacing a - RollingStat instance fed byte-by-byte through an ASCII round-trip (used - by helpers::partition_of, called for every generated family variant). - ~3x wall-clock improvement on its own, confirmed by sampling - (obiskbuilder::rolling_stat/obikentropy frames disappeared from the - hot path). CanonicalKmerOf::partition() added alongside it (wraps - minimizer().seq_hash() & mask, the same routing rule - KmerPartition/RoutableSuperKmer use).
      • -
      • Cross-partition resolution (outgoing.par_iter() in - build_layer_sibling_annex) parallelised at the partition level — one - Rayon task per non-empty outgoing[dest] bucket. For k=31/m=11, a - central-base substitution changes the winning minimiser (and thus the - destination partition) only when that window overlaps the central base: - ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants - route right back to the partition already being built. That self bucket - ends up far larger than any other, so the per-partition split pinned one - thread to it alone while the rest of the pool finished instantly — - confirmed by sampling: one thread solid in MphfLayer::find, everyone - else idle. Fixed by splitting each non-empty bucket into - total_queries / n_workers (capped 4096) chunks before par_iter(), - preserving per-partition mmap locality (each chunk stays contiguous - within one partition) while letting Rayon spread an oversized bucket - across several threads. Net effect of both fixes together: ~3 cores - average → ~10-13 cores average on the same run, and a projected total - build time of ~1h15 down to ~30min on the real phyloskims_sal_vac run - this was measured against.
      • -
      • TracedBar's ETA (obisys/src/progress.rs) was silently starved: the - custom progress message and the self-computed ETA text used to share one - pb.set_message() slot, with the ETA holding off for 2s after any custom - message — fine when custom messages are rare, broken once - build_sibling_annex's per-partition callback fires more often than - that. Fixed by keeping the two texts in separate fields, composed - together on every render instead of one overwriting the other.
      • +
      • AlignmentReducer (alignment.rs) — accumulates SnpAlignment (one + IUPAC-coded row per included genome), reading family.masked directly.
      • +
      • PairwiseReducer (pairwise.rs) — accumulates PairwiseTally/ + PartitionDispersion, also reading family.masked directly (substitution/ + agreement counts and cardinality both derive from the same masked state + the alignment uses).
      -

      Tried and reverted — do not repeat blindly:

      +

      From a tally to distances and calibration

      +

      PairwiseTally stores per-genome-pair substitution/agreement/cardinality +counts, unfiltered, from which every downstream output is cheap O(n²) +post-processing (never a second scan):

        -
      • Parallelising the outer partition loop in build_sibling_annex with - obikindex::PartitionRunner (already used by merge/build_layers), - splitting a fixed core budget between outer (partition) and inner - (pipeline + resolution) concurrency so their product wouldn't exceed the - budget. Measured worse: throughput dropped over time (26 - partitions/5min → 38/11-12min) and peak resolution concurrency fell from - ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which - scales very well on its own — to make room for outer concurrency, and - running several partitions' resolution at once scatters access across - multiple partitions' mmap regions at once, working against the - locality outgoing's per-partition grouping exists for. PartitionRunner - stayed exported from obikindex (new_capped too) since it's - general-purpose, but nothing in obikphylo calls it.
      • -
      • Splitting resolution chunks even finer (/(n_workers*8), cap 1024, - instead of /n_workers, cap 4096) to smooth the residual sawtooth. - Measured ~10% slower, wider dips, not narrower. Reverted to the - original chunk sizing.
      • +
      • algorithms::snp_distance::distance_matrix — closed-form snp-* + corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions- + only), optionally gamma-corrected (GammaShape, PartitionDispersion::estimate_alpha + for Auto).
      • +
      • algorithms::sankoff::SankoffBundle::finish — assembles the + --sankoff/--tnt/--phyg/--iqtree pipeline's inputs + (RawSnpDistanceOutput, BasePairTally, CardinalityTally) from an + already-scanned AlignmentReducer/PairwiseReducer pair, plus + SankoffBundle::snp_distance to reuse that exact sample for a snp-* + --distance in the same invocation without a second scan.
      • +
      • algorithms::cardcomp — pure functions turning CardinalityTally/ + BasePairTally into row-stochastic transition probability matrices, then + the calibrated 16-state Sankoff cost matrix (pairwise_cost_matrix).
      -

      Known remaining limitation, not yet worth fixing: within one layer, the -four stages (sequential unitigs.bin read → parallel generation → -parallel resolution → sequential annex write) never overlap — confirmed by -1s-interval sampling: generation alone occupies ~17 threads evenly, but the -next layer's read/generation never starts until the current layer's -resolution and write are both done. This produces a real, periodic (~layer -duration) alternation between "many cores" and "few cores" that neither of -the fixes above touches, since both operate within one layer's resolution -step. The only remaining lever is overlapping consecutive layers (e.g. a -depth-2 pipeline: start layer N+1's read/generation while layer N's -resolution/write is still running) — a real restructuring, not a parameter -tweak, and explicitly not to be combined with the reverted -budget-capping idea above (let each phase use however many cores it -naturally wants; only the scheduling needs to overlap). Deferred, not -started.

      -

      Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)

      -

      family_scan.rs::scan_layer_families (shared by snp_pseudo_alignment, -sibling_annex_stats, cardinality_tally, scan_family_pairs) already -implements most of a dispatch/accumulate/resolve pipeline: generation -(cheap, CPU-only — builds outgoing[dest_partition] from FamilyMask and -buckets cross-partition queries) runs on an obipipeline::throttle + -make_pipe! stage, decoupled from resolution (I/O-bound, rayon::par_iter -across partitions, one generated batch resolved at a time, never several -concurrently — this ordering is deliberate, see the module's own docs on a -reverted concurrent-batch-resolution attempt that scattered mmap access). -The fast/slow mode gate (PartitionCache::fast_mode, cache.rs:162-163) -already exists: n_layers <= 7 (checked once from the first non-empty -partition's PartitionMeta::n_layers, documented as identical across every -partition of an index — a structural, build-time property, never a -per-partition state) decides whether FamilyMask's recorded layer_value -can be trusted to skip straight to the right layer -(find_presence_batch_fast) or must fall back to scanning every layer of -the destination partition (find_presence_batch).

      -

      Real gap, confirmed not implemented: resolution is triggered by the -source batch finishing (FAMILY_BATCH = 65536 minorants read from the -scanned layer), not by an output accumulator filling up. Since most -central-base variants of a family route back to the same partition being -scanned (~48% per the k=31/m=11 measurement above), a FAMILY_BATCH's -outgoing[dest] is large for the local/self partition and thin for the -other ~255 (or however many) destination partitions — each of those gets -resolved at low query density every batch instead of being accumulated -across several source batches until resolving it is worthwhile. This is -distinct from, and not fixed by, the fast/slow layer gate above.

      -

      Redesign sketched (not built): per-destination accumulators decoupled from -FAMILY_BATCH, flushed on reaching a size threshold instead of on source-batch -completion — a "hot" accumulator for the partition being scanned (sharded -one-per-generation-worker, no lock, since all n_workers pipeline workers -write to it concurrently — this differs from an earlier, simpler mental -model of "one thread owns one layer's local collector," which doesn't hold -here since n_workers threads cooperate on scanning one layer at a time, -not one thread per layer) and "cold" mutex-per-partition accumulators for -the rest, low contention expected since traffic to any single cold -destination is a small fraction of total.

      -

      This breaks the current strict-iteration-order delivery of on_family -(today: a reorder buffer keyed by batch, since a whole FAMILY_BATCH -resolves atomically). With cross-batch accumulation, a family only becomes -complete once every accumulator holding one of its outgoing queries has -flushed, at unpredictable, independent times — no longer streamable -strictly in order without a large, unbounded pending buffer. Resolution -sketched: replace order-dependent consumers with coordinate-addressed -writes instead of order-dependent appends (see PseudoAlignment idea -below) wherever possible, since sibling_annex_stats's reduction (plain -counts) is already order-independent and needs nothing here.

      -

      Superseded 2026-08-15 by the --subsample/--shannon design below, -which sidesteps the accumulator redesign for now: bounding the number of -families actually resolved per layer (via sampling) keeps per-layer -resolution volume small enough that the batch-density problem above stops -mattering in practice for these two consumers. The accumulator redesign -remains relevant for a future unsampled, full-index run, but is not -required to ship --subsample/--shannon.

      -

      Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15)

      -

      The reference run (phyloskims_sal_vac-scale bacterial test set, -iqtree.fasta) produced a dense alignment for 13 genomes × 383,965 sites -(4.8 MB) — trivially small. The in-progress plant index is expected to -carry on the order of 9 billion minorant families; a dense byte-per-cell -alignment at that column count is unbuildable regardless of genome count -(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000– -8,000 genomes on a large machine, which makes the per-cell cost dominant in -the other dimension too. Pruning the retained family set before -materializing anything is mandatory, not an optimization.

      -

      Already free: family_size() < 2 (no sibling variant registered at -all) is a zero-cost structural filter, read directly off FamilyMask bits, -already applied in snp_pseudo_alignment. Insufficient alone — per the -~80-85% mono-family estimate from earlier discussion, this only brings 9 -billion down to roughly 1.3-1.8 billion, still unusable.

      -

      Entropy definition — settled 2026-08-15, correcting an earlier wrong -turn. The project does not encode families as IUPAC ambiguity codes -interpreted the classical way (Fitch-parsimony subset-compatibility, or -ML's "one true state, uncertain which"); see -docmd/theory/evolutionary_distances.md ("Why the IUPAC/DNA encoding used -for the first --snp test was wrong") and the Sankoff resolution that -followed it. The real model is a genuine 16-state alphabet (the powerset of -{A,C,G,T}, ∅ included as a real state) scored with a calibrated -pairwise cost matrix (obikphylo::cardcomp::pairwise_cost_matrix, -cmd/phylo/sankoff.rs), not a compatibility/subset relation between -states. Under that model, each of the 16 states — including multi-bit ones -like AC — is a first-class, independently-costed state, not an -uncertainty encoding of a single true base. So: entropy over the 15 -non-empty states (∅ excluded, matching the earlier decision to exclude -genomes where the family is absent) is the correct informativeness -measure for this project — not a 4-symbol reduction, which would discard -exactly the cardinality/composition information the calibrated cost matrix -is built to exploit.

      -

      --subsample / --shannon — sampling strategy (decided 2026-08-15)

      -

      Goal: make both the pseudo-alignment (--snp) and a Shannon-entropy -diagnostic usable at any index scale, from the 13-genome bacterial -reference run up to the 9-billion-family plant index, without requiring the -batched-accumulator redesign above.

      -

      --subsample N (integer, families to retain): bounds the pseudo-alignment -to N minorant families, sampled proportionally per layer among -non-monomorphic minorants (family_size >= 2) — this sidesteps the need -for a true global reservoir merge across layers while still approximating a -uniform sample over the whole index, and directly answers the earlier open -question of global-vs-per-layer selection scope.

      -

      Three passes, in order:

      -
        -
      1. Global count (cheap, structural, parallel across layers — same shape - as the existing sibling_family_size_histogram, extended to report a - per-layer breakdown rather than one index-wide aggregate): for each - layer, count_layer = number of non-monomorphic minorants. Gives - total_count = Σ count_layer.
      2. -
      3. Per-layer proportional reservoir sampling (cheap, structural, one - pass per layer, no cross-partition resolution): N_layer = round(N × - count_layer / total_count). Since N_layer is a proportion of - count_layer, it can never exceed it as long as N <= total_count — the - one edge case is total_count <= N, in which case sampling is skipped - entirely and every non-monomorphic minorant of every layer is kept - (no reservoir needed, N was never a real constraint). Otherwise: - Algorithm-R reservoir sampling over the layer's non-monomorphic minorant - indices, producing N_layer iteration-order indices directly, no - intermediate full list ever materialized.
      4. -
      5. Filtered resolution (the expensive step, the existing - scan_layer_families engine, unchanged): re-scan the layer, generating - and resolving cross-partition queries only for the indices selected - in step 2 (cheap membership test against a small per-layer index set) — - this is what keeps --subsample cheap even on an unsampled-scale index, - since the cross-partition resolution volume is bounded by N, not by - the layer's true size.
      6. -
      -

      Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir -sampling would waste step-3's expensive resolution work on candidates later -evicted by the reservoir. Step 1 must fully complete (every layer) before -step 2 can start for any layer, since total_count is a global quantity.

      -

      --shannon (no argument): emits a CSV of per-family Shannon entropy -(15 non-empty states, ∅/absent genomes excluded from the denominator, per -the settled definition above). Independent of --subsample — entropy is -computed and written per family as soon as its genome_mask resolves, -O(1) memory per family, so it streams fine even unsampled at full index -scale (a time cost, not a memory one). Combined with --subsample N, it -delivers the original exploratory diagnostic (e.g. --subsample 1000000 ---shannon) directly from this general machinery, rather than a -purpose-built one-off script.

      -

      Validated end-to-end (2026-08-15) against real data: --sibling-hist on -phyloskims_sal_vac (91 real genomes, k=31/m=11, 256 partitions × 2 -layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238 -total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier -guess, corrected here). --subsample/--shannon on the smaller 20-genome -bacterial reference (benchmark/global_index_presence) produced a sample -size within rounding of the request (99,742/100,000) and a -[0.8,1.2)-bucket share (40.7%) matching the full unsampled population -(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a -real bug in reservoir_sample_layer (see next section), now fixed.

      -

      Bug found and fixed (2026-08-15): family_idx numbering mismatch. -scan_layer_families's family_idx counts every minorant of a layer -(monomorphic ones included, since iter_minorants_batch filters only on -is_minorant()), not the raw annex slot (SiblingAnnex::get(slot) spans -every k-mer, minorant or not) and not a counter over non-monomorphic -minorants alone. subsample.rs's reservoir_sample_layer originally -stored raw slot numbers in its HashSet<usize> selection, which drifts -away from family_idx as soon as any monomorphic minorant is seen — -i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by -tracking two separate counters: family_idx (every minorant, matching -scan_layer_families) and seen (non-monomorphic minorants only, what -Algorithm R actually samples over) — only family_idx values are ever -stored in the selection set. The existing unit test never exercised this -(its fixture has exactly one non-monomorphic family, always hitting the -"keep everything" shortcut) — a stronger fixture with several interleaved -monomorphic/non-monomorphic families would be needed to catch a regression -here automatically; not yet written.

      -

      Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15)

      -

      Motivation: on real data (bacterial reference, full unsampled run), only -~5‰ of non-monomorphic minorants fall in the [0.5, 1.5] bit band judged -phylogenetically interesting (entropy too low = uninformative near-invariant -site; too high = saturated/noisy, see family_entropy's 15-state -discussion) — roughly 1 in 10,000 minorants overall. Computing exact -entropy for every candidate just to discard 99.99% of them is wasteful at -the full 9-billion scale.

      -

      The idea: PersistentBitMatrix::col_view(c) gives a genome's whole -presence column as a BitSliceView (sequential, no MPHF, no cross-partition -routing — purely local to one layer's own matrix). Accumulating -TempCompactIntVecBuilder::inc_present(col) (already exists in -obicompactvec/src/builder.rs:121, along with add/min/max/diff on -IntSliceView — no new low-level API needed) over every column of a layer -produces coverage[slot]: how many genomes carry each exact k-mer, in one -sequential per-layer pass, entirely decoupled from family/sibling -structure. Persisted once per layer, a family's members' coverage could -then be looked up via a plain local MPHF index() (cheap) instead of a -full cross-partition presence resolution (find_presence_batch) — the -expensive part today is specifically the cross-partition/cross-layer -routing to a sibling's own matrix, not the bit-reading itself, and -coverage[slot] sidesteps that routing entirely by moving the cost into a -one-time, purely local, embarrassingly-parallel build step.

      -

      Why not implemented: coverage[slot] is a per-member marginal — -summing members' coverages to approximate a family's entropy silently -assumes no genome carries more than one member at once. It cannot -represent or detect joint co-occurrence (a genome carrying both A and -C at once, i.e. a combined 15-symbol state) at all, which is exactly the -phenomenon family_entropy's 15-state definition exists to capture (see -CardinalityTally/cardinality_transition_probs, the project's own -existing machinery for this same co-occurrence structure, built for the -Sankoff matrix calibration). A family that is in truth uniformly AC -across every carrying genome would look like a well-balanced 2-state split -under the marginal approximation (entropy ≈ 1) while its true 15-state -entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the -"saturated, uninformative" tail this pre-filter would need to catch, -undermining the point. It stays plausible as a coarse filter for the -low tail only (a dominant single member's marginal share reliably -predicts low true entropy too), but not as a stand-in for the high tail — -not pursued further for now.

      -

      --free-loss/--tnt pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below)

      -

      Measured on phyloskims_sal_vac (91 genomes): obikmer phylo --subsample 500000 ---free-loss --tnt logs four sequential stages — -raw_snp_distance (1413s), base_pair_tally (1456s), -cardinality_tally (2078s), snp_pseudo_alignment (143s). Reading the -code (obikmer/src/cmd/phylo/mod.rs:210-242, -obikphylo/src/siblings/distance.rs, cardinality.rs, alignment.rs) -surfaced two compounding problems, not one:

      -
        -
      1. Four separate full scans of the annex, each opening its own - KmerPartition/PartitionCache and calling scan_family_pairs/ - scan_layer_families independently — nothing computed in one stage is - reused by another. base_pair_tally is explicitly documented - (distance.rs:138-143) as a second full pass over the same - traversal raw_snp_distance already did, needed only because - raw_snp_distance doesn't keep the resolved bases, only aggregate - counts. cardinality_tally and snp_pseudo_alignment are each a - third and fourth independent full pass. Per the module's own earlier - profiling note (family_scan.rs:26-28, cited already above), this - traversal is page-fault/mmap-bound, not compute-bound — the ~10-12% - CPU efficiency ("contention" status) observed on these three slow - stages is consistent with I/O stalls scaled by repeated full scans, - not lock contention (there are no Mutex/RwLock anywhere in - siblings/*.rs; shared writes use per-slot AtomicU8::fetch_or).
      2. -
      3. Worse: raw_snp_distance and cardinality_tally don't honor - --subsample at all — both call scan_layer_families with - Selection::All hardcoded, and args.subsample isn't even threaded - into their function signatures (mod.rs:212, mod.rs:226). Only - snp_pseudo_alignment(args.subsample) builds a real reservoir-sampled - Selection::Some(set) (via compute_selections, alignment.rs:89). - So today, --subsample 500000 only bounds the pseudo-alignment step — - the SNP-distance matrix, the Sankoff base-pair calibration, and the - cardinality histogram are always computed over the full, unsampled - index regardless of the flag. This is not merely "different subsamples - per stage" (which would already be a problem worth fixing) — it's that - three of the four stages never subsample, which explains most of the - ~10x runtime gap against snp_pseudo_alignment on its own.
      4. -
      -

      Decided requirement: all four stages consume one shared selection, -computed once, not each stage either scanning everything or drawing its -own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair -calibration, the cardinality histogram, and the pseudo-alignment all -describe the same set of families — the calibration and the alignment it -calibrates are now guaranteed to agree on which sites exist.

      -

      Correction to the "single pass" framing above: base_pair_tally/ -cardinality_tally both need raw_snp_distance's complete aggregate -SNP/shared counts before they can derive included[i,j] (the -ratio_ceiling filter) — a genuine sequential dependency (included -can't be known until every pair's aggregate count is final), not an -artifact of the old code's structure. So the fix is two passes over -the shared selection, not one: pass A computes the aggregate counts (and -derives included); pass B fuses base_pair_tally + cardinality_tally -+ the pseudo-alignment (mutually independent once included is known) -into a single scan. Still a 4→2 reduction, and — per the clarification -that settled this — pass A itself now runs over the same shared -selection pass B uses (not the full unsampled index): "les ratios, on -les fait sur les sites sélectionnés, c'est tout, les autres sites -n'existent pas" — once a selection is chosen, both passes are bounded by -it, so on a real --subsample/--entropy run pass A is cheap too, not -just pass B.

      -

      Entropy-biased selection's own resolution to the "forward-looking -complication" (entropy must be known before selection, but selection -happens during the same scan that would resolve it): see "Entropy-biased -selection" below — resolved via a persisted per-layer entropy annex, not -by restructuring the scan into an inline pre-pass.

      -

      Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15)

      -

      Refines the "forward-looking complication" above with a concrete -mechanism. Instead of a hard [low, high] entropy band (or any other -exact cutoff) deciding which non-monomorphic minorant families are -eligible, selection is weighted by an unnormalized Gaussian kernel -centered on a target entropy: w(entropy) = exp(-(entropy - μ)² / (2σ²)) -— deliberately not the normalized Gaussian density (which would peak -below 1 and complicate the "probability" reading) — this kernel form -equals 1 exactly at entropy = μ and decays smoothly to 0 away from it, -so it reads directly as an acceptance weight: the further a family's -entropy from the target, the less likely it is picked, with no hard -in/out boundary — a few "bad" sites can still get in, by design. μ -(default ~1.0) and σ (default ~0.5) are meant to be user-tunable.

      -

      Mechanism: joint probability, not a weighted reservoir. Not -Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex -proposal, superseded before implementation) — instead, a single -independent accept/reject draw per qualifying candidate: draw -u ~ Uniform(0,1), accept iff u < p₀ · w(entropy). p₀ is a single -index-wide rate, N / total_count (total_count = the sum of -non_monomorphic_counts across every layer, N = --subsample's -target), applied identically at every layer — this alone already gives -each layer its proportional share (the same effect the old uniform -reservoir's explicit per-layer n_layer = N · count_layer / total_count -computation achieved, but without needing to compute it: applying one -rate uniformly is mathematically the same as proportioning per layer). -p₀ = 1.0 when there is no --subsample at all — --entropy alone is a -pure soft entropy filter over the whole index, no size target. Properties: -(1) strictly generalizes the existing uniform sampler — with σ large -enough that w ≈ 1 everywhere, this reduces to the old uniform N/total_count -draw; (2) yields "approximately N", not exactly N — expected accepted -count is N · mean(w), always ≤ N — an intentional relaxation, matching -"sous-échantillonnage à environ n" rather than the old reservoir's exact-N -guarantee; (3) one streaming pass, one random draw per candidate, no -reservoir state.

      -

      Resolving "entropy must be known before selection, but selection -happens during the same resolving scan": solved with a persisted, -per-layer entropy annex (obikphylo/src/siblings/entropy_annex.rs, -EntropyAnnex/EntropyAnnexBuilder), not by restructuring the scan. -Mirrors SiblingAnnex's mmap-backed, read-only-after-build convention, -but indexed by family_idx (every minorant of the layer, monomorphic -included — the same numbering Selection/scan_layer_families already -use), one f32 entropy15 value per entry, -1.0 sentinel for monomorphic/ -not-yet-computed. First use of --entropy/--entropy-sd on an index -pays a one-time cost (ensure_entropy_annexes in entropy.rs) — every -later run (any μ/σ, any command) reads the file positionally, no -re-scan, restoring the usual Selection::Some "skip resolving excluded -families" speedup that a naive "weigh during the resolving scan" design -would have permanently forfeited.

      -

      Bug found and fixed (2026-08-15): ensure_entropy_annexes scanned with -Selection::All instead of bounding to non-monomorphic minorants. -Monomorphism (family_size() < 2) is knowable directly from the annex -bits alone, no per-genome resolution needed — but the original -implementation called scan_layer_families with Selection::All -anyway, so fill_sub_matrix_carries (the expensive per-genome -resolution) ran for every minorant, ~98% of which are monomorphic -(measured elsewhere in this doc) and had their genome_mask immediately -discarded once the callback checked family_size() < 2. Fixed by adding -[subsample::non_monomorphic_selection_layer] — a cheap, annex-only, -non-sampling pass (same shape as reservoir_sample_layer, but keeping -every non-monomorphic minorant's family_idx instead of a bounded -reservoir) — and passing Selection::Some(&eligible) instead of -Selection::All, so the expensive resolution now runs only for the ~2% -of minorants that can actually produce a real entropy value. A -debug_assert!(mask.family_size() >= 2, ...) inside the -scan_layer_families callback guards the invariant.

      -

      Resolved: the existing hard "non-monomorphic minorant" eligibility -filter stays a hard gate upstream of the Gaussian weighting — only -qualifying families ever get a stored entropy value or a weighted draw.

      -

      CLI, implemented: two phylo options, --entropy <μ> and ---entropy-sd <σ> (obikmer/src/cmd/phylo/args.rs). The entropic filter -activates as soon as either is given (mod.rs, computed once into an -Option<EntropyBias> threaded through --snp/--family-overlap/ ---shannon/the fused sankoff pipeline below). If active but one or both -are unset, defaults are μ = 1.0, σ = 0.5.

      -

      Two entropy definitions kept side by side, for comparison (2026-08-15)

      -

      --shannon's CSV carries both entropy15 (family_entropy — the settled -15-non-empty-state definition, see above) and entropy4 (family_entropy_4 -— plain nucleotide reduction), computed from the same already-resolved -genome_mask, not from the marginal approximation above. A genome carrying -several bases at once contributes to each base's count (counted once per -base present, not fractionally split, not folded into one combined state) -— a genome polymorphic for the family is present at more than one base by -construction, so it is expected to count more than once; the denominator is -the total base-occurrence count, not the genome count (the two coincide -only when no genome carries more than one base). Kept side by side -specifically to measure, on real data, how much the two diverge — not yet -analyzed.

      -

      PersistentSparseBitMatrix — implemented and measured (2026-08-15)

      -

      A row-major (k-mer-major), deduplicated sparse alternative to -obicompactvec::PersistentBitMatrix, motivated by the same sparsity that -drove --subsample/--shannon above, but pursued as a foundational -storage-layer change rather than an index-level workaround. Full design -history, rationale, and rejected alternatives (external Elias-Fano crates, -cacheline-ef, a single unsplit dict_id array) are in the dedicated -implementation plan (vivid-mapping-tiger.md at the time of writing — the -content below is the durable summary, not a pointer to a session-scoped -file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi, -"Compact Data Structures for Collections of Sets" (2025, -biblio/Alanko et al. - Compact Data Structures for Collections of -Sets.pdf) — this design implements only their exact-duplicate special -case (a plain dedup dictionary), not their full subset-containment -hierarchy.

      -

      Design: four on-disk components, each mmap-backed, built once per -layer (matching how the rest of the build pipeline already works — never -the whole multi-billion-row index at once): an is_multi rank-capable -flag per row (singleton vs. multi-genome), a fixed-bit-width array for -singleton rows (genome index directly, ceil(log2(n_cols)) bits), a -separate fixed-bit-width array for multi-genome rows (dict_id, -ceil(log2(n_distinct_multi_sets)) bits — kept apart from the singleton -array specifically because n_distinct_multi_sets can be large in -absolute terms even when multi-genome rows are a small fraction of all -rows, and a single shared array would force every row, singletons -included, to pay the wider width), and a deduplicated dictionary of -distinct multi-genome sets (Elias-Fano-encoded byte offsets + a -varint-encoded values blob). New low-level primitives added to -obicompactvec to build this: PersistentFixedIntVec (arbitrary, -runtime-parameterized bit width, width 0 included — needed once a real -bug surfaced, see below), PersistentRankSelectBitVec (rank1/rank0/select1 -on top of the crate's existing count_ones, using -common_traits::SelectInWord), EliasFano (composes the two). A new -BinaryMatrix trait (n, n_cols, row/fill_row, fill_sub_matrix, -count_ones) unifies dense and sparse at the one call site that needs -both interchangeably (obikphylo::siblings::cache::Mat) — column-oriented -methods (col, col_view, the partial_*_dist_matrix family) stay -dense-only.

      -

      Two real bugs caught by tests, not by inspection: (1) EliasFano::open -re-derived its low-bits width from the persisted low-vector file's own -width byte; the zero-width case was built with a dummy 1-bit placeholder -(the builder rejected true width 0), so every reopened value silently -doubled. Fixed by making PersistentFixedIntVec genuinely support width 0 -(no storage, get always 0) instead of working around the limitation in -EliasFano. (2) An empty row (cardinality 0 — not expected on a real -built index, but not guarded against either) was recorded as a singleton -at genome 0, indistinguishable on read-back from a real singleton at -genome 0. Fixed by routing cardinality-0 rows through the dictionary path -(a genuine empty entry) instead of the singleton shortcut. Both caught by -obicompactvec's test suite (142 tests, including disk-reopen round-trips -that drop every builder/mmap before reopening fresh), not by manual -review — worth remembering next time a "this edge case can't happen in -practice" shortcut is tempting.

      -

      Measured on real data (layer_1 of phyloskims_sal_vac's -part_00018, 30,246,774 rows, 91 genomes — #[ignore]d benchmarks in -obikphylo/src/siblings/tests.rs):

      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
      densesparseratio
      on-disk size328.1MB43.7MB7.5x smaller
      build time / peak RSS—4.26s / 628MB(per-layer, in-memory construction — comfortable)
      row access, sequential (2M reads)43ns/row32ns/rowsparse faster (smaller structure, better cache fit)
      row access, random (2M reads)409ns/row85ns/rowsparse ~4.8x faster (the real --shannon/family-lookup shape)
      column access, one full column (30.2M rows)11.5ms993mssparse 86x slower (no native column method — every read decodes a full row to keep one bit)
      -

      The row-access wins (both directions) weren't the design's stated goal — -compactness was — but turn out real: dense's genome-major layout scatters -a single row read across a much bigger file, which costs more than -sparse's rank/select/varint decode once the file is this much smaller. -The column-access cost is the flip side of the same layout choice, and is -exactly what the next item below exists to fix.

      -

      Next, not yet planned: rewrite partial_jaccard_dist_matrix/ -partial_hamming_dist_matrix/etc. (obicompactvec/src/bitmatrix/pairwise.rs) -as a row-major co-occurrence accumulation (O(Σ_rows k²), per-row -increments into an NxN genome-pair counter — the known alternative to -today's column-fold, plausibly cheaper on data this sparse, not just a -fallback) so obikindex's --metric/distance-matrix path can use the -sparse type without the measured 86x column-access penalty. Needs its own -design pass (in particular how it plugs into the BitPartials/ -ColumnWeights traits so both matrix types keep serving --metric) -before implementation — not just "port the loop", a genuinely different -algorithm.

      -

      Also still deferred, unchanged from the implementation plan: full Alanko -et al. subset-hierarchy compression (only the exact-duplicate special case -is built), a sparse PersistentCompactIntMatrix (count matrices), and -BRWT-style column-correlation exploitation.

      -

      Wired into pack and the sibling-annex build path (2026-08-15)

      -

      PersistentSparseBitMatrix went from a validated but unused type to a -real, selectable on-disk format:

      +

      Public API surface

      +

      extensions::SiblingExt (CLI-facing trait on IndexCache) covers only the +operations with no cross-invocation sharing concern: build_sibling_annex, +shannon_entropy_csv, sibling_family_size_histogram/sibling_annex_stats, +family_overlap. The sampled SNP path (sampled_scan, AlignmentReducer, +PairwiseReducer, distance_matrix, SankoffBundle) is not wrapped in +trait methods — the CLI (obikmer::cmd::phylo) composes them directly, +because it alone knows the union of reducers a given invocation needs +(--pseudo-alignment/Sankoff group/snp-* --distance can combine in one +call). A per-output trait method would have to either duplicate that +composition logic or reintroduce the "two independent scans" problem this +design replaced.

      +

      Known gaps

        -
      • Generic Layer<D>: obikindex::layer::Layer<D>'s presence-only methods - (n_cols, sub_matrix, fill_sub_matrix) are generic over any - D: LayerData<Item = Box<[bool]>> + BinaryMatrix, not hardcoded to - PersistentBitMatrix — PersistentSparseBitMatrix implements - LayerData (open/read) the same way. find_slot/index_batch were - already generic over any D: LayerData, so they needed no change. - Verified by obikindex::layer's - presence_layer_generic_over_sparse_matches_dense test: build a dense - presence layer, convert it to sparse via build_from_dense, open both - as Layer<PersistentBitMatrix>/Layer<PersistentSparseBitMatrix> on - the same directory, assert n_cols/sub_matrix/find_slot agree. - (This test must stay at k=4 with mutually non-colliding canonical - 4-mers across its input sequences — K/M are process-wide - AtomicUsizes in test builds, not thread-local, so a test using a - different k races every other test in the same crate binary; a k=11 - version of this test passed alone but failed under the full - obikindex::layer suite for exactly that reason before being fixed.)
      • -
      • obikphylo::siblings::cache::Mat gained a third variant, - SparsePresence(Layer<PersistentSparseBitMatrix>), alongside Count - and Presence — every method (find_slot, index_batch, - iter_minorants_batch, n_cols, fill_sub_matrix_carries) dispatches - to it identically to Presence, since both go through the same generic - Layer<D> code. PartitionCache::build picks the variant per layer by - checking for presence/is_multi.prsb (the sparse format's own marker - file, see the design section above) before falling back to the dense - open path.
      • -
      • pack_sparse_bit_matrix (new, obicompactvec::bitmatrix::sparse): - pack --sparse's entry point. Idempotent (checks is_multi.prsb - first); packs to dense matrix.pbmx first if that hasn't happened yet - (the dense→sparse transpose needs random row access, which only the - packed/columnar dense forms give), then build_from_denses the sparse - form into the same directory and deletes matrix.pbmx — old-format - files are removed only after the new format is fully written, mirroring - pack_bit_matrix's own crash-safety convention.
      • -
      • CLI: obikmer pack --sparse threads a sparse: bool through - KmerIndex::pack_matrices (all other call sites — select, merge, - finalize_indexed — pass false, unchanged dense behaviour). Count - matrices are untouched by --sparse (no sparse PersistentCompactIntMatrix - — see "still deferred" above).
      • -
      • End-to-end coverage: obikphylo::siblings::tests:: - sibling_annex_works_after_pack_sparse builds a two-genome index, packs - it --sparse, asserts is_multi.prsb exists, then runs - build_sibling_annex and checks the resulting FamilyMasks match the - dense-path test (sibling_annex_one_sibling_each) exactly — proves the - sparse format round-trips through the real build pipeline - (PartitionCache sparse-detection included), not just the - obicompactvec/obikindex::layer unit layers below it.
      • -
      -

      Full workspace cargo test (all crates, unit + doc tests) green after -this change.

      -

      Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15)

      -

      Replaces the "four independent scans" problem above and implements -"Entropy-biased selection" above, end to end:

      -
        -
      • SankoffBundleExt::sankoff_bundle (new, - obikphylo/src/siblings/sankoff_bundle.rs) — the --sankoff/--tnt/ - --phyg/--iqtree block in obikmer/src/cmd/phylo/mod.rs now calls - this once instead of three separate raw_snp_distance/ - base_pair_tally/cardinality_tally calls. One PartitionCache, one - shared (possibly subsampled/entropy-biased) selection computed once via - compute_selections, two scans over it: pass A (aggregate SNP/shared - counts, --exclude-genome zeroing, then included[i,j]), pass B - (base_pair_tally + cardinality_tally + the pseudo-alignment, fused - into one scan_layer_families call per layer, all three read off the - same resolved genome_mask). snp_pseudo_alignment/ - shannon_entropy_csv (still used standalone by --snp/ - --family-overlap/--shannon) both gained an entropy_bias parameter - too, so entropy-biased selection isn't sankoff-specific.
      • -
      • Regression proof: sankoff_bundle_matches_old_separate_calls - (obikphylo/src/siblings/tests.rs) asserts sankoff_bundle's four - outputs are bit-identical to calling the old, separate - raw_snp_distance/base_pair_tally/cardinality_tally/ - snp_pseudo_alignment on the same fixture with no subsample — the - fusion is a performance change, not a behavior change.
      • -
      • EntropyAnnex/EntropyAnnexBuilder (new, - obikphylo/src/siblings/entropy_annex.rs) and ensure_entropy_annexes - (entropy.rs) implement the persisted-entropy mechanism from - "Entropy-biased selection" above. entropy_annex_builds_on_demand_and_biases_selection - (tests.rs) proves, on a fixture with one known-entropy family: the - annex file doesn't exist before any entropy-biased call; compute_selections - builds it on first use; μ set to the family's exact entropy with - p₀ = 1.0 selects it deterministically (u < 1.0 always, for - u ∈ [0,1)); μ set far away with tiny σ deterministically excludes - it (w underflows to exactly 0.0); the persisted value matches - --shannon's own family_entropy computation to 1e-6.
      • -
      • EntropyBias (pub, obikphylo::siblings::EntropyBias { mu, sigma }) - is the one new public type threading --entropy/--entropy-sd through - every Option<EntropyBias>-accepting method — resolved once in - obikmer/src/cmd/phylo/mod.rs from args.entropy/args.entropy_sd - (activation: either given; defaults 1.0/0.5 for whichever is unset).
      • -
      -

      Full workspace cargo test green after this change (167 unit tests in -obicompactvec+obikindex::layer+obikphylo alone, plus every other -crate's suite, no regressions).

      -

      Still open, not part of this change (per "Correction to the 'single -pass' framing" above): --raw-snp-distance/--raw-snp-counts (the -standalone diagnostic flags, not the --sankoff pipeline) still always -scan the full unsampled index — never threaded --subsample/--entropy, -out of scope here since the reported problem was specifically about the ---sankoff/--tnt pipeline's redundant/inconsistent scans, not these -two standalone flags.

      -

      query never benefits from sparse row-major access (found 2026-08-19, not implemented)

      -

      Benchmarked obikmer query against global_index_presence (dense-packed) -vs. global_index_presence_sparse (pack --sparse), 100k simulated reads -× 2 specimens (benchmark/, see -benchmark_query_testing.md). -Correctness: 0 mismatches — sparse and dense return bit-identical query -results. Performance: sparse consistently slower than dense (~30-50%, -reproducible across two runs with warm disk cache), the opposite of -pack --sparse's stated intent ("faster for single-row access... like -query").

      -

      Root cause, read from source, not measured in isolation: -KmerPartition::query_partition_with (obikindex/src/partition/query_layer.rs:155-220) -is architecturally column-major: stage 2 walks for g in 0..n_cols { for -slot in hit_slots { layer.col_value(g, slot) } }, documented (correctly) -as the right locality strategy for the packed/columnar formats, where -col_value → PersistentBitMatrix::get is a genuine O(1) mmap'd column -read (persistent.rs:110-113).

      -

      For Self::Sparse, that same get(c, slot) (persistent.rs:114-118) -allocates a full n_cols-wide buffer and calls fill_row — materializing -the entire row — just to return one cell. Called from inside the -column-major double loop, this reconstructs the same row once per genome -column touched: O(hits × n_cols) full-row rebuilds instead of O(hits). -PersistentSparseBitMatrix's own native row-major decode -(for_each_genome_in_row, sparse.rs:164-177, used correctly by its own -row/fill_row/fill_sub_matrix) is never reached from the query path -at all.

      -

      fill_sub_matrix (the existing BinaryMatrix trait primitive, -traits.rs:13-37) is not the right replacement for query either, even -once its own dispatch bug is fixed (see next section) — its output shape -is inherently column-dense: out[col] gets an entry for every column, -including columns with zero hits among the requested slots. On real -sparse data (a hit typically touching a handful of genomes out of dozens) -that's still O(n_cols) output regardless of true sparsity. What query -actually wants is the sparse triple stream (slot, col, value) it already -consumes as QueryHit::Value — not a materialized sub-matrix.

      -

      Proposed primitive (design only, not implemented — explicit ask: keep -count matrices not excluded, even though effort right now is -presence/absence only):

      -

      Not a closure-driven for_each — a real Iterator, one concrete struct -per matrix format, so the traversal state (current position in the sorted -slot list, current column, permutation, sparse-row decode cursor…) lives -in named struct fields instead of being threaded implicitly through -recursion or a captured closure. RPITIT (stable since 1.75, and this -workspace is edition 2024) means the trait method can return it without -naming or boxing the concrete type:

      -
      /// Yields every nonzero cell among `slots`, in implementation-defined order.
      -fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator<Item = (usize, usize, u32)> + 'a;
      -// item: (idx into `slots`, col, value)
      -
      -

      This is the one foundational primitive per format — both fill_sub_matrix -and a for_each-style callback become trivial, free consumers of it -(.for_each(f) is already Iterator::for_each from std; fill_sub_matrix -becomes "drain the iterator, scatter into out[][]"), instead of two -independently-maintained traversals that can silently diverge (see the bug -below — this is exactly how it happened).

      -
        -
      • On PersistentSparseBitMatrix: the struct is nearly free to write — - it wraps the existing (currently private) for_each_genome_in_row - per-row decode, advancing to the next slots entry on exhaustion. O(Σ - row nnz), zero n_cols-wide allocation.
      • -
      • On PersistentBitMatrix::{Packed,Columnar}: revised — cheaper than - first thought, by reusing the same split already used for - fill_matrix's own implementation instead of hand-writing a resumable - state machine at the matrix level. The base-vector layer - (BitSliceView, views.rs) already separates the two concerns: - fill_batch_sorted (views.rs:55-60, sorted-slot batch lookup) and a - genuine per-bit Iterator (BitSliceIter, views.rs:94+) sit side by - side there, one level below the matrix. Adding a - "positions among sorted_slots where the bit is set" iterator at that - same vector level is a filter over the existing get() — no new - state machine, since std::iter::Filter already is one, generated by - the compiler: -
        fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator<Item = usize> + 's {
        -    sorted_slots.iter().copied().filter(move |&slot| self.get(slot))
        -}
        -
        - The matrix-level nonzero_iter then composes these per column with - flat_map over 0..n_cols (each column's hits, tagged with c, - slot mapped back through the sort permutation fill_batch/ - fill_batch_sorted already carry) — again a combinator chain, not a - hand-rolled struct. Same algorithm, same mmap/sort locality as today's - fill_sub_matrix; just assembled from std iterator adaptors instead - of a loop body writing into a buffer, mirroring the vector/matrix split - the codebase already uses for fill_batch_sorted rather than - introducing a new shape.
      • -
      • Implicit: trivial (slots.iter().map(|&i| (i, 0, 1)), one column, - always present).
      • -
      • On PersistentCompactIntMatrix (counts): same treatment as - Packed/Columnar — no sparse count format exists yet ("Explicitly - deferred" per traits.rs:9-12), so no native low-effort case the way - Sparse has one, but not excluded either: the iterator's Item is - already (usize, usize, u32), not bool, specifically so presence - (0/1) and counts (arbitrary u32) share one primitive instead of a - bool/u32 split forcing counts out of the design. Ready for a native - sparse-count struct later without a signature change.
      • -
      -

      Would let query_partition_with's stage 2 collapse to one -for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) } -per layer, format-agnostic, each backend's struct deciding the actual -traversal.

      -

      This also closes the existing dispatch bug for free, by construction: -PersistentBitMatrix::fill_sub_matrix (persistent.rs:190-215, the enum -wrapper backing BinaryMatrix's trait impl) today does not delegate to -PersistentSparseBitMatrix::fill_sub_matrix for Self::Sparse — it -reimplements the same naive per-(column, slot) fill_row_bool loop -instead, bypassing the efficient native method one file over -(sparse.rs:249-258). obikphylo::siblings::cache::Mat -(cache.rs:138-145) independently built its own parallel enum wrapper -that dispatches correctly — evidence this was worked around rather than -fixed at the source: two hand-written traversals for the same format, -free to drift apart, and they did. If fill_sub_matrix itself is -rewritten as "drain nonzero_iter, scatter into out[][]", there is only -one traversal per format left to get right — the bug class doesn't just -get fixed once, it stops being possible to reintroduce.

      -

      Implemented (2026-08-20)

      -

      Built as designed above, with one deviation from the original sketch: -nonzero_iter ended up Box<dyn Iterator<...>>, not a bare impl -Iterator, because Columnar/Packed/Sparse/Implicit are genuinely -different concrete types and this method isn't on a trait (kept off -BinaryMatrix deliberately — that trait is used as dyn BinaryMatrix in -tests/sparse.rs, and RPITIT methods aren't dyn-compatible). One Box -per nonzero_iter call, not per cell — negligible next to what it -replaces.

      -
        -
      • BitSliceView::nonzero_among_sorted / IntSliceView::nonzero_among_sorted - (obicompactvec/src/views.rs): the vector-level filter/filter_map - primitive, exactly as sketched — no new state machine, std's own.
      • -
      • PersistentSparseBitMatrix::nonzero_iter (bitmatrix/sparse.rs): native, - std::iter::from_fn over one buffered row at a time via the existing - for_each_genome_in_row — no n_cols-wide allocation, ever.
      • -
      • PersistentBitMatrix::nonzero_iter (bitmatrix/persistent.rs): dispatches - to the above for Sparse; for Columnar/Packed, loops columns, - collects each column's nonzero_among_sorted hits via .extend() (not - flat_map — a flat_map closure can't lazily return something - borrowing its own captured sort permutation across separate calls - without either boxing per-column or fighting the borrow checker; eager - collection into one Vec sidesteps it, at zero cost since - fill_sub_matrix already fully materialized anyway). Implicit trivial.
      • -
      • PersistentBitMatrix::fill_sub_matrix and sub_matrix rewritten to - drain nonzero_iter — the dispatch bug is gone because there is now - only one traversal per format, not because the old one was patched. - PersistentCompactIntMatrix::nonzero_iter added the same way (counts - not excluded, per the earlier ask) — no native low-effort case, since no - sparse count format exists, but on the same primitive, ready for one.
      • -
      • KmerPartition::query_partition_with (obikindex/src/partition/query_layer.rs): - stage 2's column-major for g { for slot { col_value } } replaced by one - layer.nonzero_iter(&slot_list) call per layer, format-agnostic.
      • -
      • Tests: nonzero_iter_matches_dense, nonzero_iter_matches_row, and — - the one that actually targets the dispatch bug rather than each type's - own correctness — enum_wrapper_dispatches_to_native_sparse (builds - PersistentBitMatrix::Sparse(...) directly, not through open, since - open only auto-detects Sparse from a presence/ dir layout). - cargo test --workspace: green, no regressions.
      • -
      -

      Measured: re-ran the benchmark/ query branch (100k reads × 2 -specimens, same setup as the original finding). Correctness still 0 -mismatches. The dense/sparse performance gap is gone — previously sparse -~30-50% slower than dense, reproducibly; now within ~1-3% either way -(7.42s dense vs 7.60s sparse for Escherichia_coli--K-12_MG1655; 5.25s vs -5.30s for Saccharolobus_islandicus--M.16.4) — noise-level, not a -systematic gap. pack --sparse's claimed query win isn't confirmed -outright by this (sparse should arguably now beat dense on truly sparse -real data, not just tie), but the pathological regression is fixed.

      -

      PersistentCompactIntMatrix::Sparse — implemented (2026-08-26)

      -

      Closes the gap flagged throughout this document ("no sparse count format -exists yet", traits.rs:9-12's "Explicitly deferred"): obicompactvec -already had PersistentSparseCompactIntMatrix (row-major, built on top of -PersistentSparseBitMatrix as its "which columns are non-zero" support, -values not deduplicated — see that struct's own doc comment), but it was -never wired into PersistentCompactIntMatrix, the dense-dispatching enum -every real consumer (TypedLayer<PersistentCompactIntMatrix>, -KmerLayer::Count) actually holds. Concretely: kmer_index.rs:: -pack_matrices(sparse=true) already called pack_sparse_compact_int_matrix -on every layer's counts/ — but PersistentCompactIntMatrix::open had no -code path back to what that just wrote, so a Count layer became -unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment -anyone ran pack --sparse on an index with count layers. Root cause, not a -workaround: add the missing Sparse variant.

      -
        -
      • -

        Enum + dispatch (intmatrix.rs): PersistentCompactIntMatrix::Sparse - (PersistentSparseCompactIntMatrix), detected in open/detect_storage - via a singleton_values.pciv marker (mirrors PersistentBitMatrix's own - sparse_meta.json check), reported via storage_kind(). col/ - col_view/col_persist panic/Unsupported on Sparse, same convention - as the bit side. sub_matrix/fill_sub_matrix and nonzero_iter - unified the same way PersistentBitMatrix's already are (drain - nonzero_iter, one traversal per format — see "Implemented - (2026-08-20)" above); nonzero_iter had to become Box<dyn Iterator<...>> - for the same reason (Columnar/Packed/Sparse are different concrete - types). No change needed in obikindex at all — KmerLayer::Count - already only ever holds TypedLayer<PersistentCompactIntMatrix>, so the - enum absorbing Sparse fixes the unreadable-layer bug for free, same as - PersistentBitMatrix::Sparse already did on the presence side.

        -
      • -
      • -

        CountPartials, non-naive (sparse_intmatrix.rs): unlike - PersistentSparseBitMatrix's dict-driven col_weights_and_pair_counts, - values here aren't deduplicated (two rows can share the same non-zero - column set via the same dict_id while carrying different counts), so - the "weight by how many rows share a dict entry" shortcut doesn't carry - over. What does: a single row-major pass (row_major_pairwise, decodes - each row once via for_each_cell_in_row, nests over that row's own - co-present columns) — O(Σ k̄²) over populated rows instead of the naive - O(n_cols² × n) column-pair rescan, same complexity class as the bit - side minus the dict multiplicity discount. Kernels used: min(a,b) - (bray, relfreq-bray — both vanish when either side is absent, so no - correction needed), a·b and √(a·b) (euclidean/relfreq-euclidean and - hellinger — these do need a correction, reconstructed from per-column - marginals via Σ(a-b)² = Σa²+Σb²-2Σab, since (a-0)² = a² ≠ 0 unlike - the min-based formulas). threshold_jaccard(1) shortcuts straight to - support's own BitPartials::partial_jaccard (threshold 1 is exactly - presence); threshold_jaccard(0) is closed-form (every u32 is ≥ 0).

        -
      • -
      • -

        Two pre-existing bugs found and fixed while wiring the threshold==1 - shortcut (bitmatrix/sparse.rs, BitPartials for - PersistentSparseBitMatrix, present since the 2026-08-15 implementation - above, never caught because no test compared Sparse's raw partial_* - output against dense on real data — only the diagonal-blind - jaccard_dist_matrix/hamming_dist_matrix finalisations were tested):

        -
      • -
      • partial_jaccard's diagonal was (0, 2×col_weights[i]) instead of a - genuine self-comparison (col_weights[i], col_weights[i]) — - col_weights_and_pair_counts's inter never pairs a column with - itself by construction.
      • -
      • partial_hamming's off-diagonal formula itself was wrong: total - - union (count of rows where neither column is present) instead of - the actual Hamming distance col_weights[i] + col_weights[j] - - 2×inter[i,j] (symmetric-difference size). Only coincides with the - correct value when col_weights[i] + col_weights[j] == total, so - small/synthetic test data could easily have hidden it.
      • -
      -

      Neither surfaced through jaccard_dist_matrix/hamming_dist_matrix - (both explicitly zero their own diagonal at finalisation, and the - off-diagonal partial_hamming bug had gone untested against dense - entirely) — only visible to a caller of the raw partial_* methods - directly, which is exactly what partial_threshold_jaccard(1)'s new - shortcut became. Fixed at the source, not patched around at the call - site; regression test added: - tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal.

      -
        -
      • Tests: tests::intmatrix::sparse_roundtrip_matches_columnar/ - sparse_roundtrip_from_packed (the open-dispatch fix, both build - paths); tests::intmatrix::sparse_count_partials_match_dense (all six - CountPartials formulas, thresholds 0/1/2/3, against Columnar on - asymmetric-presence data — this is what caught the diagonal gap in the - int side's own new code before it shipped, the same way it exposed the - two pre-existing bit-side bugs above); obikindex's - count_layer_transparently_reads_sparse_after_pack — the actual - end-to-end regression test for the original "layer unreadable after - pack --sparse" bug, built → packed sparse → reopened, compared against - the pre-pack dense read. cargo test -p obicompactvec -p obikindex: - green, no regressions (180 + 12 tests).
      • +
      • sample_index's skip_layers parameter (checkpointed-resume support) is + never called with a non-zero value by any current caller — --session + persistence was removed pending a redesign on top of sampled_scan + (single shared trunk, not per-driver checkpoint plumbing). The parameter + stays as the seam a future resume mechanism would use.
      • +
      • No cheap entropy pre-filter exists yet for the very high (saturated) tail — + a row-marginal coverage approximation was considered and rejected (cannot + represent joint multi-base co-occurrence, which is exactly what the + 15-state entropy definition exists to capture).
      diff --git a/DevDoc/implementation/benchmark_query_testing/index.html b/DevDoc/implementation/benchmark_query_testing/index.html index c8e7c979..14817889 100644 --- a/DevDoc/implementation/benchmark_query_testing/index.html +++ b/DevDoc/implementation/benchmark_query_testing/index.html @@ -1265,7 +1265,7 @@ - Sibling annex (discussion) + Sibling annex @@ -1485,8 +1485,9 @@ this pipeline.

      by verify_presence/verify_merge_presence, which only check index content against the .npz truth, never the query API.
    • pack --sparse produces a presence-matrix format documented (see - siblings.md) as faster for single-row - access (query) and slower for column-oriented access (phylo --metric). + obicompactvec.md, "Sparse matrix formats") as faster + for single-row access (query) and slower for column-oriented access + (phylo --metric). global_index_presence/ built by merge_presence.sh is always packed dense (packing is a stage inside merge, not a separate pack invocation) — there was no dense/sparse regression check.
    • diff --git a/DevDoc/implementation/chunkreader.refs/index.html b/DevDoc/implementation/chunkreader.refs/index.html index d19f2081..7bb38ea2 100644 --- a/DevDoc/implementation/chunkreader.refs/index.html +++ b/DevDoc/implementation/chunkreader.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/chunkreader/index.html b/DevDoc/implementation/chunkreader/index.html index b1ed11a9..6c386b97 100644 --- a/DevDoc/implementation/chunkreader/index.html +++ b/DevDoc/implementation/chunkreader/index.html @@ -579,7 +579,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/evidence_elimination.refs/index.html b/DevDoc/implementation/evidence_elimination.refs/index.html index 1fa176b3..b29cbf5d 100644 --- a/DevDoc/implementation/evidence_elimination.refs/index.html +++ b/DevDoc/implementation/evidence_elimination.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/evidence_elimination/index.html b/DevDoc/implementation/evidence_elimination/index.html index 70621d90..7db92264 100644 --- a/DevDoc/implementation/evidence_elimination/index.html +++ b/DevDoc/implementation/evidence_elimination/index.html @@ -1265,7 +1265,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/filtering/index.html b/DevDoc/implementation/filtering/index.html index 6fdab235..781161fe 100644 --- a/DevDoc/implementation/filtering/index.html +++ b/DevDoc/implementation/filtering/index.html @@ -1388,7 +1388,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/kmer.refs/index.html b/DevDoc/implementation/kmer.refs/index.html index b73ef4d2..898475b5 100644 --- a/DevDoc/implementation/kmer.refs/index.html +++ b/DevDoc/implementation/kmer.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/kmer/index.html b/DevDoc/implementation/kmer/index.html index f367ae72..ca7782a3 100644 --- a/DevDoc/implementation/kmer/index.html +++ b/DevDoc/implementation/kmer/index.html @@ -1265,7 +1265,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/layer_tier.refs/index.html b/DevDoc/implementation/layer_tier.refs/index.html index 92d01fcf..aff4e5af 100644 --- a/DevDoc/implementation/layer_tier.refs/index.html +++ b/DevDoc/implementation/layer_tier.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/layer_tier/index.html b/DevDoc/implementation/layer_tier/index.html index 2dc88b7a..048b4831 100644 --- a/DevDoc/implementation/layer_tier/index.html +++ b/DevDoc/implementation/layer_tier/index.html @@ -1504,7 +1504,7 @@ - Sibling annex (discussion) + Sibling annex @@ -2208,7 +2208,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec<Vec<bool>> pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec<bool>])
-

Column-first to match the on-disk column-major layout. fill_sub_matrix sorts slots once, then calls each column's fill_batch in turn — no redundant per-column sort. On PersistentSparseBitMatrix (k-mer-major, no column method) this degrades to a row-by-row decode; see siblings.md.

+

Column-first to match the on-disk column-major layout. fill_sub_matrix sorts slots once, then calls each column's fill_batch in turn — no redundant per-column sort. On PersistentSparseBitMatrix (k-mer-major, no column method) this goes through nonzero_iter instead, decoding one row at a time; see obicompactvec.md, "Sparse matrix formats".


LayeredStore\<S> and aggregation traits

LayeredStore<S> is a generic aggregation wrapper over Vec<S>. It propagates three traits from obicompactvec::traits up the hierarchy via blanket impls:

diff --git a/DevDoc/implementation/merge.refs/index.html b/DevDoc/implementation/merge.refs/index.html index 8f4e288c..22d6d050 100644 --- a/DevDoc/implementation/merge.refs/index.html +++ b/DevDoc/implementation/merge.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/merge/index.html b/DevDoc/implementation/merge/index.html index 9f91a463..b1050f4e 100644 --- a/DevDoc/implementation/merge/index.html +++ b/DevDoc/implementation/merge/index.html @@ -1348,7 +1348,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/merge_parallelism/index.html b/DevDoc/implementation/merge_parallelism/index.html index b7395e98..d78249bb 100644 --- a/DevDoc/implementation/merge_parallelism/index.html +++ b/DevDoc/implementation/merge_parallelism/index.html @@ -1332,7 +1332,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/mphf.refs/index.html b/DevDoc/implementation/mphf.refs/index.html index 331707e7..b48c8bda 100644 --- a/DevDoc/implementation/mphf.refs/index.html +++ b/DevDoc/implementation/mphf.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/mphf/index.html b/DevDoc/implementation/mphf/index.html index 010a5215..67587caa 100644 --- a/DevDoc/implementation/mphf/index.html +++ b/DevDoc/implementation/mphf/index.html @@ -1343,7 +1343,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/implementation/obicompactvec/index.html b/DevDoc/implementation/obicompactvec/index.html index 9f6c2fff..d1f0eea5 100644 --- a/DevDoc/implementation/obicompactvec/index.html +++ b/DevDoc/implementation/obicompactvec/index.html @@ -494,7 +494,7 @@ - Sibling annex (discussion) + Sibling annex @@ -632,6 +632,46 @@
  • + + + + Sparse matrix formats + + + + +
  • +
  • @@ -1191,7 +1231,7 @@ Pass 1 — byte max, SIMD-vectorizable, O(n) Sparse bit: sparse_meta.json + PFIV/Elias-Fano component files, row-major. Int: same support files (built on PersistentSparseBitMatrix internally) plus singleton_values.pciv/multi_values.pciv/multi_offsets for the per-row, non-deduplicated values -pack --sparse; see siblings.md for the sparse-vs-dense access-pattern trade-off +pack --sparse; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off Implicit (bit only) @@ -1201,7 +1241,6 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)

    PersistentBitMatrix::open(layer_dir) auto-detects the variant, in order: matrix.pbmx → Packed, presence/meta.json → Columnar, presence/sparse_meta.json → Sparse, layer_meta.json (no presence dir at all) → Implicit. PersistentCompactIntMatrix::open(layer_dir) mirrors the same priority order minus Implicit (there's no implicit count matrix — counts always have at least one on-disk column): matrix.pcmx → Packed, counts/meta.json → Columnar, counts/singleton_values.pciv → Sparse. col_view/col/sub_matrix panic on Sparse/Implicit where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through row/fill_row.

    -

    Unlike the bit side, PersistentSparseCompactIntMatrix's values are not deduplicated across rows — two rows can share the same non-zero column set (same dict_id in the shared support) while carrying different counts — so its CountPartials impl can't reuse the support's dict-multiplicity shortcut the way BitPartials for PersistentSparseBitMatrix does. It still avoids the naive O(n_cols² × n) column-pair scan via a single row-major pass (row_major_pairwise in sparse_intmatrix.rs), reconstructing the squared-difference formulas (euclidean/relfreq_euclidean/hellinger) from per-column marginals via Σ(a-b)² = Σa²+Σb²-2Σab — see siblings.md's "PersistentCompactIntMatrix::Sparse — implemented" entry for the full derivation.

    col_view(c) returns the appropriate view directly:

    // PersistentBitMatrix
     pub fn col_view(&self, c: usize) -> BitSliceView<'_>
    @@ -1212,6 +1251,183 @@ Pass 1 — byte max, SIMD-vectorizable, O(n)
     

    No wrapper enums (BitColView, IntColView): the caller receives a Copy view struct immediately usable with any view method or bulk builder method.

    pack_compact_int_matrix and pack_bit_matrix convert columnar → packed format.


    +

    Sparse matrix formats

    +

    Motivated by real k-mer-family data being extremely row-sparse (most rows — +k-mers — present in only a handful of genome columns out of dozens to +thousands): a row-major, deduplicated alternative to the dense column-major +formats above, built once per layer like the rest of the pipeline (never the +whole multi-billion-row index at once). Implements the exact-duplicate +special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data +Structures for Collections of Sets" (2025, +biblio/Alanko et al. - Compact Data Structures for Collections of +Sets.pdf) — a plain dedup dictionary, not their full subset-containment +hierarchy.

    +

    PersistentSparseBitMatrix

    +

    Four mmap-backed on-disk components: an is_multi rank-capable flag per row +(singleton vs. multi-genome), a fixed-bit-width array for singleton rows +(genome index directly, ceil(log2(n_cols)) bits), a separate fixed-width +array for multi-genome rows (dict_id, ceil(log2(n_distinct_multi_sets)) +bits — kept apart from the singleton array because n_distinct_multi_sets +can be large in absolute terms even when multi-genome rows are a small +fraction of all rows, and a shared array would force every row, singletons +included, to pay the wider width), and a deduplicated dictionary of distinct +multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values +blob). Supporting primitives: PersistentFixedIntVec (arbitrary, +runtime-parameterised bit width, width 0 included — needed once an +EliasFano reopen bug surfaced, see below), PersistentRankSelectBitVec +(rank1/rank0/select1 on top of the crate's existing count_ones, via +common_traits::SelectInWord), EliasFano (composes the two). BinaryMatrix +(n, n_cols, row/fill_row, fill_sub_matrix, count_ones) is the +trait unifying dense and sparse at the call sites that need both +interchangeably; column-oriented methods (col, col_view, the +partial_*_dist_matrix family) stay dense-only — Sparse/Implicit panic +there (see the PersistentBitMatrix::open note above).

    +

    Measured on real data (one layer, 30.2M rows, 91 genomes):

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    densesparseratio
    on-disk size328.1MB43.7MB7.5x smaller
    build time / peak RSS—4.26s / 628MBper-layer, in-memory construction
    row access, sequential (2M reads)43ns/row32ns/rowsparse faster (smaller structure, better cache fit)
    row access, random (2M reads)409ns/row85ns/rowsparse ~4.8x faster — the --shannon/family-lookup access shape
    column access, one full column (30.2M rows)11.5ms993mssparse 86x slower — no native column method, every read decodes a full row to keep one bit
    +

    The row-access wins (compactness was the stated design goal, not speed) come +from dense's genome-major layout scattering a single row read across a much +bigger file — costlier than sparse's rank/select/varint decode once the file +is this much smaller. Column access is the flip side of the same layout +choice; nonzero_iter (below) is what removes it from the hot path that +actually matters (query).

    +

    Two reopen-correctness bugs, now fixed at the source: (1) EliasFano::open +re-derived its low-bits width from the persisted low-vector's own width +byte; the width-0 case was built with a dummy 1-bit placeholder (the +builder used to reject true width 0), silently doubling every reopened +value. Fixed by making PersistentFixedIntVec genuinely support width 0 +(no storage, get always 0). (2) An empty row (cardinality 0) used to be +recorded as a singleton at genome 0, indistinguishable on read-back from a +real singleton at genome 0. Fixed by routing cardinality-0 rows through +the dictionary path (a genuine empty entry).

    +

    Still deferred: full Alanko et al. subset-hierarchy compression (only the +exact-duplicate case is built), a sparse PersistentCompactIntMatrix +variant beyond what's described below, BRWT-style column-correlation +exploitation, and a row-major rewrite of +partial_jaccard_dist_matrix/partial_hamming_dist_matrix/etc. as a +per-row co-occurrence accumulation (O(Σ_rows k̄²)) so obikindex's +--metric distance-matrix path stops paying the column-access penalty on +Sparse — needs its own design pass (how it plugs into +BitPartials/ColumnWeights so both matrix types keep serving --metric), +not just "port the loop".

    +

    nonzero_iter — the one traversal per format

    +

    fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a> +(item: index into slots, column, value) is the single foundational +per-format traversal primitive every consumer builds on — fill_sub_matrix +and a for_each-style callback are both trivial, free consumers of it +(.for_each(f) is Iterator::for_each; fill_sub_matrix drains the +iterator and scatters into out[][]), instead of two independently +hand-maintained traversals that can silently diverge. Box<dyn Iterator>, +not a bare impl Iterator: Columnar/Packed/Sparse/Implicit are +genuinely different concrete types and the method isn't on a +dyn-compatible trait (BinaryMatrix is used as dyn BinaryMatrix +elsewhere, and RPITIT methods aren't dyn-compatible) — one Box per +nonzero_iter call, not per cell.

    +

    Per-format implementation: PersistentSparseBitMatrix decodes natively, +buffered one row at a time (std::iter::from_fn over the existing +for_each_genome_in_row, no n_cols-wide allocation ever). Columnar/ +Packed loop columns, collecting each column's hits via +BitSliceView::nonzero_among_sorted/IntSliceView::nonzero_among_sorted +(a filter/filter_map over the vector-level get(), no new state +machine — std's own Filter iterator already is one) into one Vec via +.extend() (not flat_map: a flat_map closure can't lazily return +something borrowing its own captured sort permutation across separate calls +without either boxing per-column or fighting the borrow checker — eager +collection sidesteps it at zero extra cost, since fill_sub_matrix already +fully materialises anyway). Implicit is trivial (slots.iter().map(|&i| (i, 0, 1))).

    +

    This closed a real dispatch bug by construction, not by patching it: +PersistentBitMatrix::fill_sub_matrix/sub_matrix used to reimplement a +naive per-(column, slot) loop instead of delegating to +PersistentSparseBitMatrix's own efficient native method for Self::Sparse +— two hand-written traversals for the same format, free to drift apart, and +they had (obikphylo's own cache layer had independently built a parallel +enum wrapper that did dispatch correctly, evidence the bug was worked +around rather than fixed at the source). With fill_sub_matrix/sub_matrix +rewritten to drain nonzero_iter, there is only one traversal per format +left to get right. KmerPartition::query_partition_with +(obikindex/src/partition/query_layer.rs) — previously column-major +(for g { for slot { col_value } }, which for Sparse rebuilt a full row +per single cell touched, O(hits × n_cols) instead of O(hits) — the original +motivation for this primitive) — now issues one layer.nonzero_iter(&slot_list) +call per layer, format-agnostic. Dense/sparse query performance parity +confirmed on real data (previously sparse 30-50% slower than dense, +reproducibly; now within ~1-3% either way — noise level).

    +

    PersistentSparseCompactIntMatrix

    +

    The count-matrix analogue, built on PersistentSparseBitMatrix as its +"which columns are non-zero" support, values not deduplicated (unlike +the bit side, two rows can share the same non-zero column set — same +dict_id — while carrying different counts, so the dict-multiplicity +shortcut BitPartials for PersistentSparseBitMatrix uses doesn't carry +over). CountPartials here: a single row-major pass (row_major_pairwise +in sparse_intmatrix.rs, decodes each row once via +for_each_cell_in_row, nests over that row's own co-present columns) — +O(Σ k̄²) over populated rows, avoiding the naive O(n_cols² × n) +column-pair rescan. min(a,b)-based kernels (bray, relfreq_bray) need +no correction — they vanish when either side is absent. a·b and √(a·b) +kernels (euclidean/relfreq_euclidean/hellinger) do need one, since +(a-0)² = a² ≠ 0 unlike the min-based formulas — reconstructed from +per-column marginals via Σ(a-b)² = Σa²+Σb²-2Σab. +threshold_jaccard(1) shortcuts to BitPartials::partial_jaccard +(threshold 1 is exactly presence); threshold_jaccard(0) is closed-form +(every u32 is ≥ 0).

    +

    Wiring the threshold==1 shortcut surfaced two pre-existing formula bugs in +BitPartials for PersistentSparseBitMatrix (present since that type's own +introduction, never caught because no test had compared Sparse's raw +partial_* output against dense on real data — only the diagonal-blind +jaccard_dist_matrix/hamming_dist_matrix finalisations were tested), +fixed at the source:

    +
      +
    1. partial_jaccard's diagonal used to be (0, 2×col_weights[i]) instead + of a genuine self-comparison (col_weights[i], col_weights[i]) — + col_weights_and_pair_counts's inter never pairs a column with itself + by construction, so the diagonal needs its own case.
    2. +
    3. partial_hamming's off-diagonal formula itself was wrong: it computed + total - union (rows where neither column is present) instead of the + actual Hamming distance col_weights[i] + col_weights[j] - 2×inter[i,j] + (symmetric-difference size) — the two only coincide when + col_weights[i] + col_weights[j] == total, which small/synthetic test + data can easily satisfy by accident.
    4. +

    Aggregation traits (matrix level)

    ColumnWeights

    trait ColumnWeights: Send + Sync {
    diff --git a/DevDoc/implementation/obipipeline.refs/index.html b/DevDoc/implementation/obipipeline.refs/index.html
    index 46dace87..ba0cda60 100644
    --- a/DevDoc/implementation/obipipeline.refs/index.html
    +++ b/DevDoc/implementation/obipipeline.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/obipipeline/index.html b/DevDoc/implementation/obipipeline/index.html
    index edebe6b6..8c864f3f 100644
    --- a/DevDoc/implementation/obipipeline/index.html
    +++ b/DevDoc/implementation/obipipeline/index.html
    @@ -1293,7 +1293,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/obitaxonomy/index.html b/DevDoc/implementation/obitaxonomy/index.html
    index b1886a24..a9d00b9e 100644
    --- a/DevDoc/implementation/obitaxonomy/index.html
    +++ b/DevDoc/implementation/obitaxonomy/index.html
    @@ -1305,7 +1305,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/partition_layer_cache/index.html b/DevDoc/implementation/partition_layer_cache/index.html
    index 1cce3095..07dd753c 100644
    --- a/DevDoc/implementation/partition_layer_cache/index.html
    +++ b/DevDoc/implementation/partition_layer_cache/index.html
    @@ -1563,7 +1563,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_bit_vec.refs/index.html b/DevDoc/implementation/persistent_bit_vec.refs/index.html
    index 452c7db1..bccfdd30 100644
    --- a/DevDoc/implementation/persistent_bit_vec.refs/index.html
    +++ b/DevDoc/implementation/persistent_bit_vec.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_bit_vec/index.html b/DevDoc/implementation/persistent_bit_vec/index.html
    index 3e40d825..a8701785 100644
    --- a/DevDoc/implementation/persistent_bit_vec/index.html
    +++ b/DevDoc/implementation/persistent_bit_vec/index.html
    @@ -1405,7 +1405,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
    index 9e306acb..7d6b5deb 100644
    --- a/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
    +++ b/DevDoc/implementation/persistent_compact_int_vec.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/persistent_compact_int_vec/index.html b/DevDoc/implementation/persistent_compact_int_vec/index.html
    index 05ea4d9f..ab5d8aa2 100644
    --- a/DevDoc/implementation/persistent_compact_int_vec/index.html
    +++ b/DevDoc/implementation/persistent_compact_int_vec/index.html
    @@ -1388,7 +1388,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/pipeline.refs/index.html b/DevDoc/implementation/pipeline.refs/index.html
    index 3ab74967..985e28c7 100644
    --- a/DevDoc/implementation/pipeline.refs/index.html
    +++ b/DevDoc/implementation/pipeline.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/pipeline/index.html b/DevDoc/implementation/pipeline/index.html
    index a697eee7..fc08d198 100644
    --- a/DevDoc/implementation/pipeline/index.html
    +++ b/DevDoc/implementation/pipeline/index.html
    @@ -1254,7 +1254,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/select/index.html b/DevDoc/implementation/select/index.html
    index c98f84e1..c3cb4add 100644
    --- a/DevDoc/implementation/select/index.html
    +++ b/DevDoc/implementation/select/index.html
    @@ -1382,7 +1382,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/storage.refs/index.html b/DevDoc/implementation/storage.refs/index.html
    index 03bfe3b3..06d67528 100644
    --- a/DevDoc/implementation/storage.refs/index.html
    +++ b/DevDoc/implementation/storage.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/storage/index.html b/DevDoc/implementation/storage/index.html
    index 7fd77097..ce4b99a7 100644
    --- a/DevDoc/implementation/storage/index.html
    +++ b/DevDoc/implementation/storage/index.html
    @@ -1326,7 +1326,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/superkmer.refs/index.html b/DevDoc/implementation/superkmer.refs/index.html
    index 946e4da1..6622b610 100644
    --- a/DevDoc/implementation/superkmer.refs/index.html
    +++ b/DevDoc/implementation/superkmer.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/superkmer/index.html b/DevDoc/implementation/superkmer/index.html
    index d5267a43..6d6069fe 100644
    --- a/DevDoc/implementation/superkmer/index.html
    +++ b/DevDoc/implementation/superkmer/index.html
    @@ -1232,7 +1232,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/unitig_evidence.refs/index.html b/DevDoc/implementation/unitig_evidence.refs/index.html
    index 1dfbaa9c..4118486a 100644
    --- a/DevDoc/implementation/unitig_evidence.refs/index.html
    +++ b/DevDoc/implementation/unitig_evidence.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/implementation/unitig_evidence/index.html b/DevDoc/implementation/unitig_evidence/index.html
    index a66adbd3..32ef44a3 100644
    --- a/DevDoc/implementation/unitig_evidence/index.html
    +++ b/DevDoc/implementation/unitig_evidence/index.html
    @@ -1354,7 +1354,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/index.html b/DevDoc/index.html
    index e6629e0a..06cc62b0 100644
    --- a/DevDoc/index.html
    +++ b/DevDoc/index.html
    @@ -1228,7 +1228,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/index.refs/index.html b/DevDoc/index.refs/index.html
    index 5f0a7f61..b229367c 100644
    --- a/DevDoc/index.refs/index.html
    +++ b/DevDoc/index.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/installation/index.html b/DevDoc/installation/index.html
    index 42e8a762..d044a12d 100644
    --- a/DevDoc/installation/index.html
    +++ b/DevDoc/installation/index.html
    @@ -1314,7 +1314,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/kmers.refs/index.html b/DevDoc/kmers.refs/index.html
    index b12faff2..f789c1f2 100644
    --- a/DevDoc/kmers.refs/index.html
    +++ b/DevDoc/kmers.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/kmers/index.html b/DevDoc/kmers/index.html
    index a707331c..89637ba8 100644
    --- a/DevDoc/kmers/index.html
    +++ b/DevDoc/kmers/index.html
    @@ -1227,7 +1227,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/sitemap.xml.gz b/DevDoc/sitemap.xml.gz
    index 03a78e939757e3344acaf2aa3115fcca5d967fa9..15bde70bddf05ec6e4e702da2414775779e38149 100644
    GIT binary patch
    delta 13
    Ucmb=gXP58h;9#g
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/encoding/index.html b/DevDoc/theory/encoding/index.html
    index e8a2dbb1..abfa12e4 100644
    --- a/DevDoc/theory/encoding/index.html
    +++ b/DevDoc/theory/encoding/index.html
    @@ -1199,7 +1199,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/entropy.refs/index.html b/DevDoc/theory/entropy.refs/index.html
    index 49c1b70d..1a9e9081 100644
    --- a/DevDoc/theory/entropy.refs/index.html
    +++ b/DevDoc/theory/entropy.refs/index.html
    @@ -1124,7 +1124,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/entropy/index.html b/DevDoc/theory/entropy/index.html
    index 5e201572..f913aeca 100644
    --- a/DevDoc/theory/entropy/index.html
    +++ b/DevDoc/theory/entropy/index.html
    @@ -1265,7 +1265,7 @@
       
         
       
    -    Sibling annex (discussion)
    +    Sibling annex
       
     
         
    diff --git a/DevDoc/theory/evolutionary_distances/index.html b/DevDoc/theory/evolutionary_distances/index.html
    index 77c3d6cf..57634792 100644
    --- a/DevDoc/theory/evolutionary_distances/index.html
    +++ b/DevDoc/theory/evolutionary_distances/index.html
    @@ -932,6 +932,17 @@
         
       
       
    +
  • + +
  • + + + + Automatic α estimation (--gamma-shape auto) + + + +
  • @@ -943,6 +954,45 @@ +
  • + +
  • + + + + α-estimation literature survey and design discussion (2026-09-11) + + + + +
  • + +
  • + + + + Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11 + + + + + +
  • @@ -1691,7 +1741,7 @@ - Sibling annex (discussion) + Sibling annex @@ -2268,6 +2318,17 @@ +
  • + +
  • + + + + Automatic α estimation (--gamma-shape auto) + + + +
  • @@ -2279,6 +2340,45 @@ +
  • + +
  • + + + + α-estimation literature survey and design discussion (2026-09-11) + + + + +
  • + +
  • + + + + Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11 + + + + + +
  • @@ -4476,9 +4576,75 @@ F84/TN93 and is included below. snp- prefix on every CLI value.

    +Γ rate-heterogeneity modifier, applicable to snp-jc, snp-k2p, snp-k81, snp-t92, snp-f84, snp-hky85, snp-tn93 (not snp-raw, nothing to correct; not snp-logdet, no standard gamma formulation) — same -formula as the base correction, weighted by a shape parameter α supplied -by the user (--gamma-shape <alpha>), not estimated by ML. A modifier on -existing values, not a separate enum arm per distance.

    +formula as the base correction, weighted by a shape parameter α either +supplied by the user (--gamma-shape <alpha>) or estimated from the data +(--gamma-shape auto/estimate, method-of-moments — not ML; see +"Automatic α estimation" below). A modifier on existing values, not a +separate enum arm per distance.

    +

    Automatic α estimation (--gamma-shape auto)

    +

    Correction (verified against the primary source, 2026-09-11): Jin & +Nei (1990) itself ("Limitations of the Evolutionary Parsimony Method of +Phylogenetic Analysis", Mol. Biol. Evol. 7(2):82–102 — the paper this +whole +Γ correction is cited from, confirmed algebraically to match this +codebase's corrected_log/k2p exactly against their eq. A4, general, and +A8, the a = 1 case) contains no data-driven α-estimation procedure. +Their own recommendation (p. 98) is a fixed default: "we suggest that the +gamma distance with a = 1 [eq. A8] be used. However, one may choose a +different gamma distance, estimating a from data. Wilson et al. (1989) +recently used a distance with a = 1/2 for restriction-site data of +mitochondrial DNA in hominoids." — i.e. Jin & Nei explicitly punt +data-driven estimation to a different paper (Wilson et al. 1989), not +read/verified here. The estimator below is therefore not "Jin & Nei's +method" under any framing — that attribution (present in an earlier +revision of this section) was wrong, not just under-cited.

    +

    Implemented (PartitionDispersion, obikphylo/src/siblings/algorithms/pairwise.rs) +as an independent method-of-moments estimator, unrelated to any specific +published procedure: pools substitution counts by partition rather +than by genome pair, during the same reduce_pairwise pass that builds +PairwiseTally (no second scan).

    +

    For partition i: n_i = substitutions pooled over every genome pair, +L_i = eligible loci pooled over every genome pair, R_i = n_i / L_i. +Modeling among-site rate heterogeneity the same way as the +Γ correction +itself (a Gamma(α, α)-distributed, mean-1, multiplicative rate shared by +every locus in a partition — the classical Poisson-Gamma/negative-binomial +mixture, the general identity behind gamma-rate-heterogeneity corrections, +also behind Uzzell & Corbin's (1971) original observation that substitution +counts across sites/regions are over-dispersed relative to Poisson):

    +
    \[ +\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha} +\]
    +

    Weighting each partition's squared deviation by its own L_i cancels the +Poisson term before attributing what's left to α:

    +
    \[ +\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad +V = \frac{\sum_i L_i (R_i-\hat\mu)^2}{\sum_i L_i} \qquad +\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L} +\]
    +

    where \bar L is the mean partition size. Returns "no estimate" (falls +back to the uncorrected formula, warns) when fewer than 2 partitions have +data, \hat\mu \le 0, or V doesn't exceed the Poisson floor +\hat\mu/\bar L — no detectable over-dispersion, α would be unbounded.

    +

    Caveat, stated explicitly rather than left implicit: unlike every +closed-form correction in snp_distance.rs (each verified line-by-line +against ape's dist_dna.c, and now also against Jin & Nei 1990 directly +for the base +Γ formula), this estimator is derived from first +principles (the general Poisson-Gamma/negative-binomial identity) with no +primary-source procedure behind it at all — not Jin & Nei's (confirmed +above), and Wilson et al. (1989), the paper they point to instead, hasn't +been read/verified either. Mathematically self-consistent (re-derived and +checked, not guessed), but a from-scratch method, not a literature +implementation. If --gamma-shape needs a value with a literature +pedigree rather than an estimated one, Jin & Nei's own stated default, +α = 1 (--gamma-shape 1), is the better-supported choice today.

    +

    Deliberately not gated by --sankoff-ratio-ceiling the way +base_pair_tally is (same precedent as cardinality_tally — see its own +doc comment): that filter excludes individual saturated pairs from a +composition estimate computed once at the very end, from the complete +PairwiseTally; the partition axis needed here only exists transiently, +one partition at a time, while PairwiseTally is still being built — long +before any pair's final SNP ratio (and thus its ratio_ceiling eligibility) +is known. --exclude-genome isn't applied either, matching +reduce_pairwise's own raw per-pair fold.

    Implemented now: snp-raw, snp-jc, snp-k2p, snp-k81, snp-f81, snp-t92, snp-tn93, snp-tv, all with +Γ except raw/tv — see "Exact formulas" below. snp-tajima-nei, snp-f84, snp-hky85, @@ -4591,6 +4757,430 @@ special-casing K80). K81/T92/TN93's gamma branches follow the same mechanical substitution but weren't independently checked against an ape-equivalent reference for those three specifically — flagged here, not silently assumed correct.

    +

    α-estimation literature survey and design discussion (2026-09-11)

    +

    Follow-up discussion after implementing PartitionDispersion::estimate_alpha +(previous section), triggered by the user pointing out that the literature +conflates two distinct problems: estimating α from an alignment versus +using a given α in a Jin-Nei-style corrected distance. Jin & Nei (1990) +itself is squarely in the second camp — α is a user-supplied parameter +there, not something their paper estimates (confirmed by reading the paper +directly, see previous section). This section catalogs the actual +α-estimation literature and records which approaches were considered and +why each was or wasn't adopted.

    +

    Bibliography (methodological papers, not exhaustive users of +Γ):

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    MethodReferenceNeeds a tree?Estimates "true" α or a task-specific one?Cost
    ML, continuous Γ (foundational)Gu, Fu & Li (1995), MBE 12:546–557 — read in fullyes, topology fixed a prioritrue α (+ θ jointly, invariant+Γ)very high, ≤5–6 taxa in practice
    ML, discrete ΓYang (1993/94), MBE 10:1396; JME 39:306yestrue αhigh
    Parsimony-based ML on inferred changesYang & Kumar (1996), MBE 13:650yes, equal branch lengths assumedtrue α, biased when branches unequallow
    Corrected substitutions/site → NB fitGu & Zhang (1997), MBE 14:1106 — read in fullyes, + ancestral reconstructiontrue α, close to ML accuracylow (given a tree)
    Topology-optimizing grid searchGuindon & Gascuel (2002), MBE 19:534 — read in fullyes, but built from the candidate distances themselves (BIONJ)not true α — deliberately biased upward for topological accuracylow–moderate (grid × tree build)
    Free (non-parametric) rate distributionSusko et al. (2003), Syst. Biol. 52:594yestests the Γ assumption itselfhigh
    Parsimony counts/site → NB fitWakeley (1993), JME 37:613 — read in fullyes, NJ tree + Fitch parsimony ancestral reconstructiontrue α, biased upward by parsimony (conservative)low (given a tree)
    Poisson-Gamma/negative-binomial, foundationalUzzell & Corbin (1971), Science 172:1089 — not read, only seen via citationsunknown — likely also count-per-site/tree-based, unverifiedtrue α (indirect, via NB overdispersion)unknown
    Bayesian posterior over αBPP/MrBayes/BEAST2/RevBayesyesposterior, not point estimatevery high
    +

    Correction (2026-09-11, after reading Wakeley in full): an earlier +revision of this table filed Wakeley (1993) under "no tree needed" — +wrong. Wakeley's method: build a tree (neighbor-joining, Saitou & Nei +1987, on the real mtDNA data; coalescent-simulated for the simulation +study), reconstruct ancestral states at internal nodes by Fitch (1971) +parsimony, count the minimum number of changes per site this implies, +then fit either a two-rate Poisson mixture or a gamma-distributed-rates +model (⟹ negative binomial, parameters by the methods of Cohen 1965 and +Johnson & Kotz 1969) to the resulting per-site count distribution — the +same general "tree + ancestral reconstruction + per-site counts + NB fit" +shape as Gu & Zhang (1997), just using raw parsimony counts directly +rather than Gu & Zhang's multiple-hit-corrected k̂. Confirmed by +simulation (Wakeley's own Fig. 1) that parsimony underestimates both the +mean and (more severely) the variance of per-site change counts, which +biases α upward (toward apparent uniformity) — consistent with Gu & +Zhang's own citation of Wakeley for exactly this bias, and with the +general "parsimony overestimates α" pattern noted throughout this +literature. Real hypervariable-region-1 mtDNA data gave α̂ ≈ 0.44–0.60 +per data set, 0.47 combined — despite the conservative bias, still +solidly in the "strong heterogeneity" range, which is the paper's own +point: the method is biased but usefully conservative, not useless.

    +

    Where PartitionDispersion::estimate_alpha actually sits: nowhere in +this table's tree-dependent rows. It shares only the abstract +mathematical identity (Poisson-Gamma mixture ⟹ negative binomial +over-dispersion) with Wakeley/Uzzell-Corbin/Gu&Zhang, applied via a +genuinely different, tree-free and ancestral-reconstruction-free route: +pairwise genome-to-genome substitution counts pooled by partition +instead of per-site counts pooled across an alignment's columns after +ancestral state reconstruction. As far as this survey has established +(Uzzell & Corbin not read, so not fully ruled out), no tree and no +ancestral reconstruction of any kind appears to be a genuinely +distinguishing property of PartitionDispersion, not something it +inherits from prior art — consistent with it being an independent +derivation rather than a literature implementation (see previous +section's caveat, now on firmer footing).

    +

    Why a tree-free estimate is even valid — the identifiability question. +Raised directly by the user after reading Gu, Fu & Li (1995), which states +explicitly: "one cannot estimate the rate heterogeneity when the number +of sequences is <3." This is a real non-identifiability result, not a +practical inconvenience: with only 2 sequences, a site's history is +summarized by one binary observation (differ / don't differ), collapsed +across all sites into two aggregate numbers (transition and transversion +proportions) — divergence time and among-site rate variance are +confounded in that pair of numbers, with no way to separate them. Every +method surveyed above (Gu,Fu&Li, Yang, Wakeley, Gu&Zhang) needs ≥3 +sequences sharing one evolutionary history (a tree) specifically to get +multiple independent looks at the same site's rate across different +lineages, which is what identifies its variance.

    +

    PartitionDispersion doesn't violate this, because it isn't estimating +the same quantity. The literature above estimates rate variation across +sites, within one shared tree. PartitionDispersion estimates rate +variation across partitions, pooled over every genome pair in the +index — each genome pair stands in for one independent draw of a +partition's relative rate, the same identifying role multiple lineages +play in the tree-based methods, just substituting "genome pair" for +"lineage" and "partition" for "site." With P genome pairs (potentially +in the hundreds here) all contributing to every partition's pooled count, +the ≥3-samples identifiability requirement is met by the pair count, not +by tree depth — closer in spirit to Jin & Nei's own original motivating +scenario (comparing several genomic regions across one fixed panel of +taxa) than to single-tree site-rate estimation.

    +

    This substitution carries its own assumption, which must be named rather +than left implicit: that a partition's relative rate (fast/slow) is +reasonably stable across genome pairs, not just across sites within +one pair — the direct analogue of the "no lineage-specific rate variation, +only site-specific" assumption every tree-based +Γ method already makes, +just moved from lineage→pair and site→partition. If this breaks — e.g. an +index mixing very closely related and very divergent genome pairs, where a +partition's saturation behavior differs qualitatively between the two — +the estimate could be misled in a way a tree-based method would at least +have the topology to detect and a tree-free, pool-everything method +cannot. Not yet tested against real data with strongly heterogeneous +pairwise divergence; worth keeping in mind as the main open validity +question for --gamma-shape auto, not the citation question (now settled) +this whole discussion started from.

    +

    Gu, Fu & Li (1995), read in full — the foundational ML method, not a +candidate to implement. Models rate variation as invariant+Γ (θ = +proportion of invariant sites, α = gamma shape among the variable sites), +and derives the exact joint likelihood over nucleotide configurations +across all n sequences on a fixed, a-priori-known tree topology — +for 3 sequences this is closed-form (Jukes-Cantor), for n sequences it's +a sum over 4^n configurations weighted by coefficients tied to the +specific topology's branching structure (their eq. 32–34), with branch +lengths, α, and θ jointly optimized by Newton-Raphson (Hessian over all +free parameters). The site-rate integral has the same closed form +E = θ + (1-θ)(1+D/α)^{-α} that underlies +Γ distance corrections +generally (the same mechanism as corrected_log, not a coincidence — Jin & +Nei's own formula is the θ=0 special case applied to a pairwise D +rather than a whole-tree one). Explicitly stated by the authors as +infeasible beyond about 5–6 taxa (their own simulations stop at 5); not a +candidate for obikmer's typical genome counts. Notable contribution worth +keeping in mind regardless: ρ = (1+θα)/(1+α) is shown to be a more robust +summary of rate heterogeneity than α alone whenever θ isn't negligible +— α and θ are confounded (nearly-invariant sites can come from either a +small α's left tail or from a nonzero θ), so α alone can be very +unstable while ρ stays well-behaved. PartitionDispersion has no θ +term at all (no invariant-site component), so this confound doesn't arise +for it the same way, but it's worth remembering if an invariant+Γ variant +is ever considered.

    +

    Gu & Zhang (1997), read in full — rejected for now. Their procedure: +(1) a tree with least-squares branch lengths must already exist; (2) +ancestral states are reconstructed at every internal node (they use Zhang & +Nei 1997's likelihood method, but note plain parsimony would work, just +less accurately); (3) per site, branches are split into "changed"/"unchanged" +given the ancestral reconstruction, and the expected substitution count +k̂ (corrected for multiple hits) is obtained by solving their eq. (7) +(or eq. 12 for the generalized model) numerically — a root-find per site, +not a closed form; (4) the site-level k̂ values (real-valued, not +integers) are fit to a negative binomial (Uzzell & Corbin 1971) by ML to +get α — again no closed form, numerical optimization. This is a real +per-site pipeline requiring a tree before distances/α can be computed — +the reverse of obikmer's current dependency order (snp_distance computes +distances, which --nj/--upgma only turn into a tree afterward). Their +own answer to this chicken-and-egg problem is an iterative +distance→tree→α→distance loop, which they themselves flag as expensive. +Adopting this would mean a new ancestral-reconstruction subsystem (even a +parsimony/Fitch-only version) plus a per-family nonlinear solve plus an NB +ML fit — a much bigger addition than PartitionDispersion, not something +to build without a concrete need beyond "closer to Gu&Zhang's own +simulated accuracy than our moment estimator."

    +

    Guindon & Gascuel (2002), read in full — plausible future addition, not implemented. +Their Q criterion: for a candidate α, build a tree (they use BIONJ) from +the α-corrected distance matrix; for every internal branch, group taxa into +the 4 subtrees it separates (A, B, C, D), compute mean inter-subtree +distances \(\bar\delta_{AB}\), \(\bar\delta_{CD}\), etc., and let S ≤ M ≤ L +be the three pairings' sums ({AB,CD}, {AC,BD}, {AD,BC}); the branch's +reliability score is Q_branch = L − M (zero when the four points are +perfectly tree-additive). The whole-tree Q is the mean over internal +branches (negative/zero branches excluded). α* = argmin_α Q(α) over a +grid (their run: ~60–100 points, 0.1 to 5000, finer spacing where +sensitivity is highest). Complexity: O(n²l) once for the base +frequencies, then O(n²r) for the r candidate distance matrices, then +O(n³r) worst case for building r trees and evaluating Q on each +(same order as the tree-building step itself, so "free" relative to it).

    +

    Central finding, the opposite direction from an initial misreading in +this discussion: α_opt (Guindon-Gascuel's topology-optimal value) is +always ≥ the true α, not the reverse — underestimating rate +heterogeneity (i.e., picking a larger α than reality) reduces the variance +of distance estimates and thereby improves NJ/BIONJ topological accuracy, +especially when the molecular clock roughly holds (where α_opt → ∞, i.e. +no correction at all is topologically best). This is an empirical +regularity observed across their tested conditions (η ∈ {0.5, 2.0}, true +α ∈ {0.1, 0.7}, 20-taxon trees) — not a theorem — so treating it as a hard +guarantee would be another unverified-claim mistake of the kind this whole +discussion has been correcting.

    +

    Proposed integration (not yet implemented): since α_true ≤ α_opt +empirically, PartitionDispersion::estimate_alpha()'s output could seed +the lower bound of Guindon-Gascuel's grid (search [α̂, ∞) instead of +[0.1, 5000]), narrowing the grid substantially and reusing the existing +estimator rather than needing an independent "true α" source — this also +sidesteps Gu & Zhang's tree-first dependency problem entirely, since +Guindon-Gascuel's own grid already builds a tree per candidate. Still +requires: (a) a BIONJ implementation (see below), (b) the Q-criterion +subtree-grouping logic, (c) a grid-search driver. Not started.

    +

    BIONJ availability in the Rust ecosystem — checked 2026-09-11, none found. +Searched crates.io directly (API query for "bionj": 0 results) and read the +docs.rs pages of the most plausible candidates: phylotree (builds/reads/ +manipulates trees but has no distance-matrix reconstruction — no NJ, no +BIONJ, no UPGMA despite web-search summaries claiming otherwise, corrected +after checking the actual docs), speedytree (NJ only — Canonical/RapidNJ/ +Hybrid variants, explicitly no BIONJ), nj (plain NJ only), phylo (no +distance-based reconstruction at all). obikphylo::neighbor_joining +(tree.rs) is plain NJ, not BIONJ — BIONJ (Gascuel 1997) differs only in +how branch lengths are weighted during agglomeration (accounts for +distance-estimate variance, not just the sum), a modest delta over an +existing NJ implementation rather than a rewrite, but would need to be +written in-house if the Guindon-Gascuel integration above is ever pursued +(their own paper notes other tree-building methods gave similar results in +their simulations, so plain NJ may be an acceptable substitute if BIONJ +proper is judged not worth the effort).

    +

    Sharing one sample across algorithms (--sankoff/--tnt/--phyg/--iqtree + snp-* --distance) — implemented 2026-09-11

    +

    Correctness bug, not just a performance one. --sankoff/--tnt/--phyg/ +--iqtree (via SiblingExt::sankoff_bundle) and a snp-* --distance +(via SiblingExt::snp_distance) always consume the exact same selection +parameters when requested together in one obikmer phylo invocation +(n/free_loss/no_ambiguity/excluded-set/entropy_bias — the CLI has +no way to give them different values in one run, see cmd/phylo/args.rs). +Before this fix, each independently called sample_index, and +sample_layer's rand::rng() (subsample.rs:235) is a thread-local +generator that advances across calls, not reseeded each time — so the +second call silently drew a different random sample of sites than the +first, even with identical parameters. Verified empirically (throwaway +8-genome index): running snp-jc alone vs. combined with --sankoff +produced visibly different distance matrices from the same index/params. +This defeated the actual point of combining these flags — running several +algorithms (Sankoff/TNT/PhyG/IQ-TREE calibration, a snp-* distance) on +one identical site selection for direct comparison.

    +

    Fix: SankoffBundle (sankoff.rs) now retains its PairwiseTally/ +PartitionDispersion internally (previously consumed into raw/ +base_pair_tally/cardinality_tally and dropped) and exposes +SankoffBundle::snp_distance(kind, gamma_shape), which computes the +matrix from that same already-sampled tally — no second sample_index +call. The shared post-sampling logic (the --gamma-shape support check, +alpha resolution, final matrix build — previously all inline in the +snp_distance() free function) was factored into +pub(crate) fn distance_matrix(tally, dispersion, kind, gamma_shape) in +snp_distance.rs, called by both the standalone snp_distance() (after +its own fresh sampling) and SankoffBundle::snp_distance (reusing the +bundle's). cmd/phylo/mod.rs keeps the Option<SankoffBundle> alive past +the Sankoff-family if block and, when a snp-* --distance is also +requested, calls bundle.snp_distance(...) instead of +cache.snp_distance(...) whenever a bundle was built — since both branches +are driven by the same args.* fields, the parameters trivially always +match when both fire; no runtime "do the params match" check needed. +Verified end-to-end: the reused path logs (reusing the Sankoff bundle's +sample) and the snp_distance stage timer reads 0ms (formula(kind) +is O(n²) post-processing, no I/O), vs. ~150-230ms for a fresh sample on +the same tiny test index.

    +

    Explicit --session (implemented 2026-09-12)

    +

    The in-process fix above only covered reuse within one command. +--session DIR (cmd/phylo/args.rs) names a directory (outside the +index) that persists the snp-* --distance sample — the +PairwiseTally/PartitionDispersion pair sample_index would otherwise +rebuild from scratch — across separate obikmer phylo invocations.

    +

    Crate placement, as planned: a new obiksession crate +(src/obiksession/), domain-agnostic — no PairwiseTally/"site" concept +anywhere in it. Session::open(dir, params: &[u8], force: bool) handles +directory lifecycle and the params-conflict check against an opaque byte +blob; Session::store(name, bytes)/Session::restore(name) -> Option<Mmap> +handle a checksummed artifact cache. obikphylo +(siblings/algorithms/session_cache.rs, shared by snp_distance.rs and +sankoff.rs — see below) is the consumer: it serializes its own +PairwiseTally/PartitionDispersion/SnpAlignment to bytes (via rkyv) +and hands them to obiksession, which never sees their type.

    +

    Serialization: rkyv 0.8.18, added as planned. One deviation from the +original zero-copy pitch, scoped down deliberately: artifacts are restored +via rkyv::from_bytes (full owned deserialize) rather than rkyv::access +(zero-copy over the mmap). True zero-copy would require every +PairwiseTally query method (categories, base_freq, pair, ...) to +work generically over Archived<PairwiseTally> as well as the owned type +— a separate, larger change not needed to get the actual win (skipping +sample_index's expensive re-scan of the sibling annex; deserializing an +already-in-memory-sized count array is comparatively cheap CPU, not I/O). +#[derive(Archive, Serialize, Deserialize)] was added to PairStats, +PairwiseTally, and PartitionDispersion (pairwise.rs) for this.

    +

    Startup behavior, as designed: Session::open creates DIR if +missing (fresh, params recorded); if DIR already holds different saved +params, cmd/phylo reports a hard error and exits rather than silently +preferring one side, unless --session-force is given (which discards the +directory's cached artifacts and starts over under the new params) — +implements exactly the "explicit escape hatch, not silent override" +decision from the design discussion. obiksession itself doesn't decode +either side's bytes to produce a field-by-field diff (true to "opaque +blob," left to whoever needs it); cmd/phylo's conflict message is +currently a single generic sentence listing which flags could be the +cause, not a computed diff — an honest v1 simplification, not a limitation +of obiksession itself.

    +

    Per-artifact caching, as designed: only PairwiseTally/ +PartitionDispersion are cached (snp_distance.rs's restore_tally/ +store_tally) — ratio_ceiling/gamma_shape remain pure post-processing +over whatever tally is in hand, cached or fresh, exactly as planned.

    +

    Locking, reconsidered and kept as the whole-session DirLock. First +written up as a corner cut ("per-entry locking was sketched, not +implemented"). On actually working through the concrete race, whole-session +locking turned out to be the correct choice, not a simplification: +--session-force resets a session by deleting its whole artifacts/ +directory. If the directory lock were only held briefly (during +Session::open, then released so a long-running Session doesn't block +other processes from touching different artifacts), a second process +calling open(..., force: true) could acquire the lock, see mismatched +params, and wipe artifacts/ while a first process's already-open +Session is mid-checkpoint — genuine corruption risk, not just wasted +work. Holding the lock for the Session's entire lifetime makes that +sequence impossible: a force reset must wait for every currently-open +Session on that directory to be dropped first. The concurrency this +gives up (two different processes reading/writing different artifacts +of the same session at once) isn't exercised by anything today — only one +code path (snp_distance/sankoff_bundle, sequentially per invocation) +uses sessions at all.

    +

    Per-layer chunked/resumable dumps — implemented. subsample::sample_index +gained a skip_layers: usize parameter (skip the first N (partition, +layer) pairs, in its own deterministic iteration order, entirely — no +I/O, no on_layer call) and its on_layer callback gained a raw_index +argument (that pair's 0-based position in the same order), so a caller can +tell exactly how far a run has gotten. snp_distance/sankoff_bundle +checkpoint their tally/dispersion (and, for sankoff_bundle, the +in-progress SnpAlignment) to the session every +session_cache::CHECKPOINT_INTERVAL_LAYERS (8) layers or +CHECKPOINT_INTERVAL (30s) of wall time, whichever comes first — bounding +how much completed scanning a crash can lose, without paying a full +tally-rewrite on every one of what can be thousands of layers. A progress +artifact records how many layers are reflected in the checkpoint, with a +PROGRESS_COMPLETE (u64::MAX) sentinel once every layer has been +processed (distinct from any real layer count, so "finished" is never +confused with "stopped after N layers where N happens to equal the total").

    +

    Verified with a real kill: built a 10-genome/64-partition index, ran +--distance snp-jc --session DIR under timeout 0.4, confirmed a +checkpoint at progress=8 (exactly CHECKPOINT_INTERVAL_LAYERS) was on +disk, then re-ran the same command and confirmed it resumed from layer 8 +and completed successfully — 0-effort verification that the mechanism +itself is real, not just plausible-sounding.

    +

    Important correction from that same test — resume is not bit-reproducible, +and shouldn't be. The output matrix from the resumed run differed from +an uninterrupted control run against the same index/params. Root cause: +sample_layer's random draws come from rand::rng(), a per-process, +OS-seeded generator with no continuity across a process boundary — the +layers processed before a kill and the layers processed after a resume +come from two independent random streams. An earlier revision of this doc +(and of the code comments) claimed resume "reproduces the exact same +sample an uninterrupted run would have" — that's wrong, and was corrected +after the user pointed out the actual reasoning error: true randomness has +no notion of continuity in the first place, so there was never a +correctness requirement for a resumed run to match an uninterrupted one +bit-for-bit. The sample produced after a resume is exactly as legitimate a +random draw as one from an uninterrupted run — just a different one, +same as running the command twice without --session already gives two +different samples today. That variance is explicitly wanted (see below), +not a defect to fix. Comments in subsample.rs/session_cache.rs were +corrected to state this plainly instead of the false reproducibility claim.

    +

    Explicitly declined: deterministic/seeded sampling. Raised as a +possible fix for the above (seed sample_layer's RNG from a per-session +value plus (partition, layer), making a resume reproduce bit-identical +results, and, if applied everywhere, making any two runs with identical +parameters produce identical samples). The user explicitly declined this +for now: repeated runs are relied on to measure sampling variance, and a +--seed flag would need its own explicit design if wanted later — this is +not an accidental gap, it's a stated preference to revisit only if asked.

    +

    SankoffBundle integration — implemented. sankoff_bundle accepts +the same session: Option<&Session> and uses the same artifact names +(session_cache::{TALLY_ARTIFACT, DISPERSION_ARTIFACT, PROGRESS_ARTIFACT}) +as snp_distance, plus its own ALIGNMENT_ARTIFACT (sankoff.rs) folded +in lockstep with the tally at every checkpoint (same on_layer call +produces both, so they can never drift out of sync). Consequence verified +directly: a tally checkpointed by --sankoff --session DIR is restored by +a later, separate --distance snp-k2p --session DIR invocation with no +--sankoff at all, and vice versa — cross-consumer reuse, not just +same-command reuse, exactly the synergy this design aimed for.

    +

    Per-artifact caching, as designed: only the tally/dispersion (plus, +for sankoff_bundle, the alignment) are cached — ratio_ceiling/ +gamma_shape remain pure post-processing over whatever tally is in hand, +cached or fresh, exactly as planned.

    +

    Still not done — genuine follow-up, not a justified tradeoff: +exhaustive-sample caching for --subsample-varying exploration. The +design discussion's suggestion — cache the exhaustive (n = None) sample +once and subsample from it on each run instead of one session per exact +n — wasn't implemented; a session today is still scoped to one exact +parameter tuple, --subsample included.

    Output format: PHYLIP-relaxed by default for the distance matrix

    Implemented. The primary distance-matrix output (_dist.csv today) gains multiple formats: PHYLIP-relaxed becomes the diff --git a/DevDoc/theory/indexing.refs/index.html b/DevDoc/theory/indexing.refs/index.html index 890844ab..8dcb615c 100644 --- a/DevDoc/theory/indexing.refs/index.html +++ b/DevDoc/theory/indexing.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/theory/indexing/index.html b/DevDoc/theory/indexing/index.html index afbd21e6..be5ea79a 100644 --- a/DevDoc/theory/indexing/index.html +++ b/DevDoc/theory/indexing/index.html @@ -1199,7 +1199,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/theory/minimizer.refs/index.html b/DevDoc/theory/minimizer.refs/index.html index efa795a8..26fbf103 100644 --- a/DevDoc/theory/minimizer.refs/index.html +++ b/DevDoc/theory/minimizer.refs/index.html @@ -1124,7 +1124,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDoc/theory/minimizer/index.html b/DevDoc/theory/minimizer/index.html index f7d4cdc0..b789b4f2 100644 --- a/DevDoc/theory/minimizer/index.html +++ b/DevDoc/theory/minimizer/index.html @@ -1243,7 +1243,7 @@ - Sibling annex (discussion) + Sibling annex diff --git a/DevDocMD/architecture/siblings.md b/DevDocMD/architecture/siblings.md index 352956f4..a9e32377 100644 --- a/DevDocMD/architecture/siblings.md +++ b/DevDocMD/architecture/siblings.md @@ -1,1024 +1,160 @@ -# Sibling annex — architecture (discussion) - -Status: architecture decided (2026-08-14). Implementation not yet mandated. - -## Two index spaces, uncorrelated - -Every kmer stored in a `Layer` lives in two independent index spaces: - -- **Iteration order**: its position when enumerating `unitigs.bin` (the - superkmer file), deterministic but arbitrary with respect to slot. -- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns. - -The two are not correlated by any formula. Converting from one to the other -requires either recomputing the MPHF (kmer → slot) or scanning the iteration -stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a -one-way function, not an invertible bijection with a stored inverse. Any -method that reconstructs a kmer from a bare slot number is wrong by -construction, regardless of the mechanism used (MPHF re-hash, or evidence -decode + direct unitig read). See `MphfLayer::kmer_at` -(`obikindex/src/layer/mphf_layer.rs`) — flagged for removal, currently called -from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see -"Pending work" status below). - -## Two pipelines, never mixed - -| | origin of the kmer | membership known? | correct mapping | -|---|---|---|---| -| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check | -| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence | - -Evidence exists solely to answer "is this external kmer a member of the -layer" for the query pipeline. Using it (or the MPHF) to go the other way — -recover a kmer from a slot, or re-verify a kmer that was just produced by -iterating the layer — is a conceptual error: evidence can be probabilistic -(`Approx` mode), so any slot→kmer attempt is unsound in general, and -pointless even in `Exact`/`Hybrid` mode since the kmer was already known. - -## Sibling annex: an iteration-pipeline artifact only - -The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`, -`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a -family minorant and which family members are present in the index. Its only -consumers (`obikphylo/siblings/stats.rs`, `family_scan.rs`) enumerate it -exhaustively (`0..annex.len()`); no query-pipeline code path touches it. - -**Decision**: the annex must be persisted in iteration order, not slot -order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex -file directly — one linear, cache-friendly pass, no MPHF/slot indirection, -no `kmer_at`. It also enables specialized iterators building on this zip: -minorants-only iteration, batch-of-kmers → batch-of-family-members, etc. - -Today the annex is built and stored in **slot** order -(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated -via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is -threaded through the whole cross-partition reconciliation pipeline (variant -generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This -must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading -the **iteration index** instead of the slot end to end — eliminating -`kmer_at` from the build path entirely, not just the read path. No -slot-indexed intermediate is needed even during construction; the -iteration-order id is sufficient throughout. - -The cross-partition side of the same pipeline is unaffected: checking -whether a generated family-variant kmer exists in another partition is a -genuine query-pipeline operation (the variant's membership in the *target* -partition is unknown) and must keep going through -`KmerPartition::query_partition_with` (MPHF + evidence), never a raw -`index()`. - -## Pending work — done - -The plan above shipped: `obikphylo` (a new crate — phylo-domain extension -traits over `obikindex::KmerIndex`/`obikindex::layer::Layer`, replacing the -old `obikindex::siblings` module) builds and reads the annex purely in -iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with -batch variants, mirroring `Layer`'s own `KmerIter`/`KmerBatchIter` -shape). `MphfLayer::kmer_at` has no remaining callers. - -A separate, unrelated bug surfaced during this work and was fixed -(2026-08-14): `MphfLayer::enumerate_kmers_batch` computed its -`batch_start_index` via the stdlib `.enumerate()` adapter, which counts -*batches* (0, 1, 2…), not the cumulative k-mer offset the annex is actually -keyed on — every batch past the first wrote its mask/annex entries at the -wrong iteration-order position. Fixed by tracking a running offset instead; -regression tests added (`sibling_annex_no_empty_masks_after_build`, -`sibling_histogram_does_not_panic_on_partial_last_batch`). - -## Performance: `build_sibling_annex` parallelism (2026-08-14) - -Investigated on a real multi-genome run (`phyloskims_sal_vac`, k=31/m=11). -Baseline: mostly one active core, with short multi-core bursts — average -~3 cores. - -**Fixes that helped, kept:** - -- `CanonicalKmerOf::minimizer()` (`obikseq/src/kmer.rs`) — a direct O(k) - bit-arithmetic minimiser for a single isolated k-mer, replacing a - `RollingStat` instance fed byte-by-byte through an ASCII round-trip (used - by `helpers::partition_of`, called for every generated family variant). - ~3x wall-clock improvement on its own, confirmed by sampling - (`obiskbuilder::rolling_stat`/`obikentropy` frames disappeared from the - hot path). `CanonicalKmerOf::partition()` added alongside it (wraps - `minimizer().seq_hash() & mask`, the same routing rule - `KmerPartition`/`RoutableSuperKmer` use). -- Cross-partition resolution (`outgoing.par_iter()` in - `build_layer_sibling_annex`) parallelised at the *partition* level — one - Rayon task per non-empty `outgoing[dest]` bucket. For k=31/m=11, a - central-base substitution changes the winning minimiser (and thus the - destination partition) only when that window overlaps the central base: - ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants - route right back to the partition already being built. That self bucket - ends up far larger than any other, so the per-partition split pinned one - thread to it alone while the rest of the pool finished instantly — - confirmed by sampling: one thread solid in `MphfLayer::find`, everyone - else idle. Fixed by splitting each non-empty bucket into - `total_queries / n_workers` (capped 4096) chunks *before* `par_iter()`, - preserving per-partition mmap locality (each chunk stays contiguous - within one partition) while letting Rayon spread an oversized bucket - across several threads. Net effect of both fixes together: ~3 cores - average → ~10-13 cores average on the same run, and a projected total - build time of ~1h15 down to ~30min on the real `phyloskims_sal_vac` run - this was measured against. -- `TracedBar`'s ETA (`obisys/src/progress.rs`) was silently starved: the - custom progress message and the self-computed ETA text used to share one - `pb.set_message()` slot, with the ETA holding off for 2s after any custom - message — fine when custom messages are rare, broken once - `build_sibling_annex`'s per-partition callback fires more often than - that. Fixed by keeping the two texts in separate fields, composed - together on every render instead of one overwriting the other. - -**Tried and reverted — do not repeat blindly:** - -- Parallelising the *outer* partition loop in `build_sibling_annex` with - `obikindex::PartitionRunner` (already used by `merge`/`build_layers`), - splitting a fixed core budget between outer (partition) and inner - (pipeline + resolution) concurrency so their product wouldn't exceed the - budget. Measured *worse*: throughput dropped over time (26 - partitions/5min → 38/11-12min) and peak resolution concurrency fell from - ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which - scales very well on its own — to make room for outer concurrency, and - running several partitions' resolution at once scatters access across - multiple partitions' mmap regions at once, working against the - locality `outgoing`'s per-partition grouping exists for. `PartitionRunner` - stayed exported from `obikindex` (`new_capped` too) since it's - general-purpose, but nothing in `obikphylo` calls it. -- Splitting resolution chunks even finer (`/(n_workers*8)`, cap 1024, - instead of `/n_workers`, cap 4096) to smooth the residual sawtooth. - Measured ~10% *slower*, wider dips, not narrower. Reverted to the - original chunk sizing. - -**Known remaining limitation, not yet worth fixing:** within one layer, the -four stages (sequential `unitigs.bin` read → parallel generation → -parallel resolution → sequential annex write) never overlap — confirmed by -1s-interval sampling: generation alone occupies ~17 threads evenly, but the -next layer's read/generation never starts until the current layer's -resolution and write are both done. This produces a real, periodic (~layer -duration) alternation between "many cores" and "few cores" that neither of -the fixes above touches, since both operate *within* one layer's resolution -step. The only remaining lever is overlapping consecutive layers (e.g. a -depth-2 pipeline: start layer N+1's read/generation while layer N's -resolution/write is still running) — a real restructuring, not a parameter -tweak, and explicitly *not* to be combined with the reverted -budget-capping idea above (let each phase use however many cores it -naturally wants; only the *scheduling* needs to overlap). Deferred, not -started. - -## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14) - -`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`, -`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already -implements most of a dispatch/accumulate/resolve pipeline: generation -(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and -buckets cross-partition queries) runs on an `obipipeline::throttle` + -`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter` -*across partitions*, one generated batch resolved at a time, never several -concurrently — this ordering is deliberate, see the module's own docs on a -reverted concurrent-batch-resolution attempt that scattered mmap access). -The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`) -already exists: `n_layers <= 7` (checked once from the first non-empty -partition's `PartitionMeta::n_layers`, documented as identical across every -partition of an index — a structural, build-time property, never a -per-partition state) decides whether `FamilyMask`'s recorded `layer_value` -can be trusted to skip straight to the right layer -(`find_presence_batch_fast`) or must fall back to scanning every layer of -the destination partition (`find_presence_batch`). - -**Real gap, confirmed not implemented**: resolution is triggered by the -*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the -scanned layer), not by an *output* accumulator filling up. Since most -central-base variants of a family route back to the same partition being -scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s -`outgoing[dest]` is large for the local/self partition and thin for the -other ~255 (or however many) destination partitions — each of those gets -resolved at low query density every batch instead of being accumulated -across several source batches until resolving it is worthwhile. This is -distinct from, and not fixed by, the fast/slow layer gate above. - -Redesign sketched (not built): per-destination accumulators decoupled from -`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch -completion — a "hot" accumulator for the partition being scanned (sharded -one-per-generation-worker, no lock, since all `n_workers` pipeline workers -write to it concurrently — this differs from an earlier, simpler mental -model of "one thread owns one layer's local collector," which doesn't hold -here since `n_workers` threads cooperate on scanning *one* layer at a time, -not one thread per layer) and "cold" mutex-per-partition accumulators for -the rest, low contention expected since traffic to any single cold -destination is a small fraction of total. - -This breaks the current strict-iteration-order delivery of `on_family` -(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH` -resolves atomically). With cross-batch accumulation, a family only becomes -complete once *every* accumulator holding one of its outgoing queries has -flushed, at unpredictable, independent times — no longer streamable -strictly in order without a large, unbounded pending buffer. Resolution -sketched: replace order-dependent consumers with coordinate-addressed -writes instead of order-dependent appends (see `PseudoAlignment` idea -below) wherever possible, since `sibling_annex_stats`'s reduction (plain -counts) is already order-independent and needs nothing here. - -**Superseded 2026-08-15** by the `--subsample`/`--shannon` design below, -which sidesteps the accumulator redesign for now: bounding the number of -families actually resolved per layer (via sampling) keeps per-layer -resolution volume small enough that the batch-density problem above stops -mattering in practice for these two consumers. The accumulator redesign -remains relevant for a future *unsampled, full-index* run, but is not -required to ship `--subsample`/`--shannon`. - -## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15) - -The reference run (`phyloskims_sal_vac`-scale bacterial test set, -`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites -(4.8 MB) — trivially small. The in-progress plant index is expected to -carry on the order of 9 billion minorant families; a dense byte-per-cell -alignment at that column count is unbuildable regardless of genome count -(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000– -8,000 genomes on a large machine, which makes the per-cell cost dominant in -the other dimension too. Pruning the retained family set before -materializing anything is mandatory, not an optimization. - -**Already free**: `family_size() < 2` (no sibling variant registered at -all) is a zero-cost structural filter, read directly off `FamilyMask` bits, -already applied in `snp_pseudo_alignment`. Insufficient alone — per the -~80-85% mono-family estimate from earlier discussion, this only brings 9 -billion down to roughly 1.3-1.8 billion, still unusable. - -**Entropy definition — settled 2026-08-15, correcting an earlier wrong -turn.** The project does **not** encode families as IUPAC ambiguity codes -interpreted the classical way (Fitch-parsimony subset-compatibility, or -ML's "one true state, uncertain which"); see -`docmd/theory/evolutionary_distances.md` ("Why the IUPAC/DNA encoding used -for the first `--snp` test was wrong") and the Sankoff resolution that -followed it. The real model is a genuine 16-state alphabet (the powerset of -`{A,C,G,T}`, `∅` included as a real state) scored with a *calibrated -pairwise cost matrix* (`obikphylo::cardcomp::pairwise_cost_matrix`, -`cmd/phylo/sankoff.rs`), not a compatibility/subset relation between -states. Under that model, each of the 16 states — including multi-bit ones -like `AC` — is a first-class, independently-costed state, not an -uncertainty encoding of a single true base. So: **entropy over the 15 -non-empty states (`∅` excluded, matching the earlier decision to exclude -genomes where the family is absent) is the correct informativeness -measure** for this project — not a 4-symbol reduction, which would discard -exactly the cardinality/composition information the calibrated cost matrix -is built to exploit. - -## `--subsample` / `--shannon` — sampling strategy (decided 2026-08-15) - -Goal: make both the pseudo-alignment (`--snp`) and a Shannon-entropy -diagnostic usable at any index scale, from the 13-genome bacterial -reference run up to the 9-billion-family plant index, without requiring the -batched-accumulator redesign above. - -**`--subsample N`** (integer, families to retain): bounds the pseudo-alignment -to `N` minorant families, sampled **proportionally per layer** among -non-monomorphic minorants (`family_size >= 2`) — this sidesteps the need -for a true global reservoir merge across layers while still approximating a -uniform sample over the whole index, and directly answers the earlier open -question of global-vs-per-layer selection scope. - -Three passes, in order: - -1. **Global count** (cheap, structural, parallel across layers — same shape - as the existing `sibling_family_size_histogram`, extended to report a - **per-layer** breakdown rather than one index-wide aggregate): for each - layer, `count_layer` = number of non-monomorphic minorants. Gives - `total_count = Σ count_layer`. -2. **Per-layer proportional reservoir sampling** (cheap, structural, one - pass per layer, no cross-partition resolution): `N_layer = round(N × - count_layer / total_count)`. Since `N_layer` is a proportion of - `count_layer`, it can never exceed it as long as `N <= total_count` — the - one edge case is `total_count <= N`, in which case sampling is skipped - entirely and *every* non-monomorphic minorant of every layer is kept - (no reservoir needed, `N` was never a real constraint). Otherwise: - Algorithm-R reservoir sampling over the layer's non-monomorphic minorant - indices, producing `N_layer` iteration-order indices directly, no - intermediate full list ever materialized. -3. **Filtered resolution** (the expensive step, the existing - `scan_layer_families` engine, unchanged): re-scan the layer, generating - and resolving cross-partition queries **only** for the indices selected - in step 2 (cheap membership test against a small per-layer index set) — - this is what keeps `--subsample` cheap even on an unsampled-scale index, - since the cross-partition resolution volume is bounded by `N`, not by - the layer's true size. - -Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir -sampling would waste step-3's expensive resolution work on candidates later -evicted by the reservoir. Step 1 must fully complete (every layer) before -step 2 can start for any layer, since `total_count` is a global quantity. - -**`--shannon`** (no argument): emits a CSV of per-family Shannon entropy -(15 non-empty states, `∅`/absent genomes excluded from the denominator, per -the settled definition above). Independent of `--subsample` — entropy is -computed and written per family as soon as its `genome_mask` resolves, -O(1) memory per family, so it streams fine even unsampled at full index -scale (a time cost, not a memory one). Combined with `--subsample N`, it -delivers the original exploratory diagnostic (e.g. `--subsample 1000000 ---shannon`) directly from this general machinery, rather than a -purpose-built one-off script. - -Validated end-to-end (2026-08-15) against real data: `--sibling-hist` on -`phyloskims_sal_vac` (91 real genomes, k=31/m=11, 256 partitions × 2 -layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238 -total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier -guess, corrected here). `--subsample`/`--shannon` on the smaller 20-genome -bacterial reference (`benchmark/global_index_presence`) produced a sample -size within rounding of the request (99,742/100,000) and a -[0.8,1.2)-bucket share (40.7%) matching the full unsampled population -(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a -real bug in `reservoir_sample_layer` (see next section), now fixed. - -**Bug found and fixed (2026-08-15): `family_idx` numbering mismatch.** -`scan_layer_families`'s `family_idx` counts *every minorant* of a layer -(monomorphic ones included, since `iter_minorants_batch` filters only on -`is_minorant()`), not the raw annex slot (`SiblingAnnex::get(slot)` spans -every k-mer, minorant or not) and not a counter over non-monomorphic -minorants alone. `subsample.rs`'s `reservoir_sample_layer` originally -stored raw slot numbers in its `HashSet` selection, which drifts -away from `family_idx` as soon as *any* monomorphic minorant is seen — -i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by -tracking two separate counters: `family_idx` (every minorant, matching -`scan_layer_families`) and `seen` (non-monomorphic minorants only, what -Algorithm R actually samples over) — only `family_idx` values are ever -stored in the selection set. The existing unit test never exercised this -(its fixture has exactly one non-monomorphic family, always hitting the -"keep everything" shortcut) — a stronger fixture with several interleaved -monomorphic/non-monomorphic families would be needed to catch a regression -here automatically; not yet written. - -## Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15) - -Motivation: on real data (bacterial reference, full unsampled run), only -~5‰ of non-monomorphic minorants fall in the `[0.5, 1.5]` bit band judged -phylogenetically interesting (entropy too low = uninformative near-invariant -site; too high = saturated/noisy, see `family_entropy`'s 15-state -discussion) — roughly 1 in 10,000 minorants overall. Computing exact -entropy for every candidate just to discard 99.99% of them is wasteful at -the full 9-billion scale. - -**The idea**: `PersistentBitMatrix::col_view(c)` gives a genome's whole -presence column as a `BitSliceView` (sequential, no MPHF, no cross-partition -routing — purely local to one layer's own matrix). Accumulating -`TempCompactIntVecBuilder::inc_present(col)` (already exists in -`obicompactvec/src/builder.rs:121`, along with `add`/`min`/`max`/`diff` on -`IntSliceView` — no new low-level API needed) over every column of a layer -produces `coverage[slot]`: how many genomes carry each exact k-mer, in one -sequential per-layer pass, entirely decoupled from family/sibling -structure. Persisted once per layer, a family's members' coverage could -then be looked up via a plain local MPHF `index()` (cheap) instead of a -full cross-partition presence resolution (`find_presence_batch`) — the -expensive part today is specifically the cross-partition/cross-layer -routing to a sibling's own matrix, not the bit-reading itself, and -`coverage[slot]` sidesteps that routing entirely by moving the cost into a -one-time, purely local, embarrassingly-parallel build step. - -**Why not implemented**: `coverage[slot]` is a per-member marginal — -summing members' coverages to approximate a family's entropy silently -assumes no genome carries more than one member at once. It cannot -represent or detect joint co-occurrence (a genome carrying both `A` and -`C` at once, i.e. a combined 15-symbol state) at all, which is exactly the -phenomenon `family_entropy`'s 15-state definition exists to capture (see -`CardinalityTally`/`cardinality_transition_probs`, the project's own -existing machinery for this same co-occurrence structure, built for the -Sankoff matrix calibration). A family that is in truth uniformly `AC` -across every carrying genome would look like a well-balanced 2-state split -under the marginal approximation (entropy ≈ 1) while its true 15-state -entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the -"saturated, uninformative" tail this pre-filter would need to catch, -undermining the point. It stays plausible as a coarse filter for the -*low* tail only (a dominant single member's marginal share reliably -predicts low true entropy too), but not as a stand-in for the high tail — -not pursued further for now. - -## `--free-loss`/`--tnt` pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below) - -Measured on `phyloskims_sal_vac` (91 genomes): `obikmer phylo --subsample 500000 ---free-loss --tnt` logs four sequential stages — -`raw_snp_distance` (1413s), `base_pair_tally` (1456s), -`cardinality_tally` (2078s), `snp_pseudo_alignment` (143s). Reading the -code (`obikmer/src/cmd/phylo/mod.rs:210-242`, -`obikphylo/src/siblings/distance.rs`, `cardinality.rs`, `alignment.rs`) -surfaced two compounding problems, not one: - -1. **Four separate full scans of the annex**, each opening its own - `KmerPartition`/`PartitionCache` and calling `scan_family_pairs`/ - `scan_layer_families` independently — nothing computed in one stage is - reused by another. `base_pair_tally` is explicitly documented - (`distance.rs:138-143`) as a second full pass over the same - traversal `raw_snp_distance` already did, needed only because - `raw_snp_distance` doesn't keep the resolved bases, only aggregate - counts. `cardinality_tally` and `snp_pseudo_alignment` are each a - third and fourth independent full pass. Per the module's own earlier - profiling note (`family_scan.rs:26-28`, cited already above), this - traversal is page-fault/mmap-bound, not compute-bound — the ~10-12% - CPU efficiency ("contention" status) observed on these three slow - stages is consistent with I/O stalls scaled by repeated full scans, - not lock contention (there are no `Mutex`/`RwLock` anywhere in - `siblings/*.rs`; shared writes use per-slot `AtomicU8::fetch_or`). -2. **Worse: `raw_snp_distance` and `cardinality_tally` don't honor - `--subsample` at all** — both call `scan_layer_families` with - `Selection::All` hardcoded, and `args.subsample` isn't even threaded - into their function signatures (`mod.rs:212`, `mod.rs:226`). Only - `snp_pseudo_alignment(args.subsample)` builds a real reservoir-sampled - `Selection::Some(set)` (via `compute_selections`, `alignment.rs:89`). - So today, `--subsample 500000` only bounds the pseudo-alignment step — - the SNP-distance matrix, the Sankoff base-pair calibration, and the - cardinality histogram are always computed over the **full, unsampled** - index regardless of the flag. This is not merely "different subsamples - per stage" (which would already be a problem worth fixing) — it's that - three of the four stages never subsample, which explains most of the - ~10x runtime gap against `snp_pseudo_alignment` on its own. - -**Decided requirement**: all four stages consume **one shared selection**, -computed once, not each stage either scanning everything or drawing its -own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair -calibration, the cardinality histogram, and the pseudo-alignment all -describe the same set of families — the calibration and the alignment it -calibrates are now guaranteed to agree on which sites exist. - -**Correction to the "single pass" framing above**: `base_pair_tally`/ -`cardinality_tally` both need `raw_snp_distance`'s *complete* aggregate -SNP/shared counts before they can derive `included[i,j]` (the -`ratio_ceiling` filter) — a genuine sequential dependency (`included` -can't be known until every pair's aggregate count is final), not an -artifact of the old code's structure. So the fix is **two** passes over -the shared selection, not one: pass A computes the aggregate counts (and -derives `included`); pass B fuses `base_pair_tally` + `cardinality_tally` -+ the pseudo-alignment (mutually independent once `included` is known) -into a single scan. Still a 4→2 reduction, and — per the clarification -that settled this — pass A itself now runs over the *same shared -selection* pass B uses (not the full unsampled index): "les ratios, on -les fait sur les sites sélectionnés, c'est tout, les autres sites -n'existent pas" — once a selection is chosen, both passes are bounded by -it, so on a real `--subsample`/`--entropy` run pass A is cheap too, not -just pass B. - -**Entropy-biased selection's own resolution to the "forward-looking -complication"** (entropy must be known before selection, but selection -happens during the same scan that would resolve it): see "Entropy-biased -selection" below — resolved via a persisted per-layer entropy annex, not -by restructuring the scan into an inline pre-pass. - -## Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15) - -Refines the "forward-looking complication" above with a concrete -mechanism. Instead of a hard `[low, high]` entropy band (or any other -exact cutoff) deciding which non-monomorphic minorant families are -eligible, selection is weighted by an **unnormalized Gaussian kernel** -centered on a target entropy: `w(entropy) = exp(-(entropy - μ)² / (2σ²))` -— deliberately not the normalized Gaussian density (which would peak -below 1 and complicate the "probability" reading) — this kernel form -equals 1 exactly at `entropy = μ` and decays smoothly to 0 away from it, -so it reads directly as an acceptance weight: the further a family's -entropy from the target, the less likely it is picked, with no hard -in/out boundary — a few "bad" sites can still get in, by design. `μ` -(default ~1.0) and `σ` (default ~0.5) are meant to be user-tunable. - -**Mechanism: joint probability, not a weighted reservoir**. Not -Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex -proposal, superseded before implementation) — instead, a single -independent accept/reject draw per qualifying candidate: draw -`u ~ Uniform(0,1)`, accept iff `u < p₀ · w(entropy)`. `p₀` is a single -**index-wide** rate, `N / total_count` (`total_count` = the sum of -`non_monomorphic_counts` across every layer, `N` = `--subsample`'s -target), applied identically at every layer — this alone already gives -each layer its proportional share (the same effect the old uniform -reservoir's explicit per-layer `n_layer = N · count_layer / total_count` -computation achieved, but without needing to compute it: applying one -rate uniformly is mathematically the same as proportioning per layer). -`p₀ = 1.0` when there is no `--subsample` at all — `--entropy` alone is a -pure soft entropy filter over the whole index, no size target. Properties: -(1) **strictly generalizes the existing uniform sampler** — with `σ` large -enough that `w ≈ 1` everywhere, this reduces to the old uniform `N/total_count` -draw; (2) yields "approximately N", not exactly N — expected accepted -count is `N · mean(w)`, always `≤ N` — an intentional relaxation, matching -"sous-échantillonnage à environ n" rather than the old reservoir's exact-N -guarantee; (3) one streaming pass, one random draw per candidate, no -reservoir state. - -**Resolving "entropy must be known before selection, but selection -happens during the same resolving scan"**: solved with a **persisted, -per-layer entropy annex** (`obikphylo/src/siblings/entropy_annex.rs`, -`EntropyAnnex`/`EntropyAnnexBuilder`), not by restructuring the scan. -Mirrors `SiblingAnnex`'s mmap-backed, read-only-after-build convention, -but indexed by `family_idx` (every minorant of the layer, monomorphic -included — the same numbering `Selection`/`scan_layer_families` already -use), one `f32` entropy15 value per entry, `-1.0` sentinel for monomorphic/ -not-yet-computed. First use of `--entropy`/`--entropy-sd` on an index -pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`) — every -later run (any `μ`/`σ`, any command) reads the file positionally, no -re-scan, restoring the usual `Selection::Some` "skip resolving excluded -families" speedup that a naive "weigh during the resolving scan" design -would have permanently forfeited. - -**Bug found and fixed (2026-08-15): `ensure_entropy_annexes` scanned with -`Selection::All` instead of bounding to non-monomorphic minorants.** -Monomorphism (`family_size() < 2`) is knowable directly from the annex -bits alone, no per-genome resolution needed — but the original -implementation called `scan_layer_families` with `Selection::All` -anyway, so `fill_sub_matrix_carries` (the expensive per-genome -resolution) ran for *every* minorant, ~98% of which are monomorphic -(measured elsewhere in this doc) and had their `genome_mask` immediately -discarded once the callback checked `family_size() < 2`. Fixed by adding -[`subsample::non_monomorphic_selection_layer`] — a cheap, annex-only, -non-sampling pass (same shape as `reservoir_sample_layer`, but keeping -every non-monomorphic minorant's `family_idx` instead of a bounded -reservoir) — and passing `Selection::Some(&eligible)` instead of -`Selection::All`, so the expensive resolution now runs only for the ~2% -of minorants that can actually produce a real entropy value. A -`debug_assert!(mask.family_size() >= 2, ...)` inside the -`scan_layer_families` callback guards the invariant. - -**Resolved**: the existing hard "non-monomorphic minorant" eligibility -filter stays a hard gate upstream of the Gaussian weighting — only -qualifying families ever get a stored entropy value or a weighted draw. - -**CLI, implemented**: two `phylo` options, `--entropy <μ>` and -`--entropy-sd <σ>` (`obikmer/src/cmd/phylo/args.rs`). The entropic filter -activates as soon as *either* is given (`mod.rs`, computed once into an -`Option` threaded through `--snp`/`--family-overlap`/ -`--shannon`/the fused sankoff pipeline below). If active but one or both -are unset, defaults are `μ = 1.0`, `σ = 0.5`. - -## Two entropy definitions kept side by side, for comparison (2026-08-15) - -`--shannon`'s CSV carries both `entropy15` (`family_entropy` — the settled -15-non-empty-state definition, see above) and `entropy4` (`family_entropy_4` -— plain nucleotide reduction), computed from the same already-resolved -`genome_mask`, not from the marginal approximation above. A genome carrying -several bases at once contributes to *each* base's count (counted once per -base present, not fractionally split, not folded into one combined state) -— a genome polymorphic for the family is present at more than one base by -construction, so it is expected to count more than once; the denominator is -the total base-occurrence count, not the genome count (the two coincide -only when no genome carries more than one base). Kept side by side -specifically to measure, on real data, how much the two diverge — not yet -analyzed. - -## `PersistentSparseBitMatrix` — implemented and measured (2026-08-15) - -A row-major (k-mer-major), deduplicated sparse alternative to -`obicompactvec::PersistentBitMatrix`, motivated by the same sparsity that -drove `--subsample`/`--shannon` above, but pursued as a foundational -storage-layer change rather than an index-level workaround. Full design -history, rationale, and rejected alternatives (external Elias-Fano crates, -`cacheline-ef`, a single unsplit `dict_id` array) are in the dedicated -implementation plan (`vivid-mapping-tiger.md` at the time of writing — the -content below is the durable summary, not a pointer to a session-scoped -file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi, -"Compact Data Structures for Collections of Sets" (2025, -`biblio/Alanko et al. - Compact Data Structures for Collections of -Sets.pdf`) — this design implements only their exact-duplicate special -case (a plain dedup dictionary), not their full subset-containment -hierarchy. - -**Design**: four on-disk components, each mmap-backed, built once per -layer (matching how the rest of the build pipeline already works — never -the whole multi-billion-row index at once): an `is_multi` rank-capable -flag per row (singleton vs. multi-genome), a fixed-bit-width array for -singleton rows (genome index directly, `ceil(log2(n_cols))` bits), a -separate fixed-bit-width array for multi-genome rows (`dict_id`, -`ceil(log2(n_distinct_multi_sets))` bits — kept apart from the singleton -array specifically because `n_distinct_multi_sets` can be large in -absolute terms even when multi-genome rows are a small *fraction* of all -rows, and a single shared array would force every row, singletons -included, to pay the wider width), and a deduplicated dictionary of -distinct multi-genome sets (Elias-Fano-encoded byte offsets + a -varint-encoded values blob). New low-level primitives added to -`obicompactvec` to build this: `PersistentFixedIntVec` (arbitrary, -runtime-parameterized bit width, width 0 included — needed once a real -bug surfaced, see below), `PersistentRankSelectBitVec` (rank1/rank0/select1 -on top of the crate's existing `count_ones`, using -`common_traits::SelectInWord`), `EliasFano` (composes the two). A new -`BinaryMatrix` trait (`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, -`count_ones`) unifies dense and sparse at the one call site that needs -both interchangeably (`obikphylo::siblings::cache::Mat`) — column-oriented -methods (`col`, `col_view`, the `partial_*_dist_matrix` family) stay -dense-only. - -**Two real bugs caught by tests, not by inspection**: (1) `EliasFano::open` -re-derived its low-bits width from the persisted low-vector file's own -width byte; the zero-width case was built with a dummy 1-bit placeholder -(the builder rejected true width 0), so every reopened value silently -doubled. Fixed by making `PersistentFixedIntVec` genuinely support width 0 -(no storage, `get` always 0) instead of working around the limitation in -`EliasFano`. (2) An empty row (cardinality 0 — not expected on a real -built index, but not guarded against either) was recorded as a singleton -at genome 0, indistinguishable on read-back from a *real* singleton at -genome 0. Fixed by routing cardinality-0 rows through the dictionary path -(a genuine empty entry) instead of the singleton shortcut. Both caught by -`obicompactvec`'s test suite (142 tests, including disk-reopen round-trips -that drop every builder/mmap before reopening fresh), not by manual -review — worth remembering next time a "this edge case can't happen in -practice" shortcut is tempting. - -**Measured on real data** (`layer_1` of `phyloskims_sal_vac`'s -`part_00018`, 30,246,774 rows, 91 genomes — `#[ignore]`d benchmarks in -`obikphylo/src/siblings/tests.rs`): - -| | dense | sparse | ratio | -|---|---|---|---| -| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller | -| build time / peak RSS | — | 4.26s / 628MB | (per-layer, in-memory construction — comfortable) | -| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse **faster** (smaller structure, better cache fit) | -| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** (the real `--shannon`/family-lookup shape) | -| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** (no native column method — every read decodes a full row to keep one bit) | - -The row-access wins (both directions) weren't the design's stated goal — -compactness was — but turn out real: dense's genome-major layout scatters -a single row read across a much bigger file, which costs more than -sparse's rank/select/varint decode once the file is this much smaller. -The column-access cost is the flip side of the same layout choice, and is -exactly what the next item below exists to fix. - -**Next, not yet planned**: rewrite `partial_jaccard_dist_matrix`/ -`partial_hamming_dist_matrix`/etc. (`obicompactvec/src/bitmatrix/pairwise.rs`) -as a row-major co-occurrence accumulation (`O(Σ_rows k²)`, per-row -increments into an `NxN` genome-pair counter — the known alternative to -today's column-fold, plausibly cheaper on data this sparse, not just a -fallback) so `obikindex`'s `--metric`/distance-matrix path can use the -sparse type without the measured 86x column-access penalty. Needs its own -design pass (in particular how it plugs into the `BitPartials`/ -`ColumnWeights` traits so both matrix types keep serving `--metric`) -before implementation — not just "port the loop", a genuinely different -algorithm. - -Also still deferred, unchanged from the implementation plan: full Alanko -et al. subset-hierarchy compression (only the exact-duplicate special case -is built), a sparse `PersistentCompactIntMatrix` (count matrices), and -BRWT-style column-correlation exploitation. - -## Wired into `pack` and the sibling-annex build path (2026-08-15) - -`PersistentSparseBitMatrix` went from a validated but unused type to a -real, selectable on-disk format: - -- **Generic `Layer`**: `obikindex::layer::Layer`'s presence-only methods - (`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any - `D: LayerData> + BinaryMatrix`, not hardcoded to - `PersistentBitMatrix` — `PersistentSparseBitMatrix` implements - `LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were - already generic over any `D: LayerData`, so they needed no change. - Verified by `obikindex::layer`'s - `presence_layer_generic_over_sparse_matches_dense` test: build a dense - presence layer, convert it to sparse via `build_from_dense`, open both - as `Layer`/`Layer` on - the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree. - (This test must stay at `k=4` with mutually non-colliding canonical - 4-mers across its input sequences — `K`/`M` are process-wide - `AtomicUsize`s in test builds, not thread-local, so a test using a - different `k` races every other test in the same crate binary; a k=11 - version of this test passed alone but failed under the full - `obikindex::layer` suite for exactly that reason before being fixed.) -- **`obikphylo::siblings::cache::Mat`** gained a third variant, - `SparsePresence(Layer)`, alongside `Count` - and `Presence` — every method (`find_slot`, `index_batch`, - `iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches - to it identically to `Presence`, since both go through the same generic - `Layer` code. `PartitionCache::build` picks the variant per layer by - checking for `presence/is_multi.prsb` (the sparse format's own marker - file, see the design section above) before falling back to the dense - open path. -- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`): - `pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb` - first); packs to dense `matrix.pbmx` first if that hasn't happened yet - (the dense→sparse transpose needs random row access, which only the - packed/columnar dense forms give), then `build_from_dense`s the sparse - form into the same directory and deletes `matrix.pbmx` — old-format - files are removed only after the new format is fully written, mirroring - `pack_bit_matrix`'s own crash-safety convention. -- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through - `KmerIndex::pack_matrices` (all other call sites — `select`, `merge`, - `finalize_indexed` — pass `false`, unchanged dense behaviour). Count - matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix` - — see "still deferred" above). -- **End-to-end coverage**: `obikphylo::siblings::tests:: - sibling_annex_works_after_pack_sparse` builds a two-genome index, packs - it `--sparse`, asserts `is_multi.prsb` exists, then runs - `build_sibling_annex` and checks the resulting `FamilyMask`s match the - dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the - sparse format round-trips through the real build pipeline - (`PartitionCache` sparse-detection included), not just the - `obicompactvec`/`obikindex::layer` unit layers below it. - -Full workspace `cargo test` (all crates, unit + doc tests) green after -this change. - -## Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15) - -Replaces the "four independent scans" problem above and implements -"Entropy-biased selection" above, end to end: - -- **`SankoffBundleExt::sankoff_bundle`** (new, - `obikphylo/src/siblings/sankoff_bundle.rs`) — the `--sankoff`/`--tnt`/ - `--phyg`/`--iqtree` block in `obikmer/src/cmd/phylo/mod.rs` now calls - this once instead of three separate `raw_snp_distance`/ - `base_pair_tally`/`cardinality_tally` calls. One `PartitionCache`, one - shared (possibly subsampled/entropy-biased) selection computed once via - `compute_selections`, two scans over it: pass A (aggregate SNP/shared - counts, `--exclude-genome` zeroing, then `included[i,j]`), pass B - (`base_pair_tally` + `cardinality_tally` + the pseudo-alignment, fused - into one `scan_layer_families` call per layer, all three read off the - same resolved `genome_mask`). `snp_pseudo_alignment`/ - `shannon_entropy_csv` (still used standalone by `--snp`/ - `--family-overlap`/`--shannon`) both gained an `entropy_bias` parameter - too, so entropy-biased selection isn't sankoff-specific. -- **Regression proof**: `sankoff_bundle_matches_old_separate_calls` - (`obikphylo/src/siblings/tests.rs`) asserts `sankoff_bundle`'s four - outputs are bit-identical to calling the old, separate - `raw_snp_distance`/`base_pair_tally`/`cardinality_tally`/ - `snp_pseudo_alignment` on the same fixture with no subsample — the - fusion is a performance change, not a behavior change. -- **`EntropyAnnex`/`EntropyAnnexBuilder`** (new, - `obikphylo/src/siblings/entropy_annex.rs`) and **`ensure_entropy_annexes`** - (`entropy.rs`) implement the persisted-entropy mechanism from - "Entropy-biased selection" above. `entropy_annex_builds_on_demand_and_biases_selection` - (`tests.rs`) proves, on a fixture with one known-entropy family: the - annex file doesn't exist before any entropy-biased call; `compute_selections` - builds it on first use; `μ` set to the family's exact entropy with - `p₀ = 1.0` selects it deterministically (`u < 1.0` always, for - `u ∈ [0,1)`); `μ` set far away with tiny `σ` deterministically excludes - it (`w` underflows to exactly `0.0`); the persisted value matches - `--shannon`'s own `family_entropy` computation to `1e-6`. -- **`EntropyBias`** (`pub`, `obikphylo::siblings::EntropyBias { mu, sigma }`) - is the one new public type threading `--entropy`/`--entropy-sd` through - every `Option`-accepting method — resolved once in - `obikmer/src/cmd/phylo/mod.rs` from `args.entropy`/`args.entropy_sd` - (activation: either given; defaults `1.0`/`0.5` for whichever is unset). - -Full workspace `cargo test` green after this change (167 unit tests in -`obicompactvec`+`obikindex::layer`+`obikphylo` alone, plus every other -crate's suite, no regressions). - -**Still open, not part of this change** (per "Correction to the 'single -pass' framing" above): `--raw-snp-distance`/`--raw-snp-counts` (the -standalone diagnostic flags, not the `--sankoff` pipeline) still always -scan the full unsampled index — never threaded `--subsample`/`--entropy`, -out of scope here since the reported problem was specifically about the -`--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these -two standalone flags. - -## `query` never benefits from sparse row-major access (found 2026-08-19, not implemented) - -Benchmarked `obikmer query` against `global_index_presence` (dense-packed) -vs. `global_index_presence_sparse` (`pack --sparse`), 100k simulated reads -× 2 specimens (`benchmark/`, see -[benchmark_query_testing.md](../implementation/benchmark_query_testing.md)). -Correctness: 0 mismatches — sparse and dense return bit-identical query -results. Performance: sparse consistently *slower* than dense (~30-50%, -reproducible across two runs with warm disk cache), the opposite of -`pack --sparse`'s stated intent ("faster for single-row access... like -query"). - -**Root cause, read from source, not measured in isolation:** -`KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs:155-220`) -is architecturally column-major: stage 2 walks `for g in 0..n_cols { for -slot in hit_slots { layer.col_value(g, slot) } }`, documented (correctly) -as the right locality strategy for the packed/columnar formats, where -`col_value` → `PersistentBitMatrix::get` is a genuine O(1) mmap'd column -read (`persistent.rs:110-113`). - -For `Self::Sparse`, that same `get(c, slot)` (`persistent.rs:114-118`) -allocates a full `n_cols`-wide buffer and calls `fill_row` — materializing -the *entire row* — just to return one cell. Called from inside the -column-major double loop, this reconstructs the same row once per genome -column touched: O(hits × n_cols) full-row rebuilds instead of O(hits). -`PersistentSparseBitMatrix`'s own native row-major decode -(`for_each_genome_in_row`, `sparse.rs:164-177`, used correctly by its own -`row`/`fill_row`/`fill_sub_matrix`) is never reached from the query path -at all. - -**`fill_sub_matrix` (the existing `BinaryMatrix` trait primitive, -`traits.rs:13-37`) is not the right replacement for `query` either**, even -once its own dispatch bug is fixed (see next section) — its output shape -is inherently column-dense: `out[col]` gets an entry for every column, -including columns with zero hits among the requested slots. On real -sparse data (a hit typically touching a handful of genomes out of dozens) -that's still O(n_cols) output regardless of true sparsity. What `query` -actually wants is the sparse triple stream `(slot, col, value)` it already -consumes as `QueryHit::Value` — not a materialized sub-matrix. - -**Proposed primitive** (design only, not implemented — explicit ask: keep -count matrices *not excluded*, even though effort right now is -presence/absence only): - -Not a closure-driven `for_each` — a real `Iterator`, one concrete struct -per matrix format, so the traversal state (current position in the sorted -slot list, current column, permutation, sparse-row decode cursor…) lives -in named struct fields instead of being threaded implicitly through -recursion or a captured closure. RPITIT (stable since 1.75, and this -workspace is edition 2024) means the trait method can return it without -naming or boxing the concrete type: - -```rust -/// Yields every nonzero cell among `slots`, in implementation-defined order. -fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> impl Iterator + 'a; -// item: (idx into `slots`, col, value) -``` - -This is the one foundational primitive per format — both `fill_sub_matrix` -and a `for_each`-style callback become trivial, free consumers of it -(`.for_each(f)` is already `Iterator::for_each` from std; `fill_sub_matrix` -becomes "drain the iterator, scatter into `out[][]`"), instead of two -independently-maintained traversals that can silently diverge (see the bug -below — this is exactly how it happened). - -- **On `PersistentSparseBitMatrix`**: the struct is nearly free to write — - it wraps the existing (currently private) `for_each_genome_in_row` - per-row decode, advancing to the next `slots` entry on exhaustion. O(Σ - row nnz), zero `n_cols`-wide allocation. -- **On `PersistentBitMatrix::{Packed,Columnar}`**: revised — cheaper than - first thought, by reusing the same split already used for - `fill_matrix`'s own implementation instead of hand-writing a resumable - state machine at the matrix level. The base-vector layer - (`BitSliceView`, `views.rs`) already separates the two concerns: - `fill_batch_sorted` (`views.rs:55-60`, sorted-slot batch lookup) and a - genuine per-bit `Iterator` (`BitSliceIter`, `views.rs:94+`) sit side by - side there, one level below the matrix. Adding a - "positions among `sorted_slots` where the bit is set" iterator at that - same vector level is a `filter` over the existing `get()` — no new - state machine, since `std::iter::Filter` already *is* one, generated by - the compiler: - ```rust - fn nonzero_among_sorted<'s>(&'s self, sorted_slots: &'s [usize]) -> impl Iterator + 's { - sorted_slots.iter().copied().filter(move |&slot| self.get(slot)) - } - ``` - The matrix-level `nonzero_iter` then composes these per column with - `flat_map` over `0..n_cols` (each column's hits, tagged with `c`, - slot mapped back through the sort permutation `fill_batch`/ - `fill_batch_sorted` already carry) — again a combinator chain, not a - hand-rolled struct. Same algorithm, same mmap/sort locality as today's - `fill_sub_matrix`; just assembled from `std` iterator adaptors instead - of a loop body writing into a buffer, mirroring the vector/matrix split - the codebase already uses for `fill_batch_sorted` rather than - introducing a new shape. -- **`Implicit`**: trivial (`slots.iter().map(|&i| (i, 0, 1))`, one column, - always present). -- **On `PersistentCompactIntMatrix` (counts)**: same treatment as - `Packed`/`Columnar` — no sparse count format exists yet ("Explicitly - deferred" per `traits.rs:9-12`), so no native low-effort case the way - `Sparse` has one, but not excluded either: the iterator's `Item` is - already `(usize, usize, u32)`, not `bool`, specifically so presence - (`0`/`1`) and counts (arbitrary `u32`) share one primitive instead of a - bool/u32 split forcing counts out of the design. Ready for a native - sparse-count struct later without a signature change. - -Would let `query_partition_with`'s stage 2 collapse to one -`for (i, g, v) in layer.matrix().nonzero_iter(&hit_slots) { on_event(...) }` -per layer, format-agnostic, each backend's struct deciding the actual -traversal. - -**This also closes the existing dispatch bug for free, by construction**: -`PersistentBitMatrix::fill_sub_matrix` (`persistent.rs:190-215`, the enum -wrapper backing `BinaryMatrix`'s trait impl) today does *not* delegate to -`PersistentSparseBitMatrix::fill_sub_matrix` for `Self::Sparse` — it -reimplements the same naive per-(column, slot) `fill_row_bool` loop -instead, bypassing the efficient native method one file over -(`sparse.rs:249-258`). `obikphylo::siblings::cache::Mat` -(`cache.rs:138-145`) independently built its own parallel enum wrapper -that dispatches correctly — evidence this was worked around rather than -fixed at the source: two hand-written traversals for the same format, -free to drift apart, and they did. If `fill_sub_matrix` itself is -rewritten as "drain `nonzero_iter`, scatter into `out[][]`", there is only -one traversal per format left to get right — the bug class doesn't just -get fixed once, it stops being possible to reintroduce. - -## Implemented (2026-08-20) - -Built as designed above, with one deviation from the original sketch: -`nonzero_iter` ended up `Box>`, not a bare `impl -Iterator`, because `Columnar`/`Packed`/`Sparse`/`Implicit` are genuinely -different concrete types and this method isn't on a trait (kept off -`BinaryMatrix` deliberately — that trait is used as `dyn BinaryMatrix` in -`tests/sparse.rs`, and RPITIT methods aren't dyn-compatible). One `Box` -per `nonzero_iter` call, not per cell — negligible next to what it -replaces. - -- `BitSliceView::nonzero_among_sorted` / `IntSliceView::nonzero_among_sorted` - (`obicompactvec/src/views.rs`): the vector-level `filter`/`filter_map` - primitive, exactly as sketched — no new state machine, `std`'s own. -- `PersistentSparseBitMatrix::nonzero_iter` (`bitmatrix/sparse.rs`): native, - `std::iter::from_fn` over one buffered row at a time via the existing - `for_each_genome_in_row` — no `n_cols`-wide allocation, ever. -- `PersistentBitMatrix::nonzero_iter` (`bitmatrix/persistent.rs`): dispatches - to the above for `Sparse`; for `Columnar`/`Packed`, loops columns, - collects each column's `nonzero_among_sorted` hits via `.extend()` (not - `flat_map` — a `flat_map` closure can't lazily return something - borrowing its own captured sort permutation across separate calls - without either boxing per-column or fighting the borrow checker; eager - collection into one `Vec` sidesteps it, at zero cost since - `fill_sub_matrix` already fully materialized anyway). `Implicit` trivial. -- `PersistentBitMatrix::fill_sub_matrix` and `sub_matrix` rewritten to - drain `nonzero_iter` — the dispatch bug is gone because there is now - only one traversal per format, not because the old one was patched. - `PersistentCompactIntMatrix::nonzero_iter` added the same way (counts - not excluded, per the earlier ask) — no native low-effort case, since no - sparse count format exists, but on the same primitive, ready for one. -- `KmerPartition::query_partition_with` (`obikindex/src/partition/query_layer.rs`): - stage 2's column-major `for g { for slot { col_value } }` replaced by one - `layer.nonzero_iter(&slot_list)` call per layer, format-agnostic. -- Tests: `nonzero_iter_matches_dense`, `nonzero_iter_matches_row`, and — - the one that actually targets the dispatch bug rather than each type's - own correctness — `enum_wrapper_dispatches_to_native_sparse` (builds - `PersistentBitMatrix::Sparse(...)` directly, not through `open`, since - `open` only auto-detects `Sparse` from a `presence/` dir layout). - `cargo test --workspace`: green, no regressions. - -**Measured**: re-ran the `benchmark/` query branch (100k reads × 2 -specimens, same setup as the original finding). Correctness still 0 -mismatches. The dense/sparse performance gap is gone — previously sparse -~30-50% slower than dense, reproducibly; now within ~1-3% either way -(7.42s dense vs 7.60s sparse for `Escherichia_coli--K-12_MG1655`; 5.25s vs -5.30s for `Saccharolobus_islandicus--M.16.4`) — noise-level, not a -systematic gap. `pack --sparse`'s claimed query win isn't confirmed -outright by this (sparse should arguably now *beat* dense on truly sparse -real data, not just tie), but the pathological regression is fixed. - -## `PersistentCompactIntMatrix::Sparse` — implemented (2026-08-26) - -Closes the gap flagged throughout this document ("no sparse count format -exists yet", `traits.rs:9-12`'s "Explicitly deferred"): `obicompactvec` -already had `PersistentSparseCompactIntMatrix` (row-major, built on top of -`PersistentSparseBitMatrix` as its "which columns are non-zero" support, -values *not* deduplicated — see that struct's own doc comment), but it was -never wired into `PersistentCompactIntMatrix`, the dense-dispatching enum -every real consumer (`TypedLayer`, -`KmerLayer::Count`) actually holds. Concretely: `kmer_index.rs:: -pack_matrices(sparse=true)` already called `pack_sparse_compact_int_matrix` -on every layer's `counts/` — but `PersistentCompactIntMatrix::open` had no -code path back to what that just wrote, so a `Count` layer became -unreadable ("no count matrix found ... run 'obikmer upgrade'") the moment -anyone ran `pack --sparse` on an index with count layers. Root cause, not a -workaround: add the missing `Sparse` variant. - -- **Enum + dispatch** (`intmatrix.rs`): `PersistentCompactIntMatrix::Sparse - (PersistentSparseCompactIntMatrix)`, detected in `open`/`detect_storage` - via a `singleton_values.pciv` marker (mirrors `PersistentBitMatrix`'s own - `sparse_meta.json` check), reported via `storage_kind()`. `col`/ - `col_view`/`col_persist` panic/`Unsupported` on `Sparse`, same convention - as the bit side. `sub_matrix`/`fill_sub_matrix` and `nonzero_iter` - unified the same way `PersistentBitMatrix`'s already are (drain - `nonzero_iter`, one traversal per format — see "Implemented - (2026-08-20)" above); `nonzero_iter` had to become `Box>` - for the same reason (`Columnar`/`Packed`/`Sparse` are different concrete - types). No change needed in `obikindex` at all — `KmerLayer::Count` - already only ever holds `TypedLayer`, so the - enum absorbing `Sparse` fixes the unreadable-layer bug for free, same as - `PersistentBitMatrix::Sparse` already did on the presence side. - -- **`CountPartials`, non-naive** (`sparse_intmatrix.rs`): unlike - `PersistentSparseBitMatrix`'s dict-driven `col_weights_and_pair_counts`, - values here aren't deduplicated (two rows can share the same non-zero - column set via the same `dict_id` while carrying different counts), so - the "weight by how many rows share a dict entry" shortcut doesn't carry - over. What does: a single row-major pass (`row_major_pairwise`, decodes - each row once via `for_each_cell_in_row`, nests over that row's own - co-present columns) — `O(Σ k̄²)` over populated rows instead of the naive - `O(n_cols² × n)` column-pair rescan, same complexity class as the bit - side minus the dict multiplicity discount. Kernels used: `min(a,b)` - (bray, relfreq-bray — both vanish when either side is absent, so no - correction needed), `a·b` and `√(a·b)` (euclidean/relfreq-euclidean and - hellinger — these *do* need a correction, reconstructed from per-column - marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`, since `(a-0)² = a² ≠ 0` unlike - the `min`-based formulas). `threshold_jaccard(1)` shortcuts straight to - `support`'s own `BitPartials::partial_jaccard` (threshold 1 is exactly - presence); `threshold_jaccard(0)` is closed-form (every `u32` is `≥ 0`). - -- **Two pre-existing bugs found and fixed while wiring the `threshold==1` - shortcut** (`bitmatrix/sparse.rs`, `BitPartials for - PersistentSparseBitMatrix`, present since the 2026-08-15 implementation - above, never caught because no test compared `Sparse`'s raw `partial_*` - output against dense on real data — only the diagonal-blind - `jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested): - 1. `partial_jaccard`'s diagonal was `(0, 2×col_weights[i])` instead of a - genuine self-comparison `(col_weights[i], col_weights[i])` — - `col_weights_and_pair_counts`'s `inter` never pairs a column with - itself by construction. - 2. `partial_hamming`'s off-diagonal formula itself was wrong: `total - - union` (count of rows where *neither* column is present) instead of - the actual Hamming distance `col_weights[i] + col_weights[j] - - 2×inter[i,j]` (symmetric-difference size). Only coincides with the - correct value when `col_weights[i] + col_weights[j] == total`, so - small/synthetic test data could easily have hidden it. - - Neither surfaced through `jaccard_dist_matrix`/`hamming_dist_matrix` - (both explicitly zero their own diagonal at finalisation, and the - off-diagonal `partial_hamming` bug had gone untested against dense - entirely) — only visible to a caller of the raw `partial_*` methods - directly, which is exactly what `partial_threshold_jaccard(1)`'s new - shortcut became. Fixed at the source, not patched around at the call - site; regression test added: - `tests::sparse::partial_jaccard_and_hamming_match_dense_including_diagonal`. - -- **Tests**: `tests::intmatrix::sparse_roundtrip_matches_columnar`/ - `sparse_roundtrip_from_packed` (the `open`-dispatch fix, both build - paths); `tests::intmatrix::sparse_count_partials_match_dense` (all six - `CountPartials` formulas, thresholds 0/1/2/3, against `Columnar` on - asymmetric-presence data — this is what caught the diagonal gap in the - int side's own new code before it shipped, the same way it exposed the - two pre-existing bit-side bugs above); `obikindex`'s - `count_layer_transparently_reads_sparse_after_pack` — the actual - end-to-end regression test for the original "layer unreadable after - `pack --sparse`" bug, built → packed sparse → reopened, compared against - the pre-pack dense read. `cargo test -p obicompactvec -p obikindex`: - green, no regressions (180 + 12 tests). +# Sibling annex & SNP-family pipeline — architecture + +## Data model + +A **family** is the up-to-4 k-mers sharing all flanks and differing only at +the central base (well-defined for odd `k`). `FamilyMask` (`siblingannex.rs`) +is a 4-bit presence mask (bit `b` set iff the family member whose own +canonical central base is `b` is observed anywhere in the index) plus a +minorant flag; `family_size()`/`siblings()` derive member count from it. +`SiblingAnnex`/`SiblingAnnexBuilder` persist one `FamilyMask` per k-mer, +**keyed by iteration order** (the physical layout of `unitigs.bin`), never by +MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream +against the annex file directly, no MPHF/slot indirection. + +## Construction + +`algorithms::annex::build_layer_sibling_annex` builds one layer's annex: +seeds each k-mer's own central base, generates its up-to-3 central-base +variants, resolves each cross-partition (`IndexCache::find_in_partition`, +grouped and chunked per destination partition for mmap locality), writes the +final mask (with the minorant flag) in a second streamed pass. +`SiblingExt::build_sibling_annex` drives this for every layer of every +partition, then invalidates and rebuilds the entropy and family-overlap +annexes (their `family_idx` numbering depends on the sibling annex). + +## Shared traversal + +`algorithms::family_scan::scan_layer_families` is the **only** place that +resolves a family's full per-genome presence (`genome_mask: &[u8]`, +cross-partition) — every other consumer in this module is built on it, never +reimplements the resolution. Two decoupled concurrency mechanisms: batch +generation (CPU-only) runs on an `obipipeline` throttle/pipe; per-batch +resolution (I/O-bound, mmap page faults) runs `rayon`-parallel across +destination partitions, one generated batch at a time (never several +concurrently, to keep one partition's pages hot instead of scattering +access). `fast_mode` (`is_fast_mode`, `<= 7` layers per partition) picks +between trusting a family member's recorded layer index outright or probing +every layer of the destination partition. + +`Selection<'a>` bounds which families of a layer are resolved: + +- `All` — every minorant, monomorphic included (`SiblingExt::shannon_entropy_csv`, + `sibling_annex_stats`). +- `Some(&HashSet)` — a caller-picked subset, skipping the expensive + cross-partition resolution entirely for excluded families + (`subsample::sample_layer`'s accepted-candidates batch; + `entropy::ensure_layer_entropy_annex`'s non-monomorphic-only restriction). + +`minorant_selection::eligible_bitset_layer`/`non_monomorphic_reindex_layer` +provide the cheap, annex-bits-only (no cross-partition resolution) filters +these restricted scans are built from: monomorphism (`family_size() < 2`) is +knowable straight from the annex, so nothing pays `scan_layer_families`'s +resolution cost just to discard a monomorphic result. + +## Masking — computed once, shared everywhere + +`algorithms::masking::masked_state(m, free_loss, no_ambiguity)` is the single +function that turns a raw per-genome 4-bit state into the masked view every +consumer needs (`None` for absence under `--free-loss` or ambiguity under +`--no-ambiguity`). It is called **exactly once per (family, genome)**, in +`subsample::sample_layer`, immediately after `scan_layer_families` resolves +the raw state — never recomputed downstream. `survives_masking` (does the +family still carry ≥2 distinct masked states among included genomes?) and +`SurvivingFamily::masked: Vec>` both consume this same +precomputed array, which is also what `AlignmentReducer`/`PairwiseReducer` +read (see below) — this is what guarantees the alignment and the pairwise +tally can never disagree about a genome's state at a given site. + +## Precomputed annexes (entropy, family overlap) + +`algorithms::entropy::ensure_layer_entropy_annex` builds `EntropyAnnex` +(`entropy_annex.rs`, one `f32` Shannon-entropy value per non-monomorphic +minorant, mmap-backed) and/or feeds `FamilyOverlapAccumulator` +(`family_overlap.rs`) from one shared `Selection::Some` scan — both need the +same `genome_mask` resolution, so a missing one never pays for it twice. +`family_entropy` (15 non-empty states of `{A,C,G,T}`'s powerset, `∅` +excluded from the denominator — the project's settled informativeness +definition, matching the calibrated 16-state Sankoff cost matrix) and +`family_entropy_4` (plain 4-symbol reduction, kept only for comparison) are +pure functions of `genome_mask`. `extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes` +drives this for every cached layer, lazily (only what's missing). + +## Sampling and the shared scan trunk + +`algorithms::subsample::sample_index` draws up to `n` non-monomorphic +families index-wide, proportionally per layer: for each layer, a target +quota sized from that layer's share of the index-wide eligible count, filled +by a circular walk with a Bernoulli(`p`) draw per candidate — `p` optionally +weighted by an unnormalized Gaussian kernel over the candidate's entropy +(`EntropyBias { mu, sigma }`, read from the entropy annex). Accepted +candidates are resolved in batches (`scan_layer_families`) and checked +against `survives_masking`; rejected-after-masking candidates don't count +toward quota, triggering further "topup" rounds (bounded, `MAX_TOPUP_ROUNDS += 5`) rather than silently under-filling. Every `SurvivingFamily` reaching a +caller has already survived both the draw and post-masking informativeness — +no second filter pass needed downstream. + +`algorithms::scan::sampled_scan` is the single entry point every sampled +computation goes through: one `sample_index` pass, fanned out to whichever +`LayerReducer` implementations the caller actually needs, each receiving the +exact same per-layer `SurvivingFamily` batch (an `Arc`, cloned not copied). +This is what guarantees several outputs requested together +(`--pseudo-alignment` and `--sankoff` and a `snp-*` `--distance` in one CLI +invocation) come from **one** random sample, not independently-drawn ones — +there is no other driver of `sample_index` anywhere in this crate. + +Two `LayerReducer` implementations exist today: + +- `AlignmentReducer` (`alignment.rs`) — accumulates `SnpAlignment` (one + IUPAC-coded row per included genome), reading `family.masked` directly. +- `PairwiseReducer` (`pairwise.rs`) — accumulates `PairwiseTally`/ + `PartitionDispersion`, also reading `family.masked` directly (substitution/ + agreement counts and cardinality both derive from the same masked state + the alignment uses). + +## From a tally to distances and calibration + +`PairwiseTally` stores per-genome-pair substitution/agreement/cardinality +counts, unfiltered, from which every downstream output is cheap `O(n²)` +post-processing (never a second scan): + +- `algorithms::snp_distance::distance_matrix` — closed-form `snp-*` + corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions- + only), optionally gamma-corrected (`GammaShape`, `PartitionDispersion::estimate_alpha` + for `Auto`). +- `algorithms::sankoff::SankoffBundle::finish` — assembles the + `--sankoff`/`--tnt`/`--phyg`/`--iqtree` pipeline's inputs + (`RawSnpDistanceOutput`, `BasePairTally`, `CardinalityTally`) from an + already-scanned `AlignmentReducer`/`PairwiseReducer` pair, plus + `SankoffBundle::snp_distance` to reuse that exact sample for a `snp-*` + `--distance` in the same invocation without a second scan. +- `algorithms::cardcomp` — pure functions turning `CardinalityTally`/ + `BasePairTally` into row-stochastic transition probability matrices, then + the calibrated 16-state Sankoff cost matrix (`pairwise_cost_matrix`). + +## Public API surface + +`extensions::SiblingExt` (CLI-facing trait on `IndexCache`) covers only the +operations with no cross-invocation sharing concern: `build_sibling_annex`, +`shannon_entropy_csv`, `sibling_family_size_histogram`/`sibling_annex_stats`, +`family_overlap`. The sampled SNP path (`sampled_scan`, `AlignmentReducer`, +`PairwiseReducer`, `distance_matrix`, `SankoffBundle`) is **not** wrapped in +trait methods — the CLI (`obikmer::cmd::phylo`) composes them directly, +because it alone knows the union of reducers a given invocation needs +(`--pseudo-alignment`/Sankoff group/`snp-*` `--distance` can combine in one +call). A per-output trait method would have to either duplicate that +composition logic or reintroduce the "two independent scans" problem this +design replaced. + +## Known gaps + +- `sample_index`'s `skip_layers` parameter (checkpointed-resume support) is + never called with a non-zero value by any current caller — `--session` + persistence was removed pending a redesign on top of `sampled_scan` + (single shared trunk, not per-driver checkpoint plumbing). The parameter + stays as the seam a future resume mechanism would use. +- No cheap entropy pre-filter exists yet for the very high (saturated) tail — + a row-marginal coverage approximation was considered and rejected (cannot + represent joint multi-base co-occurrence, which is exactly what the + 15-state entropy definition exists to capture). diff --git a/DevDocMD/implementation/benchmark_query_testing.md b/DevDocMD/implementation/benchmark_query_testing.md index e234e984..cb9c358c 100644 --- a/DevDocMD/implementation/benchmark_query_testing.md +++ b/DevDocMD/implementation/benchmark_query_testing.md @@ -12,8 +12,9 @@ this pipeline. by `verify_presence`/`verify_merge_presence`, which only check index *content* against the `.npz` truth, never the query API. - `pack --sparse` produces a presence-matrix format documented (see - [siblings.md](../architecture/siblings.md)) as faster for single-row - access (query) and slower for column-oriented access (phylo `--metric`). + [obicompactvec.md](obicompactvec.md), "Sparse matrix formats") as faster + for single-row access (query) and slower for column-oriented access + (phylo `--metric`). `global_index_presence/` built by `merge_presence.sh` is always packed dense (packing is a stage inside `merge`, not a separate `pack` invocation) — there was no dense/sparse regression check. diff --git a/DevDocMD/implementation/layer_tier.md b/DevDocMD/implementation/layer_tier.md index 57499b99..e61c34a7 100644 --- a/DevDocMD/implementation/layer_tier.md +++ b/DevDocMD/implementation/layer_tier.md @@ -299,7 +299,7 @@ pub fn sub_matrix(&self, slots: &[usize]) -> Vec> pub fn fill_sub_matrix(&self, slots: &[usize], out: &mut [Vec]) ``` -Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this degrades to a row-by-row decode; see [siblings.md](../architecture/siblings.md). +Column-first to match the on-disk column-major layout. `fill_sub_matrix` sorts `slots` once, then calls each column's `fill_batch` in turn — no redundant per-column sort. On `PersistentSparseBitMatrix` (k-mer-major, no column method) this goes through `nonzero_iter` instead, decoding one row at a time; see [obicompactvec.md](obicompactvec.md), "Sparse matrix formats". --- diff --git a/DevDocMD/implementation/obicompactvec.md b/DevDocMD/implementation/obicompactvec.md index b999793d..fd29112d 100644 --- a/DevDocMD/implementation/obicompactvec.md +++ b/DevDocMD/implementation/obicompactvec.md @@ -298,13 +298,11 @@ Both matrix types are enums behind a transparent API — the caller never matche |---|---|---| | `Columnar` | one `.pbiv`/`.pciv` file per column + `meta.json` | build-time default (`*Builder::new`) | | `Packed` | single `matrix.pbmx`/`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix`/`pack_compact_int_matrix` | -| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see [siblings.md](../architecture/siblings.md) for the sparse-vs-dense access-pattern trade-off | +| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv`/`multi_values.pciv`/`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse`; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off | | `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1`, every value is `true` | `PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view`/`col`/`sub_matrix` panic on `Sparse`/`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row`/`fill_row`. -Unlike the bit side, `PersistentSparseCompactIntMatrix`'s values are *not* deduplicated across rows — two rows can share the same non-zero column set (same `dict_id` in the shared support) while carrying different counts — so its `CountPartials` impl can't reuse the support's dict-multiplicity shortcut the way `BitPartials for PersistentSparseBitMatrix` does. It still avoids the naive `O(n_cols² × n)` column-pair scan via a single row-major pass (`row_major_pairwise` in `sparse_intmatrix.rs`), reconstructing the squared-difference formulas (`euclidean`/`relfreq_euclidean`/`hellinger`) from per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` — see [siblings.md](../architecture/siblings.md)'s "`PersistentCompactIntMatrix::Sparse` — implemented" entry for the full derivation. - `col_view(c)` returns the appropriate view directly: ```rust @@ -321,6 +319,163 @@ No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view --- +## Sparse matrix formats + +Motivated by real k-mer-family data being extremely row-sparse (most rows — +k-mers — present in only a handful of genome columns out of dozens to +thousands): a row-major, deduplicated alternative to the dense column-major +formats above, built once per layer like the rest of the pipeline (never the +whole multi-billion-row index at once). Implements the exact-duplicate +special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data +Structures for Collections of Sets" (2025, +`biblio/Alanko et al. - Compact Data Structures for Collections of +Sets.pdf`) — a plain dedup dictionary, not their full subset-containment +hierarchy. + +### `PersistentSparseBitMatrix` + +Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row +(singleton vs. multi-genome), a fixed-bit-width array for singleton rows +(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width +array for multi-genome rows (`dict_id`, `ceil(log2(n_distinct_multi_sets))` +bits — kept apart from the singleton array because `n_distinct_multi_sets` +can be large in absolute terms even when multi-genome rows are a small +*fraction* of all rows, and a shared array would force every row, singletons +included, to pay the wider width), and a deduplicated dictionary of distinct +multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values +blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary, +runtime-parameterised bit width, width `0` included — needed once an +`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec` +(rank1/rank0/select1 on top of the crate's existing `count_ones`, via +`common_traits::SelectInWord`), `EliasFano` (composes the two). `BinaryMatrix` +(`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`, `count_ones`) is the +trait unifying dense and sparse at the call sites that need both +interchangeably; column-oriented methods (`col`, `col_view`, the +`partial_*_dist_matrix` family) stay dense-only — `Sparse`/`Implicit` panic +there (see the `PersistentBitMatrix::open` note above). + +Measured on real data (one layer, 30.2M rows, 91 genomes): + +| | dense | sparse | ratio | +|---|---|---|---| +| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller | +| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction | +| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) | +| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** — the `--shannon`/family-lookup access shape | +| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit | + +The row-access wins (compactness was the stated design goal, not speed) come +from dense's genome-major layout scattering a single row read across a much +bigger file — costlier than sparse's rank/select/varint decode once the file +is this much smaller. Column access is the flip side of the same layout +choice; `nonzero_iter` (below) is what removes it from the hot path that +actually matters (`query`). + +Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open` +re-derived its low-bits width from the persisted low-vector's own width +byte; the width-`0` case was built with a dummy 1-bit placeholder (the +builder used to reject true width `0`), silently doubling every reopened +value. Fixed by making `PersistentFixedIntVec` genuinely support width `0` +(no storage, `get` always `0`). (2) An empty row (cardinality 0) used to be +recorded as a singleton at genome `0`, indistinguishable on read-back from a +*real* singleton at genome `0`. Fixed by routing cardinality-0 rows through +the dictionary path (a genuine empty entry). + +Still deferred: full Alanko et al. subset-hierarchy compression (only the +exact-duplicate case is built), a sparse `PersistentCompactIntMatrix` +variant beyond what's described below, BRWT-style column-correlation +exploitation, and a row-major rewrite of +`partial_jaccard_dist_matrix`/`partial_hamming_dist_matrix`/etc. as a +per-row co-occurrence accumulation (`O(Σ_rows k̄²)`) so `obikindex`'s +`--metric` distance-matrix path stops paying the column-access penalty on +`Sparse` — needs its own design pass (how it plugs into +`BitPartials`/`ColumnWeights` so both matrix types keep serving `--metric`), +not just "port the loop". + +### `nonzero_iter` — the one traversal per format + +`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box + 'a>` +(item: index into `slots`, column, value) is the single foundational +per-format traversal primitive every consumer builds on — `fill_sub_matrix` +and a `for_each`-style callback are both trivial, free consumers of it +(`.for_each(f)` is `Iterator::for_each`; `fill_sub_matrix` drains the +iterator and scatters into `out[][]`), instead of two independently +hand-maintained traversals that can silently diverge. `Box`, +not a bare `impl Iterator`: `Columnar`/`Packed`/`Sparse`/`Implicit` are +genuinely different concrete types and the method isn't on a +dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix` +elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per +`nonzero_iter` call, not per cell. + +Per-format implementation: `PersistentSparseBitMatrix` decodes natively, +buffered one row at a time (`std::iter::from_fn` over the existing +`for_each_genome_in_row`, no `n_cols`-wide allocation ever). `Columnar`/ +`Packed` loop columns, collecting each column's hits via +`BitSliceView::nonzero_among_sorted`/`IntSliceView::nonzero_among_sorted` +(a `filter`/`filter_map` over the vector-level `get()`, no new state +machine — `std`'s own `Filter` iterator already is one) into one `Vec` via +`.extend()` (not `flat_map`: a `flat_map` closure can't lazily return +something borrowing its own captured sort permutation across separate calls +without either boxing per-column or fighting the borrow checker — eager +collection sidesteps it at zero extra cost, since `fill_sub_matrix` already +fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))`). + +This closed a real dispatch bug by construction, not by patching it: +`PersistentBitMatrix::fill_sub_matrix`/`sub_matrix` used to reimplement a +naive per-`(column, slot)` loop instead of delegating to +`PersistentSparseBitMatrix`'s own efficient native method for `Self::Sparse` +— two hand-written traversals for the same format, free to drift apart, and +they had (`obikphylo`'s own cache layer had independently built a parallel +enum wrapper that *did* dispatch correctly, evidence the bug was worked +around rather than fixed at the source). With `fill_sub_matrix`/`sub_matrix` +rewritten to drain `nonzero_iter`, there is only one traversal per format +left to get right. `KmerPartition::query_partition_with` +(`obikindex/src/partition/query_layer.rs`) — previously column-major +(`for g { for slot { col_value } }`, which for `Sparse` rebuilt a full row +per single cell touched, O(hits × n_cols) instead of O(hits) — the original +motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)` +call per layer, format-agnostic. Dense/sparse query performance parity +confirmed on real data (previously sparse 30-50% slower than dense, +reproducibly; now within ~1-3% either way — noise level). + +### `PersistentSparseCompactIntMatrix` + +The count-matrix analogue, built on `PersistentSparseBitMatrix` as its +"which columns are non-zero" support, values **not** deduplicated (unlike +the bit side, two rows can share the same non-zero column set — same +`dict_id` — while carrying different counts, so the dict-multiplicity +shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry +over). `CountPartials` here: a single row-major pass (`row_major_pairwise` +in `sparse_intmatrix.rs`, decodes each row once via +`for_each_cell_in_row`, nests over that row's own co-present columns) — +`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)` +column-pair rescan. `min(a,b)`-based kernels (`bray`, `relfreq_bray`) need +no correction — they vanish when either side is absent. `a·b` and `√(a·b)` +kernels (`euclidean`/`relfreq_euclidean`/`hellinger`) do need one, since +`(a-0)² = a² ≠ 0` unlike the `min`-based formulas — reconstructed from +per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab`. +`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard` +(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form +(every `u32` is `≥ 0`). + +Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in +`BitPartials for PersistentSparseBitMatrix` (present since that type's own +introduction, never caught because no test had compared `Sparse`'s raw +`partial_*` output against dense on real data — only the diagonal-blind +`jaccard_dist_matrix`/`hamming_dist_matrix` finalisations were tested), +fixed at the source: + +1. `partial_jaccard`'s diagonal used to be `(0, 2×col_weights[i])` instead + of a genuine self-comparison `(col_weights[i], col_weights[i])` — + `col_weights_and_pair_counts`'s `inter` never pairs a column with itself + by construction, so the diagonal needs its own case. +2. `partial_hamming`'s off-diagonal formula itself was wrong: it computed + `total - union` (rows where *neither* column is present) instead of the + actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]` + (symmetric-difference size) — the two only coincide when + `col_weights[i] + col_weights[j] == total`, which small/synthetic test + data can easily satisfy by accident. + ## Aggregation traits (matrix level) ### ColumnWeights diff --git a/mkdocs.yml b/mkdocs.yml index f1aa9f13..6f493d68 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -60,7 +60,7 @@ nav: - Architecture: - Sequences: architecture/sequences/invariant.md - Kmer index: architecture/index_architecture.md - - Sibling annex (discussion): architecture/siblings.md + - Sibling annex: architecture/siblings.md - NUMA-aware worker pools: architecture/numa_worker_pools.md - NUMA-aware partition runner: architecture/numa_partition_runner.md