Files
obikmer/DevDocMD/architecture/siblings.md
T
Eric Coissac 95bbc0b2ae Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
2026-10-13 08:52:37 +02:00

8.9 KiB

Sibling annex & SNP-family pipeline — architecture

Data model

A family is the up-to-4 k-mers sharing all flanks and differing only at the central base (well-defined for odd k). FamilyMask (siblingannex.rs) is a 4-bit presence mask (bit b set iff the family member whose own canonical central base is b is observed anywhere in the index) plus a minorant flag; family_size()/siblings() derive member count from it. SiblingAnnex/SiblingAnnexBuilder persist one FamilyMask per k-mer, keyed by iteration order (the physical layout of unitigs.bin), never by MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream against the annex file directly, no MPHF/slot indirection.

Construction

algorithms::annex::build_layer_sibling_annex builds one layer's annex: seeds each k-mer's own central base, generates its up-to-3 central-base variants, resolves each cross-partition (IndexCache::find_in_partition, grouped and chunked per destination partition for mmap locality), writes the final mask (with the minorant flag) in a second streamed pass. SiblingExt::build_sibling_annex drives this for every layer of every partition, then invalidates and rebuilds the entropy and family-overlap annexes (their family_idx numbering depends on the sibling annex).

Shared traversal

algorithms::family_scan::scan_layer_families is the only place that resolves a family's full per-genome presence (genome_mask: &[u8], cross-partition) — every other consumer in this module is built on it, never reimplements the resolution. Two decoupled concurrency mechanisms: batch generation (CPU-only) runs on an obipipeline throttle/pipe; per-batch resolution (I/O-bound, mmap page faults) runs rayon-parallel across destination partitions, one generated batch at a time (never several concurrently, to keep one partition's pages hot instead of scattering access). fast_mode (is_fast_mode, <= 7 layers per partition) picks between trusting a family member's recorded layer index outright or probing every layer of the destination partition.

Selection<'a> bounds which families of a layer are resolved:

  • All — every minorant, monomorphic included (SiblingExt::shannon_entropy_csv, sibling_annex_stats).
  • Some(&HashSet<usize>) — a caller-picked subset, skipping the expensive cross-partition resolution entirely for excluded families (subsample::sample_layer's accepted-candidates batch; entropy::ensure_layer_entropy_annex's non-monomorphic-only restriction).

minorant_selection::eligible_bitset_layer/non_monomorphic_reindex_layer provide the cheap, annex-bits-only (no cross-partition resolution) filters these restricted scans are built from: monomorphism (family_size() < 2) is knowable straight from the annex, so nothing pays scan_layer_families's resolution cost just to discard a monomorphic result.

Masking — computed once, shared everywhere

algorithms::masking::masked_state(m, free_loss, no_ambiguity) is the single function that turns a raw per-genome 4-bit state into the masked view every consumer needs (None for absence under --free-loss or ambiguity under --no-ambiguity). It is called exactly once per (family, genome), in subsample::sample_layer, immediately after scan_layer_families resolves the raw state — never recomputed downstream. survives_masking (does the family still carry ≥2 distinct masked states among included genomes?) and SurvivingFamily::masked: Vec<Option<u8>> both consume this same precomputed array, which is also what AlignmentReducer/PairwiseReducer read (see below) — this is what guarantees the alignment and the pairwise tally can never disagree about a genome's state at a given site.

Precomputed annexes (entropy, family overlap)

algorithms::entropy::ensure_layer_entropy_annex builds EntropyAnnex (entropy_annex.rs, one f32 Shannon-entropy value per non-monomorphic minorant, mmap-backed) and/or feeds FamilyOverlapAccumulator (family_overlap.rs) from one shared Selection::Some scan — both need the same genome_mask resolution, so a missing one never pays for it twice. family_entropy (15 non-empty states of {A,C,G,T}'s powerset, ∅ excluded from the denominator — the project's settled informativeness definition, matching the calibrated 16-state Sankoff cost matrix) and family_entropy_4 (plain 4-symbol reduction, kept only for comparison) are pure functions of genome_mask. extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes drives this for every cached layer, lazily (only what's missing).

Sampling and the shared scan trunk

algorithms::subsample::sample_index draws up to n non-monomorphic families index-wide, proportionally per layer: for each layer, a target quota sized from that layer's share of the index-wide eligible count, filled by a circular walk with a Bernoulli(p) draw per candidate — p optionally weighted by an unnormalized Gaussian kernel over the candidate's entropy (EntropyBias { mu, sigma }, read from the entropy annex). Accepted candidates are resolved in batches (scan_layer_families) and checked against survives_masking; rejected-after-masking candidates don't count toward quota, triggering further "topup" rounds (bounded, MAX_TOPUP_ROUNDS = 5) rather than silently under-filling. Every SurvivingFamily reaching a caller has already survived both the draw and post-masking informativeness — no second filter pass needed downstream.

algorithms::scan::sampled_scan is the single entry point every sampled computation goes through: one sample_index pass, fanned out to whichever LayerReducer implementations the caller actually needs, each receiving the exact same per-layer SurvivingFamily batch (an Arc, cloned not copied). This is what guarantees several outputs requested together (--pseudo-alignment and --sankoff and a snp-* --distance in one CLI invocation) come from one random sample, not independently-drawn ones — there is no other driver of sample_index anywhere in this crate.

Two LayerReducer implementations exist today:

  • AlignmentReducer (alignment.rs) — accumulates SnpAlignment (one IUPAC-coded row per included genome), reading family.masked directly.
  • PairwiseReducer (pairwise.rs) — accumulates PairwiseTally/ PartitionDispersion, also reading family.masked directly (substitution/ agreement counts and cardinality both derive from the same masked state the alignment uses).

From a tally to distances and calibration

PairwiseTally stores per-genome-pair substitution/agreement/cardinality counts, unfiltered, from which every downstream output is cheap O(n²) post-processing (never a second scan):

  • algorithms::snp_distance::distance_matrix — closed-form snp-* corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions- only), optionally gamma-corrected (GammaShape, PartitionDispersion::estimate_alpha for Auto).
  • algorithms::sankoff::SankoffBundle::finish — assembles the --sankoff/--tnt/--phyg/--iqtree pipeline's inputs (RawSnpDistanceOutput, BasePairTally, CardinalityTally) from an already-scanned AlignmentReducer/PairwiseReducer pair, plus SankoffBundle::snp_distance to reuse that exact sample for a snp-* --distance in the same invocation without a second scan.
  • algorithms::cardcomp — pure functions turning CardinalityTally/ BasePairTally into row-stochastic transition probability matrices, then the calibrated 16-state Sankoff cost matrix (pairwise_cost_matrix).

Public API surface

extensions::SiblingExt (CLI-facing trait on IndexCache) covers only the operations with no cross-invocation sharing concern: build_sibling_annex, shannon_entropy_csv, sibling_family_size_histogram/sibling_annex_stats, family_overlap. The sampled SNP path (sampled_scan, AlignmentReducer, PairwiseReducer, distance_matrix, SankoffBundle) is not wrapped in trait methods — the CLI (obikmer::cmd::phylo) composes them directly, because it alone knows the union of reducers a given invocation needs (--pseudo-alignment/Sankoff group/snp-* --distance can combine in one call). A per-output trait method would have to either duplicate that composition logic or reintroduce the "two independent scans" problem this design replaced.

Known gaps

  • sample_index's skip_layers parameter (checkpointed-resume support) is never called with a non-zero value by any current caller — --session persistence was removed pending a redesign on top of sampled_scan (single shared trunk, not per-driver checkpoint plumbing). The parameter stays as the seam a future resume mechanism would use.
  • No cheap entropy pre-filter exists yet for the very high (saturated) tail — a row-marginal coverage approximation was considered and rejected (cannot represent joint multi-base co-occurrence, which is exactly what the 15-state entropy definition exists to capture).