Files
obikmer/DevDocMD/architecture/siblings.md
T
Eric Coissac 95bbc0b2ae Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
2026-10-13 08:52:37 +02:00

161 lines
8.9 KiB
Markdown

# Sibling annex & SNP-family pipeline — architecture
## Data model
A **family** is the up-to-4 k-mers sharing all flanks and differing only at
the central base (well-defined for odd `k`). `FamilyMask` (`siblingannex.rs`)
is a 4-bit presence mask (bit `b` set iff the family member whose own
canonical central base is `b` is observed anywhere in the index) plus a
minorant flag; `family_size()`/`siblings()` derive member count from it.
`SiblingAnnex`/`SiblingAnnexBuilder` persist one `FamilyMask` per k-mer,
**keyed by iteration order** (the physical layout of `unitigs.bin`), never by
MPHF slot — this lets every reader zip-iterate a layer's own k-mer stream
against the annex file directly, no MPHF/slot indirection.
## Construction
`algorithms::annex::build_layer_sibling_annex` builds one layer's annex:
seeds each k-mer's own central base, generates its up-to-3 central-base
variants, resolves each cross-partition (`IndexCache::find_in_partition`,
grouped and chunked per destination partition for mmap locality), writes the
final mask (with the minorant flag) in a second streamed pass.
`SiblingExt::build_sibling_annex` drives this for every layer of every
partition, then invalidates and rebuilds the entropy and family-overlap
annexes (their `family_idx` numbering depends on the sibling annex).
## Shared traversal
`algorithms::family_scan::scan_layer_families` is the **only** place that
resolves a family's full per-genome presence (`genome_mask: &[u8]`,
cross-partition) — every other consumer in this module is built on it, never
reimplements the resolution. Two decoupled concurrency mechanisms: batch
generation (CPU-only) runs on an `obipipeline` throttle/pipe; per-batch
resolution (I/O-bound, mmap page faults) runs `rayon`-parallel across
destination partitions, one generated batch at a time (never several
concurrently, to keep one partition's pages hot instead of scattering
access). `fast_mode` (`is_fast_mode`, `<= 7` layers per partition) picks
between trusting a family member's recorded layer index outright or probing
every layer of the destination partition.
`Selection<'a>` bounds which families of a layer are resolved:
- `All` — every minorant, monomorphic included (`SiblingExt::shannon_entropy_csv`,
`sibling_annex_stats`).
- `Some(&HashSet<usize>)` — a caller-picked subset, skipping the expensive
cross-partition resolution entirely for excluded families
(`subsample::sample_layer`'s accepted-candidates batch;
`entropy::ensure_layer_entropy_annex`'s non-monomorphic-only restriction).
`minorant_selection::eligible_bitset_layer`/`non_monomorphic_reindex_layer`
provide the cheap, annex-bits-only (no cross-partition resolution) filters
these restricted scans are built from: monomorphism (`family_size() < 2`) is
knowable straight from the annex, so nothing pays `scan_layer_families`'s
resolution cost just to discard a monomorphic result.
## Masking — computed once, shared everywhere
`algorithms::masking::masked_state(m, free_loss, no_ambiguity)` is the single
function that turns a raw per-genome 4-bit state into the masked view every
consumer needs (`None` for absence under `--free-loss` or ambiguity under
`--no-ambiguity`). It is called **exactly once per (family, genome)**, in
`subsample::sample_layer`, immediately after `scan_layer_families` resolves
the raw state — never recomputed downstream. `survives_masking` (does the
family still carry ≥2 distinct masked states among included genomes?) and
`SurvivingFamily::masked: Vec<Option<u8>>` both consume this same
precomputed array, which is also what `AlignmentReducer`/`PairwiseReducer`
read (see below) — this is what guarantees the alignment and the pairwise
tally can never disagree about a genome's state at a given site.
## Precomputed annexes (entropy, family overlap)
`algorithms::entropy::ensure_layer_entropy_annex` builds `EntropyAnnex`
(`entropy_annex.rs`, one `f32` Shannon-entropy value per non-monomorphic
minorant, mmap-backed) and/or feeds `FamilyOverlapAccumulator`
(`family_overlap.rs`) from one shared `Selection::Some` scan — both need the
same `genome_mask` resolution, so a missing one never pays for it twice.
`family_entropy` (15 non-empty states of `{A,C,G,T}`'s powerset, `∅`
excluded from the denominator — the project's settled informativeness
definition, matching the calibrated 16-state Sankoff cost matrix) and
`family_entropy_4` (plain 4-symbol reduction, kept only for comparison) are
pure functions of `genome_mask`. `extensions::SiblingBuilder::ensure_entropy_and_overlap_annexes`
drives this for every cached layer, lazily (only what's missing).
## Sampling and the shared scan trunk
`algorithms::subsample::sample_index` draws up to `n` non-monomorphic
families index-wide, proportionally per layer: for each layer, a target
quota sized from that layer's share of the index-wide eligible count, filled
by a circular walk with a Bernoulli(`p`) draw per candidate — `p` optionally
weighted by an unnormalized Gaussian kernel over the candidate's entropy
(`EntropyBias { mu, sigma }`, read from the entropy annex). Accepted
candidates are resolved in batches (`scan_layer_families`) and checked
against `survives_masking`; rejected-after-masking candidates don't count
toward quota, triggering further "topup" rounds (bounded, `MAX_TOPUP_ROUNDS
= 5`) rather than silently under-filling. Every `SurvivingFamily` reaching a
caller has already survived both the draw and post-masking informativeness —
no second filter pass needed downstream.
`algorithms::scan::sampled_scan` is the single entry point every sampled
computation goes through: one `sample_index` pass, fanned out to whichever
`LayerReducer` implementations the caller actually needs, each receiving the
exact same per-layer `SurvivingFamily` batch (an `Arc`, cloned not copied).
This is what guarantees several outputs requested together
(`--pseudo-alignment` and `--sankoff` and a `snp-*` `--distance` in one CLI
invocation) come from **one** random sample, not independently-drawn ones —
there is no other driver of `sample_index` anywhere in this crate.
Two `LayerReducer` implementations exist today:
- `AlignmentReducer` (`alignment.rs`) — accumulates `SnpAlignment` (one
IUPAC-coded row per included genome), reading `family.masked` directly.
- `PairwiseReducer` (`pairwise.rs`) — accumulates `PairwiseTally`/
`PartitionDispersion`, also reading `family.masked` directly (substitution/
agreement counts and cardinality both derive from the same masked state
the alignment uses).
## From a tally to distances and calibration
`PairwiseTally` stores per-genome-pair substitution/agreement/cardinality
counts, unfiltered, from which every downstream output is cheap `O(n²)`
post-processing (never a second scan):
- `algorithms::snp_distance::distance_matrix` — closed-form `snp-*`
corrections (raw p-distance, JC69, K80, K81, F81, T92, TN93, transversions-
only), optionally gamma-corrected (`GammaShape`, `PartitionDispersion::estimate_alpha`
for `Auto`).
- `algorithms::sankoff::SankoffBundle::finish` — assembles the
`--sankoff`/`--tnt`/`--phyg`/`--iqtree` pipeline's inputs
(`RawSnpDistanceOutput`, `BasePairTally`, `CardinalityTally`) from an
already-scanned `AlignmentReducer`/`PairwiseReducer` pair, plus
`SankoffBundle::snp_distance` to reuse that exact sample for a `snp-*`
`--distance` in the same invocation without a second scan.
- `algorithms::cardcomp` — pure functions turning `CardinalityTally`/
`BasePairTally` into row-stochastic transition probability matrices, then
the calibrated 16-state Sankoff cost matrix (`pairwise_cost_matrix`).
## Public API surface
`extensions::SiblingExt` (CLI-facing trait on `IndexCache`) covers only the
operations with no cross-invocation sharing concern: `build_sibling_annex`,
`shannon_entropy_csv`, `sibling_family_size_histogram`/`sibling_annex_stats`,
`family_overlap`. The sampled SNP path (`sampled_scan`, `AlignmentReducer`,
`PairwiseReducer`, `distance_matrix`, `SankoffBundle`) is **not** wrapped in
trait methods — the CLI (`obikmer::cmd::phylo`) composes them directly,
because it alone knows the union of reducers a given invocation needs
(`--pseudo-alignment`/Sankoff group/`snp-*` `--distance` can combine in one
call). A per-output trait method would have to either duplicate that
composition logic or reintroduce the "two independent scans" problem this
design replaced.
## Known gaps
- `sample_index`'s `skip_layers` parameter (checkpointed-resume support) is
never called with a non-zero value by any current caller — `--session`
persistence was removed pending a redesign on top of `sampled_scan`
(single shared trunk, not per-driver checkpoint plumbing). The parameter
stays as the seam a future resume mechanism would use.
- No cheap entropy pre-filter exists yet for the very high (saturated) tail —
a row-marginal coverage approximation was considered and rejected (cannot
represent joint multi-base co-occurrence, which is exactly what the
15-state entropy definition exists to capture).