Introduce `--vcf` and `--fasta` options for phylo commands, allowing users to export pseudo-alignment data as VCF or FASTA. Implement comprehensive VCF logic including stable virtual coordinates, canonical ID generation, and flexible genotype representation based on ambiguity settings. Refactor internal alignment structures to track k-mer families and site loci, ensuring consistent data flow from scanning through to output generation. - Add `src/obikmer/src/cmd/phylo/alignment_out.rs` with `create` and `write_alignment_outputs` to handle file creation and dispatching of FASTA/VCF formats based on `PhyloArgs`. - Add `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` implementing VCF-specific logic: `write_vcf`, `alignment`, `cell_alleles`, `majority_base`, and site construction via `site` using fixed $K=11$. Includes helpers for IUPAC state inversion, genotype ordering, and stable coordinate generation. - Update `src/obikmer/src/cmd/phylo/args.rs` to add `fasta` and `vcf` flags and a new `alignment_output()` method that returns true if any output format is requested. Add `pseudo_alignment` field documentation. - Modify `src/obikmer/src/cmd/phylo/mod.rs` to use `alignment_output()` for determining alignment needs, update error messages for `--n-sites`, and import `write_alignment_outputs` from the new module. - Refactor `src/obikphylo/src/siblings/algorithms/alignment.rs` by adding `SiteLocus` struct (partition, layer, family_idx, family_id) and updating `SnpAlignment` and `AlignmentReducer` to track sites. Update `new`, `finish`, and `reduce` to populate site information from `SurvivingFamily`. - Update `src/obikphylo/src/siblings/algorithms/iqtree.rs`: `alignment` now returns `SnpAlignment` with empty sites; update `drop_ascertainment_noninformative` and `recode_symbols_as_absent` to handle the new `sites` field. - Update `src/obikphylo/src/siblings/algorithms/family_scan.rs`: `GeneratedBatch` now includes a `kmers` field; `scan_layer_families` accepts `CanonicalKmer` and passes it to closures instead of masks. - Update `src/obikphylo/src/siblings/algorithms/subsample.rs`: `SurvivingFamily` gains a `kmer` field; `sample_layer` closure accepts and uses this k-mer. - Update `src/obikphylo/src/siblings/algorithms/snp_distance.rs`: `tally_2genomes` initializes `SurvivingFamily` with explicit `CanonicalKmer` values. - Update `src/obikphylo/src/siblings/algorithms/stats.rs`: `sibling_annex_stats` signature adds `_kmer` parameter. - Update `src/obikphylo/src/siblings/algorithms/family_annexes.rs`: closures in `ensure_layer_family_annexes` and `ensure_layer_pair_weight_annexes` now accept `_kmer`. - Update `src/obikphylo/src/siblings/algorithms/pairwise.rs`: `ambiguous_vs_single` initializes k-mer using `CanonicalKmer::from_raw_unchecked(0)`. - Add `iupac_state` function to `src/obikphylo/src/siblings/algorithms/masking.rs` mapping DNA characters to binary codes. - Update `src/obikphylo/src/siblings/algorithms/mod.rs` to export `SiteLocus` and include new `alignment_vcf` module. - Add `family_id` and `write_family_id_ascii` methods to `src/obikseq/src/kmer.rs` for stable k-mer family identification. - Add tests `family_id_partitions_kmers_into_families` and `family_id_ascii_masks_centre` to `src/obikseq/src/tests/kmer.rs`. - Update `src/obikphylo/src/siblings/mod.rs` imports to include new algorithm exports.
596 lines
27 KiB
Rust
596 lines
27 KiB
Rust
//! Per-genome-pair composition/cardinality tally — a single reduce over
|
|
//! [`super::subsample::SurvivingFamily`] batches, storing enough per pair
|
|
//! (`i < j`, upper triangle only — reciprocal by construction, half the
|
|
//! naive `n × n` storage) that every one of [`RawSnpDistanceOutput`],
|
|
//! [`PairwiseTally::included`]'s `ratio_ceiling` filter,
|
|
//! [`BasePairTally`] and [`CardinalityTally`] derives from it afterward as
|
|
//! cheap `O(n²)` post-processing — never a second scan of the index. This
|
|
//! replaces the old two-pass design (`raw_snp_distance` fully computed
|
|
//! before `base_pair_tally`/`cardinality_tally` could even start, because
|
|
//! only the *complete* aggregate told you which pairs were
|
|
//! `ratio_ceiling`-eligible): keeping the *unfiltered* per-pair detail
|
|
//! instead of collapsing straight to a pooled tally means the filter can be
|
|
//! applied after the fact, on data already in hand.
|
|
//!
|
|
//! Deliberately `O(n²)` in genome count (unlike everything else in this
|
|
//! crate, which is `O(n)` or bounded per layer/round) — accepted as a
|
|
//! one-time cost for the target scale this project runs at (large-memory
|
|
//! machines), not something to optimise away.
|
|
|
|
use ndarray::Array2;
|
|
use rkyv::{Archive, Deserialize, Serialize};
|
|
|
|
use super::subsample::SurvivingFamily;
|
|
|
|
/// One pair's own tallies, unfiltered. Both `subst` and `same_all` are
|
|
/// populated by every family reaching [`reduce_pairwise`] — no separate
|
|
/// "variable families only" gate needed for either: [`super::subsample::sample_layer`]
|
|
/// only ever draws from `minorant_selection::eligible_bitset_layer`, which
|
|
/// contains exclusively non-monomorphic minorants (`family_size() >= 2`,
|
|
/// enforced by a `debug_assert!` right where a [`SurvivingFamily`] is
|
|
/// built), so *every* family folded here already varies somewhere in the
|
|
/// index. `same_all` is what both [`RawSnpDistanceOutput`] and
|
|
/// [`BasePairTally`] read (there used to be a second, `same_variable`
|
|
/// field restricted to variable families for the latter — removed once it
|
|
/// was confirmed, via the invariant above, to always hold the exact same
|
|
/// values as `same_all`). `cardinality` is populated for every family too,
|
|
/// same reasoning.
|
|
///
|
|
/// `Archive`/`Serialize`/`Deserialize`: part of [`PairwiseTally`]'s own
|
|
/// `--session` persistence (see that struct's docs) — `rkyv` needs every
|
|
/// field type to derive these too, hence deriving here on an otherwise
|
|
/// purely internal struct.
|
|
#[derive(Default, Clone, Copy, Archive, Serialize, Deserialize)]
|
|
struct PairStats {
|
|
subst: [[u64; 4]; 4],
|
|
same_all: [u64; 4],
|
|
cardinality: [[u64; 5]; 5],
|
|
}
|
|
|
|
/// See the module docs. `pub`: returned by [`PairwiseReducer::finish`].
|
|
///
|
|
/// `Archive`/`Serialize`/`Deserialize` (`rkyv`): kept from when this type
|
|
/// was dumped to `--session` as opaque bytes via `obiksession::Session`;
|
|
/// no active user right now (`--session` support was removed pending a
|
|
/// rebuild on top of `algorithms::scan::sampled_scan`), but cheap to keep
|
|
/// derived for when it returns.
|
|
#[derive(Archive, Serialize, Deserialize)]
|
|
pub struct PairwiseTally {
|
|
n_genomes: usize,
|
|
/// Upper triangle only (`i < j`), flat-indexed via [`Self::flat_index`].
|
|
pairs: Vec<PairStats>,
|
|
}
|
|
|
|
impl PairwiseTally {
|
|
pub(crate) fn new(n_genomes: usize) -> Self {
|
|
let n_pairs = n_genomes.saturating_sub(1) * n_genomes / 2;
|
|
Self { n_genomes, pairs: vec![PairStats::default(); n_pairs] }
|
|
}
|
|
|
|
pub(crate) fn n_genomes(&self) -> usize {
|
|
self.n_genomes
|
|
}
|
|
|
|
/// See `crate::siblings::helpers::triangle_index`.
|
|
fn flat_index(&self, i: usize, j: usize) -> usize {
|
|
crate::siblings::helpers::triangle_index(self.n_genomes, i, j)
|
|
}
|
|
|
|
#[inline]
|
|
fn pair(&self, i: usize, j: usize) -> &PairStats {
|
|
&self.pairs[self.flat_index(i, j)]
|
|
}
|
|
|
|
#[inline]
|
|
fn pair_mut(&mut self, i: usize, j: usize) -> &mut PairStats {
|
|
let idx = self.flat_index(i, j);
|
|
&mut self.pairs[idx]
|
|
}
|
|
|
|
/// This pair's total eligible-locus count (`snp + shared`, unfiltered)
|
|
/// and its raw SNP ratio (`snp / (snp + shared)`, `None` if no eligible
|
|
/// locus at all) — the two quantities [`Self::included`] and
|
|
/// [`RawSnpDistanceOutput`] both derive from.
|
|
fn snp_shared(&self, i: usize, j: usize) -> (u64, u64) {
|
|
let stats = self.pair(i, j);
|
|
// Every substitution was recorded twice (`subst[bi][bj]` and
|
|
// `subst[bj][bi]`, matching `BasePairTally::counts`'s own
|
|
// reciprocal convention) — halve the flat sum back to a true count.
|
|
let snp: u64 = stats.subst.iter().flatten().sum::<u64>() / 2;
|
|
let shared: u64 = stats.same_all.iter().sum();
|
|
(snp, shared)
|
|
}
|
|
|
|
pub(crate) fn raw_snp_distance(&self) -> RawSnpDistanceOutput {
|
|
let mut snp = Array2::<u64>::zeros((self.n_genomes, self.n_genomes));
|
|
let mut shared = Array2::<u64>::zeros((self.n_genomes, self.n_genomes));
|
|
for i in 0..self.n_genomes {
|
|
for j in (i + 1)..self.n_genomes {
|
|
let (s, sh) = self.snp_shared(i, j);
|
|
snp[[i, j]] = s;
|
|
snp[[j, i]] = s;
|
|
shared[[i, j]] = sh;
|
|
shared[[j, i]] = sh;
|
|
}
|
|
}
|
|
RawSnpDistanceOutput { snp, shared }
|
|
}
|
|
|
|
/// Whether pair `(i, j)` is eligible for `base_pair_tally`/
|
|
/// `cardinality_tally`'s calibration pool: enough eligible loci to have
|
|
/// a real SNP ratio, that ratio at or below `ratio_ceiling` (a pair this
|
|
/// close to saturation carries no information about the true
|
|
/// substitution rate — see `--sankoff-ratio-ceiling`'s own docs), and
|
|
/// neither genome excluded (`excluded[g]`, `--exclude-genome`) — an
|
|
/// excluded genome must never influence the calibrated matrices, not
|
|
/// just be absent from the reported output.
|
|
pub(crate) fn included(&self, ratio_ceiling: f64, excluded: &[bool]) -> Array2<bool> {
|
|
Array2::from_shape_fn((self.n_genomes, self.n_genomes), |(i, j)| {
|
|
if i == j {
|
|
return false;
|
|
}
|
|
if excluded.get(i).copied().unwrap_or(false) || excluded.get(j).copied().unwrap_or(false) {
|
|
return false;
|
|
}
|
|
let (snp, shared) = self.snp_shared(i, j);
|
|
let total = snp + shared;
|
|
total > 0 && (snp as f64 / total as f64) <= ratio_ceiling
|
|
})
|
|
}
|
|
|
|
pub(crate) fn base_pair_tally(&self, included: &Array2<bool>) -> BasePairTally {
|
|
let mut counts = [[0u64; 4]; 4];
|
|
let mut same = [0u64; 4];
|
|
for i in 0..self.n_genomes {
|
|
for j in (i + 1)..self.n_genomes {
|
|
if !included[[i, j]] {
|
|
continue;
|
|
}
|
|
let stats = self.pair(i, j);
|
|
for a in 0..4 {
|
|
same[a] += stats.same_all[a];
|
|
for b in 0..4 {
|
|
counts[a][b] += stats.subst[a][b];
|
|
}
|
|
}
|
|
}
|
|
}
|
|
BasePairTally { counts, same }
|
|
}
|
|
|
|
/// Unlike [`Self::base_pair_tally`], **not** gated by `ratio_ceiling`:
|
|
/// that filter exists because a substitution-saturated pair's base
|
|
/// composition trends toward random noise, no longer informative about
|
|
/// the true substitution spectrum — a fact about *sequence divergence*
|
|
/// between the two genomes. Cardinality (how many family members each
|
|
/// genome carries) reflects each genome's own coverage/assembly
|
|
/// completeness and duplication/paralogy structure, not the pair's
|
|
/// mutual divergence — reusing the SNP-ratio filter here would exclude
|
|
/// pairs for a reason that has no established bearing on cardinality
|
|
/// co-occurrence. Only `--exclude-genome` applies.
|
|
pub(crate) fn cardinality_tally(&self, excluded: &[bool]) -> CardinalityTally {
|
|
let mut counts = [[0u64; 5]; 5];
|
|
for i in 0..self.n_genomes {
|
|
if excluded.get(i).copied().unwrap_or(false) {
|
|
continue;
|
|
}
|
|
for j in (i + 1)..self.n_genomes {
|
|
if excluded.get(j).copied().unwrap_or(false) {
|
|
continue;
|
|
}
|
|
let stats = self.pair(i, j);
|
|
for a in 0..5 {
|
|
for b in 0..5 {
|
|
counts[a][b] += stats.cardinality[a][b];
|
|
}
|
|
}
|
|
}
|
|
}
|
|
CardinalityTally { counts }
|
|
}
|
|
|
|
/// Categorised substitution counts for pair `(i, j)`, plus its eligible
|
|
/// ("shared") locus count — the sufficient statistic every closed-form
|
|
/// `algorithms::snp_distance` correction is built from. Base order
|
|
/// `0=A,1=C,2=G,3=T` (same convention as `FamilyMask`/`STATE_SYMBOL`).
|
|
/// `ts1`/`ts2` split transitions by purine (A↔G) vs pyrimidine (C↔T)
|
|
/// pair — required for TN93, collapse to a single `ts1+ts2` for
|
|
/// K80/K81/T92, which don't distinguish them. `tv1`/`tv2` split
|
|
/// transversions the way Kimura's 3-parameter model does (A↔C/G↔T vs
|
|
/// A↔T/C↔G) — collapse to `tv1+tv2` for every model that doesn't need
|
|
/// the distinction (K80, F81, T92, TN93).
|
|
pub(crate) fn categories(&self, i: usize, j: usize) -> PairCategories {
|
|
let stats = self.pair(i, j);
|
|
PairCategories {
|
|
ts1: stats.subst[0][2],
|
|
ts2: stats.subst[1][3],
|
|
tv1: stats.subst[0][1] + stats.subst[2][3],
|
|
tv2: stats.subst[0][3] + stats.subst[1][2],
|
|
shared: stats.same_all.iter().sum(),
|
|
}
|
|
}
|
|
|
|
/// Pooled base composition for pair `(i, j)`, estimated from both
|
|
/// genomes' observed calls at their `shared`/differing eligible loci —
|
|
/// `2 * same_all[a]` (each agreement locus contributes `a` to both
|
|
/// genomes) plus `Σ_b subst[a][b]` (each differing locus contributes
|
|
/// `a` to whichever of the two genomes carried it — `subst` is
|
|
/// symmetric by construction, so summing one row already counts each
|
|
/// such event exactly once, see `reduce_pairwise`'s own docs), over
|
|
/// `2 * n_eligible_loci` total base observations. Feeds F81/T92/TN93,
|
|
/// which all correct for base-composition bias using exactly this kind
|
|
/// of pooled empirical frequency (Nei & Kumar's standard estimator for
|
|
/// a pairwise comparison, not a whole-index average).
|
|
pub(crate) fn base_freq(&self, i: usize, j: usize) -> [f64; 4] {
|
|
let stats = self.pair(i, j);
|
|
let mut counts = [0u64; 4];
|
|
for a in 0..4 {
|
|
counts[a] = 2 * stats.same_all[a] + (0..4).map(|b| stats.subst[a][b]).sum::<u64>();
|
|
}
|
|
let total: u64 = counts.iter().sum();
|
|
if total == 0 {
|
|
return [0.25; 4];
|
|
}
|
|
counts.map(|c| c as f64 / total as f64)
|
|
}
|
|
}
|
|
|
|
/// See [`PairwiseTally::categories`]'s own docs.
|
|
pub(crate) struct PairCategories {
|
|
pub ts1: u64,
|
|
pub ts2: u64,
|
|
pub tv1: u64,
|
|
pub tv2: u64,
|
|
pub shared: u64,
|
|
}
|
|
|
|
impl PairCategories {
|
|
pub(crate) fn n_eligible(&self) -> u64 {
|
|
self.ts1 + self.ts2 + self.tv1 + self.tv2 + self.shared
|
|
}
|
|
}
|
|
|
|
/// Raw p-distance restricted to loci that are single-copy in **both**
|
|
/// genomes of a pair — see `PairwiseTally`'s module docs. `pub`: part of
|
|
/// [`super::SankoffBundle`]'s public signature.
|
|
pub struct RawSnpDistanceOutput {
|
|
/// n×n count of eligible loci where the two genomes' single forms differ.
|
|
pub snp: Array2<u64>,
|
|
/// n×n count of eligible loci where the two genomes' single forms agree.
|
|
pub shared: Array2<u64>,
|
|
}
|
|
|
|
/// Symmetric 6-category base-pair substitution tally (indexed
|
|
/// `0=A,1=C,2=G,3=T`), pooled over [`PairwiseTally::included`] genome
|
|
/// pairs. `pub`: part of [`super::SankoffBundle`]'s public signature.
|
|
pub struct BasePairTally {
|
|
/// `counts[a][b] == counts[b][a]` = number of eligible loci, pooled over
|
|
/// included genome pairs, where the two genomes' single forms are `a`
|
|
/// and `b`. Diagonal always `0` — an `a == b` locus is counted in
|
|
/// `same`, not here.
|
|
pub counts: [[u64; 4]; 4],
|
|
/// `same[a]` = number of eligible loci, pooled over included genome
|
|
/// pairs, where both genomes' single forms are `a` — every family
|
|
/// folded into [`PairwiseTally`] already varies somewhere in the index
|
|
/// by construction (see `PairStats`'s own docs), so there is no
|
|
/// further "variable families only" restriction to apply here.
|
|
pub same: [u64; 4],
|
|
}
|
|
|
|
/// Cardinality (`0..=4` members of a family) co-occurrence, pooled over
|
|
/// every genome pair except `--exclude-genome` ones (see
|
|
/// [`PairwiseTally::cardinality_tally`]'s own docs for why this is *not*
|
|
/// gated by `ratio_ceiling` the way [`BasePairTally`] is). `pub`: part of
|
|
/// [`super::SankoffBundle`]'s public signature.
|
|
pub struct CardinalityTally {
|
|
/// `counts[a][b] == counts[b][a]` = number of family sites, pooled over
|
|
/// included genome pairs, where one genome's family cardinality is `a`
|
|
/// and the other's is `b`. Diagonal is real data here (both genomes at
|
|
/// the same cardinality), unlike [`BasePairTally::counts`].
|
|
pub counts: [[u64; 5]; 5],
|
|
}
|
|
|
|
/// Per-partition, pooled-over-every-genome-pair `(substitutions, eligible
|
|
/// loci)` counts — feeds [`Self::estimate_alpha`], the method-of-moments
|
|
/// `--gamma-shape auto` estimator (an independent derivation — see that
|
|
/// method's own docs for why this is *not* Jin & Nei's (1990) method,
|
|
/// verified against their paper directly, despite using the same `+Γ`
|
|
/// distance formula). Pooling across pairs (rather than keeping one series
|
|
/// per pair) treats every included genome pair as one more observation of
|
|
/// the same underlying rate-heterogeneity signal — comparable in spirit to
|
|
/// combining information across several genomic regions/genes for a fixed
|
|
/// set of taxa, generalised here to every pair at once.
|
|
///
|
|
/// Deliberately **not** gated by `--sankoff-ratio-ceiling` the way
|
|
/// [`PairwiseTally::base_pair_tally`] is (see
|
|
/// [`PairwiseTally::cardinality_tally`]'s own docs for the same
|
|
/// precedent): that filter excludes individual saturated *pairs* from a
|
|
/// composition estimate computed once at the very end, from the complete
|
|
/// `PairwiseTally`; the partition axis needed here only exists transiently,
|
|
/// one partition at a time, while `PairwiseTally` is still being built —
|
|
/// long before any pair's final SNP ratio (and thus its ratio_ceiling
|
|
/// eligibility) is known. Genome exclusion (`--exclude-genome`) isn't
|
|
/// applied here either, matching `reduce_pairwise`'s own raw per-pair fold
|
|
/// (`PairStats` accumulates over every genome pair unconditionally;
|
|
/// exclusion is a filter applied only in derived, post-hoc views).
|
|
#[derive(Default, Clone, Archive, Serialize, Deserialize)]
|
|
pub struct PartitionDispersion {
|
|
/// One entry per partition id seen so far (grows lazily — partitions
|
|
/// are visited in increasing order in practice, but nothing here
|
|
/// assumes that).
|
|
counts: Vec<(u64, u64)>,
|
|
}
|
|
|
|
impl PartitionDispersion {
|
|
/// Add one partition/layer's worth of pooled counts (from one
|
|
/// [`reduce_pairwise`] call) to that partition's running total —
|
|
/// additive so a partition with several layers accumulates correctly
|
|
/// across multiple calls.
|
|
fn record(&mut self, partition: usize, subst: u64, eligible: u64) {
|
|
if partition >= self.counts.len() {
|
|
self.counts.resize(partition + 1, (0, 0));
|
|
}
|
|
let entry = &mut self.counts[partition];
|
|
entry.0 += subst;
|
|
entry.1 += eligible;
|
|
}
|
|
|
|
/// Method-of-moments estimate of the gamma shape parameter `alpha`
|
|
/// used by the `+Γ` distance correction, from the over-dispersion of
|
|
/// per-partition substitution *rates* relative to what a single shared
|
|
/// rate would produce under pure Poisson sampling noise — the general
|
|
/// negative-binomial (Poisson-Gamma mixture) identity, the same kind of
|
|
/// reasoning behind Uzzell & Corbin's (1971) original observation that
|
|
/// substitution counts across sites/regions are over-dispersed
|
|
/// relative to Poisson.
|
|
///
|
|
/// **Not Jin & Nei's (1990) method** — checked directly against their
|
|
/// paper (*"Limitations of the Evolutionary Parsimony Method of
|
|
/// Phylogenetic Analysis"*, Mol. Biol. Evol. 7(2):82-102, the source of
|
|
/// the `+Γ` distance formula this alpha feeds into, confirmed to match
|
|
/// `corrected_log`/`k2p` exactly against their eq. A4/A8): that paper
|
|
/// contains no data-driven alpha estimator. Their own recommendation
|
|
/// (p. 98) is the fixed default `alpha = 1`; for estimating alpha from
|
|
/// data they point to a different paper (Wilson et al. 1989, not
|
|
/// read/verified here) rather than proposing their own procedure. This
|
|
/// estimator is therefore a from-scratch derivation (Poisson count
|
|
/// within a partition, with a Gamma(alpha, alpha)-distributed (mean 1)
|
|
/// multiplicative rate shared by every locus in that partition), not a
|
|
/// literature implementation — unlike every closed-form correction in
|
|
/// `snp_distance.rs`, each verified line-by-line against `ape`'s
|
|
/// `dist_dna.c`.
|
|
///
|
|
/// Partitions have unequal eligible-locus counts (`l_i`), so a plain
|
|
/// unweighted variance across raw per-partition rates would conflate
|
|
/// genuine rate heterogeneity with the extra Poisson noise smaller
|
|
/// partitions carry. Weighting each partition's squared deviation by
|
|
/// its own `l_i` cancels that: for `R_i = subst_i / l_i`, `E[R_i] =
|
|
/// mu`, `Var[R_i] = mu/l_i + mu²/alpha`, so `E[Σ l_i (R_i-mu)² / Σ l_i]
|
|
/// = mu/mean(l) + mu²/alpha` — the Poisson floor
|
|
/// (`mu/mean(l)`) is subtracted below before solving for `alpha`.
|
|
///
|
|
/// Returns `None` when there's nothing to estimate from: fewer than 2
|
|
/// partitions with data, a pooled mean rate of exactly 0, or a
|
|
/// measured variance at or below the Poisson floor (no detectable
|
|
/// over-dispersion — `alpha` would be unbounded, not just large).
|
|
pub(crate) fn estimate_alpha(&self) -> Option<f64> {
|
|
let rows: Vec<(f64, f64)> = self
|
|
.counts
|
|
.iter()
|
|
.filter(|&&(_, eligible)| eligible > 0)
|
|
.map(|&(subst, eligible)| (subst as f64 / eligible as f64, eligible as f64))
|
|
.collect();
|
|
if rows.len() < 2 {
|
|
return None;
|
|
}
|
|
|
|
let weight_total: f64 = rows.iter().map(|&(_, w)| w).sum();
|
|
let mean: f64 = rows.iter().map(|&(rate, w)| rate * w).sum::<f64>() / weight_total;
|
|
if mean <= 0.0 {
|
|
return None;
|
|
}
|
|
|
|
let weighted_variance: f64 =
|
|
rows.iter().map(|&(rate, w)| w * (rate - mean).powi(2)).sum::<f64>() / weight_total;
|
|
let mean_eligible = weight_total / rows.len() as f64;
|
|
let poisson_floor = mean / mean_eligible;
|
|
let excess = weighted_variance - poisson_floor;
|
|
if excess <= 0.0 {
|
|
return None;
|
|
}
|
|
|
|
Some(mean * mean / excess)
|
|
}
|
|
}
|
|
|
|
/// One layer's worth of [`SurvivingFamily`] folded into `tally` (and
|
|
/// `dispersion`, pooled under `partition` — see [`PartitionDispersion`]'s
|
|
/// own docs for why the partition axis must be captured here rather than
|
|
/// after the fact) — the "reduce" half of the map/reduce
|
|
/// [`super::subsample::sample_index`] drives, self-contained (no
|
|
/// sampling/masking logic of its own) so it can run alongside any other
|
|
/// reduce over the same batch (see `algorithms::alignment::reduce_alignment`,
|
|
/// run alongside this one whenever both an alignment and a tally are
|
|
/// wanted from the same [`super::scan::sampled_scan`] call).
|
|
///
|
|
/// Reads `family.masked` directly — [`super::subsample::SurvivingFamily`]'s
|
|
/// own docs explain why that's already the masked (`--free-loss`/
|
|
/// `--no-ambiguity`-applied) per-genome state, computed once upstream, the
|
|
/// same array [`algorithms::alignment::reduce_alignment`] reads too: this
|
|
/// is what guarantees `cardinality` agrees with what the alignment
|
|
/// actually shows for the same family/genome (a masked-away genome
|
|
/// contributes `0`, matching its `?`). No "variable families only" gate
|
|
/// on any field: see [`PairStats`]'s own docs for why every family reaching
|
|
/// this function already varies somewhere in the index, by construction.
|
|
pub(crate) fn reduce_pairwise(
|
|
survivors: &[SurvivingFamily],
|
|
partition: usize,
|
|
tally: &mut PairwiseTally,
|
|
dispersion: &mut PartitionDispersion,
|
|
) {
|
|
let n_genomes = tally.n_genomes;
|
|
let mut partition_subst: u64 = 0;
|
|
let mut partition_eligible: u64 = 0;
|
|
for family in survivors {
|
|
let single_form = |g: usize| -> Option<u8> {
|
|
family.masked[g].filter(|m| m.count_ones() == 1).map(|m| m.trailing_zeros() as u8)
|
|
};
|
|
let masked_cardinality = |g: usize| -> usize { family.masked[g].map_or(0, |m| m.count_ones() as usize) };
|
|
|
|
for i in 0..n_genomes {
|
|
let bi = single_form(i);
|
|
let card_i = masked_cardinality(i);
|
|
for j in (i + 1)..n_genomes {
|
|
let stats = tally.pair_mut(i, j);
|
|
|
|
if let (Some(bi), Some(bj)) = (bi, single_form(j)) {
|
|
partition_eligible += 1;
|
|
if bi != bj {
|
|
partition_subst += 1;
|
|
stats.subst[bi as usize][bj as usize] += 1;
|
|
stats.subst[bj as usize][bi as usize] += 1;
|
|
} else {
|
|
stats.same_all[bi as usize] += 1;
|
|
}
|
|
}
|
|
|
|
let card_j = masked_cardinality(j);
|
|
stats.cardinality[card_i][card_j] += 1;
|
|
if card_i != card_j {
|
|
stats.cardinality[card_j][card_i] += 1;
|
|
}
|
|
}
|
|
}
|
|
}
|
|
dispersion.record(partition, partition_subst, partition_eligible);
|
|
}
|
|
|
|
/// Accumulates a [`PairwiseTally`]/[`PartitionDispersion`] pair over a
|
|
/// [`super::scan::sampled_scan`] pass — a thin
|
|
/// [`super::scan::LayerReducer`] wrapper around [`reduce_pairwise`], so it
|
|
/// can run alongside any other reducer (e.g.
|
|
/// `algorithms::alignment::AlignmentReducer`) over the exact same batches
|
|
/// in one shared scan, instead of each needing its own `sample_index`
|
|
/// call.
|
|
pub struct PairwiseReducer {
|
|
tally: PairwiseTally,
|
|
dispersion: PartitionDispersion,
|
|
}
|
|
|
|
impl PairwiseReducer {
|
|
/// No `free_loss`/`no_ambiguity` of its own: `SurvivingFamily::masked`
|
|
/// has already applied them, once, upstream in
|
|
/// `subsample::sample_layer` — see that field's own docs.
|
|
pub fn new(n_genomes: usize) -> Self {
|
|
Self { tally: PairwiseTally::new(n_genomes), dispersion: PartitionDispersion::default() }
|
|
}
|
|
|
|
pub fn finish(self) -> (PairwiseTally, PartitionDispersion) {
|
|
(self.tally, self.dispersion)
|
|
}
|
|
}
|
|
|
|
impl super::scan::LayerReducer for PairwiseReducer {
|
|
fn reduce(&mut self, partition: usize, _layer: usize, survivors: &[SurvivingFamily]) {
|
|
reduce_pairwise(survivors, partition, &mut self.tally, &mut self.dispersion);
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod cardinality_masking_tests {
|
|
use super::{PairwiseTally, PartitionDispersion, reduce_pairwise};
|
|
use crate::siblings::FamilyMask;
|
|
use crate::siblings::algorithms::SurvivingFamily;
|
|
|
|
// Genome 0 is ambiguous (carries both A and C, mask 0b0011 = cardinality
|
|
// 2); genome 1 carries only A (mask 0b0001 = cardinality 1). The family
|
|
// is variable (both A and C are observed somewhere), so it contributes
|
|
// to `cardinality` regardless of masking. `masked` is what
|
|
// `subsample::sample_layer` would have computed upstream (see
|
|
// `SurvivingFamily::masked`'s own docs) — the masking decision is baked
|
|
// in here, not passed to `reduce_pairwise` itself anymore.
|
|
fn ambiguous_vs_single(masked: Vec<Option<u8>>) -> Vec<SurvivingFamily> {
|
|
vec![SurvivingFamily { family_idx: 0, kmer: obikseq::CanonicalKmer::from_raw_unchecked(0), mask: FamilyMask::EMPTY.with(0).with(1), masked }]
|
|
}
|
|
|
|
#[test]
|
|
fn cardinality_uses_raw_mask_without_no_ambiguity() {
|
|
let mut tally = PairwiseTally::new(2);
|
|
let mut dispersion = PartitionDispersion::default();
|
|
let families = ambiguous_vs_single(vec![Some(0b0011), Some(0b0001)]);
|
|
reduce_pairwise(&families, 0, &mut tally, &mut dispersion);
|
|
let counts = tally.cardinality_tally(&[]).counts;
|
|
assert_eq!(counts[2][1], 1, "genome 0's raw cardinality (2) should be tallied");
|
|
assert_eq!(counts[0][1], 0, "no masking requested — nothing should collapse to 0");
|
|
}
|
|
|
|
#[test]
|
|
fn cardinality_masks_ambiguous_genome_to_zero_under_no_ambiguity() {
|
|
// `masked[0] == None` matches what `sample_layer` would have
|
|
// computed for genome 0 at this site under `--no-ambiguity`: `?`
|
|
// in the alignment, i.e. no information — cardinality 0, not its
|
|
// raw 2.
|
|
let mut tally = PairwiseTally::new(2);
|
|
let mut dispersion = PartitionDispersion::default();
|
|
let families = ambiguous_vs_single(vec![None, Some(0b0001)]);
|
|
reduce_pairwise(&families, 0, &mut tally, &mut dispersion);
|
|
let counts = tally.cardinality_tally(&[]).counts;
|
|
assert_eq!(counts[2][1], 0, "raw cardinality must not leak through under --no-ambiguity");
|
|
assert_eq!(counts[0][1], 1, "the ambiguous genome should be masked to cardinality 0");
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod dispersion_tests {
|
|
use super::PartitionDispersion;
|
|
|
|
#[test]
|
|
fn fewer_than_two_partitions_gives_no_estimate() {
|
|
let mut d = PartitionDispersion::default();
|
|
assert!(d.estimate_alpha().is_none());
|
|
d.record(0, 5, 100);
|
|
assert!(d.estimate_alpha().is_none());
|
|
}
|
|
|
|
#[test]
|
|
fn zero_mean_rate_gives_no_estimate() {
|
|
let mut d = PartitionDispersion::default();
|
|
d.record(0, 0, 100);
|
|
d.record(1, 0, 100);
|
|
assert!(d.estimate_alpha().is_none());
|
|
}
|
|
|
|
#[test]
|
|
fn identical_per_partition_rate_gives_no_estimate() {
|
|
// Every partition sees exactly the same rate — variance across
|
|
// partitions is *below* the Poisson floor (it's exactly 0), so
|
|
// there's no over-dispersion to attribute to rate heterogeneity.
|
|
let mut d = PartitionDispersion::default();
|
|
for _ in 0..5 {
|
|
d.record(d.counts.len(), 10, 100);
|
|
}
|
|
assert!(d.estimate_alpha().is_none());
|
|
}
|
|
|
|
#[test]
|
|
fn clearly_over_dispersed_rates_give_a_finite_positive_alpha() {
|
|
// Two partitions with no substitutions, two with twice the pooled
|
|
// mean rate — far more spread than Poisson noise alone would
|
|
// produce at these counts, so a finite alpha should come out.
|
|
let mut d = PartitionDispersion::default();
|
|
d.record(0, 0, 100);
|
|
d.record(1, 20, 100);
|
|
d.record(2, 0, 100);
|
|
d.record(3, 20, 100);
|
|
|
|
// Hand-computed from the weighted-moments formula in
|
|
// `estimate_alpha`'s own docs: mean = 0.1, weighted variance =
|
|
// 0.01, Poisson floor = mean/mean_eligible = 0.001, alpha =
|
|
// mean² / (variance - floor) = 0.01 / 0.009.
|
|
let expected = 0.01 / 0.009;
|
|
let alpha = d.estimate_alpha().expect("clear over-dispersion should yield an estimate");
|
|
assert!(alpha > 0.0, "alpha should be positive, got {alpha}");
|
|
assert!((alpha - expected).abs() < 1e-9, "got {alpha}, expected {expected}");
|
|
}
|
|
}
|