Push wpvskkyozppn #78

Merged
coissac merged 15 commits from push-wpvskkyozppn into main 2026-10-15 10:38:59 +00:00
3 changed files with 70 additions and 10 deletions
Showing only changes of commit c226744405 - Show all commits

No files matched your search

+40
View File
@@ -291,6 +291,46 @@ Added so the measurements this file reports no longer need ad hoc scripts, and s
- `--shannon-summary`: `--shannon` and the summary share one traversal (`scan_family_entropies`, which replaces the loop that lived in `shannon_entropy_csv`); the summary keeps counts per (entropy kind, genomes present, genomes without ambiguity, 0.05-bit bin) instead of one line per family: 156 lines for the 46 706 families of the testbed (a few hundred thousand lines at most on any index, by construction: the bins are bounded). Checked: its `entropy15` counts sum to the number of variable families.
- Writers are in `cmd/phylo/stats.rs`; the library computes, the command layer formats and writes.
## Combined draw on Primulaceae: `--quartet --pair-balance --entropy 1` (measured 2026-10-13)
Command: `phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance -o primulaceae_quartet_balance --family-overlap --sibling-stats primulaceae`. One draw, 180 genomes, 5 000 000 columns requested and obtained (no shortfall; no `--sample-plan`, so per-layer quotas are not inspectable). The three options are on together: **effects of entropy, quartet and pair balance are not separable here**; the testbed result that the entropy kernel adds nothing at small budgets was not rechecked.
**Baseline.** The same statistics were recomputed with one script on the alignment fed to IQ-TREE (`primulaceae-gamma_alignment.fasta`, 5 003 414 columns). It reproduces the figures of the earlier sections (sample overlap min 4 / median 1 665 / max 387 057 / CV 1.57; sampling fraction $r$ quantiles 0.0025 / 0.0037 / 0.0055; 5.2 % median calls per genome), hence **is the entropy-drawn sample** (`--entropy 1 --entropy-sd 0.5`, section on entropy bias), not a uniform draw. The "uniform draw" label of the nesting section is inconsistent with this (to settle). So the comparison below is *entropy-only vs entropy + quartet + pair balance*, never against a uniform control. A call is any character other than `?`/`-`. With `--free-loss` absence is written `?` (`masking::masked_state`); `-` only occurs without it. Neither file contains IUPAC characters.
| | Baseline (entropy only) | Combined |
|---|---|---|
| Informative columns (≥ 2 bases, each in ≥ 2 genomes) | 21.4 % | 100 % |
| Carriers per column: median / share ≤ 4 / share ≥ 33 | 4 / 58.1 % / 8.0 % | 8 / 12.4 % / 11.3 % |
| Calls per genome, median; genomes < 1 % / < 2 % | 5.2 %; 8 / 23 | 8.1 %; 4 / 6 |
| Sample pair overlap min / median / max | 4 / 1 665 / 387 057 | 282 / 27 380 / 491 038 |
| Sample overlap CV; max/min | 1.57; 96 764 | 1.10; 1 741 |
| 4-way count (20 000 random quartets, 2 seeds): median | 200-202 | 3 738-3 797 |
| 4-way p1 / p10 | 3 / 66-67 | 93-95 / 1 092-1 104 |
| 4-way p90 / p99 | 3 000-3 644 / 65 700-67 300 | 14 500-15 600 / 82 900-84 600 |
| 4-way CV; share of quartets with 0 | 3.4-3.5; 0.11-0.14 % | 1.83-1.87; 0 % |
(p90/p99 move by 20 % between the two seeds: read them as orders of magnitude only.) The gain is at the bottom of the 4-way distribution (median ×19, p1 ×30); the top is unchanged. 4-way coverage is not stratified by clade depth here (needs the tree). "Informative" is measured over the 180 genomes: a column informative overall is not necessarily informative among the four genomes of a given quartet.
**Pair representation.** Index-wide overlap used for the comparison: baseline, the raw matrix (`primulaceae_phylo_family_overlap.csv`); combined, the `--no-ambiguity` matrix of the same run. Different conventions, so the two slopes are indicative.
| | Baseline | Combined |
|---|---|---|
| Slope of log(sample overlap) on log(index overlap) | 1.12 ($R^2$ 0.997) | 0.54 ($R^2$ 0.91) |
| Pairs with index overlap > 0 and no sample column | 0 | 0 |
| Median $r$ = sample/index, poorest decile of index overlap | 0.0027 | 0.059 |
| Median $r$, richest decile | 0.0053 | 0.0067 |
| Median $r$ by decile 1 → 10 | 0.0027 … 0.0053, rising | 0.059, 0.065, 0.067, 0.067, 0.069, 0.074, 0.023, 0.0082, 0.0077, 0.0067 |
The weight does not merely flatten: the poorest pairs are now about 9× better represented (relative to their pool) than the richest, where the baseline gave them about half. Deciles 1-6 are flat; the drop sits at deciles 7-10, around the gap of the index-overlap distribution already noted in the entropy-bias section (cause of that gap still unchecked). **Caveat on $r$:** the numerator only holds quartet-informative families while the denominator counts every variable family, and the matrices differ in convention, so $r$ is no longer a sampling fraction of one pool; the decile pattern is a measurement of the sample, not of an inclusion probability.
**Per-genome richness.** `--sibling-stats` now gives the variable families (size ≥ 2) per genome: 0.72 M to 93.9 M, median 34.5 M. Spearman between a genome's calls in the sample and its variable-family count: **0.974 in the baseline, 0.435 in the combined draw**: the baseline mirrored genome richness, the combined draw largely decouples from it (on the same genomes the correlation with *all* families is 0.66 and 0.32). Genomes poorest in variable families: *A. saussurei* 0.72 M (0.30 % of its families), *A. triflora* 1.26 M (0.72 %), *P. stricta var. obesior* 2.39 M (1.33 %), then *P. pumila* 5.2 M, *P. auricula* subsp. *serratifolia* 5.9 M (1.92 %), *P. carniolica* 6.8 M (1.77 %), *A. pyrenaica* PHA000543 8.4 M (1.46 %). Their calls in the sample went from 6 986 / 1 936 / 2 403 (saussurei, triflora, stricta) to 15 725 / 22 988 / 25 392, and carniolica from 4 937 to 52 310; *saussurei* stays the poorest genome in both. These genomes are data-poor, not empty: their families are mostly of size 1. This is the per-genome $c_i$ the normalisation of the data-poor section calls for.
Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 / 2 / 3 / 4 = 33 512 358 231 (98.43 %) / 479 910 992 (1.41 %) / 45 621 663 (0.13 %) / 9 725 201 (0.03 %). Sizes ≥ 2 sum to 535 257 856, matching the 535 257 857 of the entropy table within 1.
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment has not been given to IQ-TREE.
## Open questions
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
@@ -229,6 +229,9 @@ pub(crate) fn scan_layer_families(
let mut genome_mask = vec![0u8; n * ctx.n_genomes];
let mut outgoing: Vec<Vec<(CanonicalKmer, usize, u8, u8)>> =
(0..ctx.n_parts).map(|_| Vec::new()).collect();
// Batch positions of the families `selected` keeps — the only
// ones pass 2 reads the matrix for.
let mut picked: Vec<usize> = Vec::new();
// Pass 1: cheap, no matrix access — own base and this
// batch's cross-partition queries. Kmer and mask already in
@@ -239,8 +242,9 @@ pub(crate) fn scan_layer_families(
// hands back each one's annex-recorded layer alongside.
// Families outside `selected` (sampling only) still get a
// mask/base entry — pass 2 indexes uniformly by `i` — but
// never an outgoing query: that's the expensive part a
// sampled run exists to avoid paying for every family.
// are neither queried across partitions nor read from the
// matrix in pass 2: that's the expensive part a sampled run
// exists to avoid paying for every family.
for (i, entry) in batch.entries.iter().enumerate() {
let (kmer, mask) = (entry.kmer, entry.mask);
masks.push(mask);
@@ -250,6 +254,7 @@ pub(crate) fn scan_layer_families(
if !is_selected(&worker_selected, family_idx) {
continue;
}
picked.push(i);
for (member, layer) in mask.family_members(kmer, ctx.k) {
if member == kmer {
continue; // local — resolved below straight from `mat`, no lookup
@@ -269,14 +274,22 @@ pub(crate) fn scan_layer_families(
// each entry's kmer is mapped to its slot via `hash_batch`
// — a pure MPHF lookup, no evidence check, since these are
// this layer's own kmers, known members by construction.
let kmers: Vec<CanonicalKmer> = batch.entries.iter().map(|e| e.kmer).collect();
let slots = ctx.mat.hash_batch(&kmers);
let mut carries: Vec<Vec<bool>> = (0..ctx.n_cols).map(|_| Vec::new()).collect();
ctx.mat.fill_sub_matrix_carries(&slots, &mut carries);
for (g, col) in carries.iter().enumerate() {
for (i, &carries_it) in col.iter().enumerate() {
if carries_it {
genome_mask[i * ctx.n_genomes + g] |= 1 << bases[i];
// Only the `picked` families are read: a sampled or
// annex-restricted scan visits a handful of a layer's
// minorants, and a full-batch matrix pass would cost as much
// as `Selection::All`. `carries[g][j]` belongs to batch
// position `picked[j]`.
if !picked.is_empty() {
let kmers: Vec<CanonicalKmer> =
picked.iter().map(|&i| batch.entries[i].kmer).collect();
let slots = ctx.mat.hash_batch(&kmers);
let mut carries: Vec<Vec<bool>> = (0..ctx.n_cols).map(|_| Vec::new()).collect();
ctx.mat.fill_sub_matrix_carries(&slots, &mut carries);
for (g, col) in carries.iter().enumerate() {
for (&i, &carries_it) in picked.iter().zip(col) {
if carries_it {
genome_mask[i * ctx.n_genomes + g] |= 1 << bases[i];
}
}
}
}
@@ -49,6 +49,7 @@ use std::sync::Arc;
use obicompactvec::TempBitVecBuilder;
use obikidxcache::index_cache::IndexCache;
use obikindex::OKIResult;
use obisys::progress_bar;
use rand::RngExt;
use super::family_scan::{Selection, scan_layer_families};
@@ -263,6 +264,7 @@ pub(crate) fn sample_index(
// — needed before any layer's quota can be computed (each layer's share
// depends on every other layer's own mass).
let mut plans: Vec<LayerPlan> = Vec::new();
let pb = progress_bar("sampling_plan", cache.n_partition() as u64, "partitions");
for partition in cache.partitions() {
let n_layer = cache.n_layer(partition).unwrap_or(0);
for layer_idx in 0..n_layer {
@@ -293,7 +295,9 @@ pub(crate) fn sample_index(
mass,
});
}
pb.inc(1);
}
pb.finish_and_clear();
let total_mass: f64 = plans.iter().map(|plan| plan.mass).sum();
let mut report = SamplePlan { target: n, layers: Vec::with_capacity(plans.len()) };
@@ -308,6 +312,7 @@ pub(crate) fn sample_index(
// them, when `n` is smaller than the number of layers.
let mut cumulative_mass = 0.0f64;
let mut allotted = 0usize;
let pb = progress_bar("sampling", plans.len() as u64, "layers");
for (raw_index, mut plan) in plans.into_iter().enumerate() {
cumulative_mass += plan.mass;
let cumulative_quota = (n as f64 * cumulative_mass / total_mass) as usize;
@@ -337,7 +342,9 @@ pub(crate) fn sample_index(
resolved,
kept,
});
pb.inc(1);
}
pb.finish_and_clear();
Ok(report)
}