From 0d12494f1a05a075dd448a5ac739df79e81ccbf3 Mon Sep 17 00:00:00 2001 From: Eric Coissac Date: Mon, 12 Oct 2026 21:00:48 +0200 Subject: [PATCH] Refactor indexing, sampling, and entropy documentation This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts. --- DevDocMD/theory/chance_sharing.md | 69 +++ DevDocMD/theory/pair_coverage_sampling.md | 222 ++++++++++ .../theory/sibling_partition_unification.md | 60 +++ UserDocMD/molecular-ecology-modified.csl | 78 +++- UserDocMD/references.bib | 89 ++++ UserDocMD/theory/kmer_indexing/kmers.md | 240 ++++++++++- UserDocMD/usage/phylo.md | 3 +- mkdocs.yml | 3 + .../pseudo_alignment_nesting.cpython-314.pyc | Bin 0 -> 29161 bytes scripts/pseudo_alignment_nesting.py | 395 ++++++++++++++++++ .../src/siblings/algorithms/entropy.rs | 62 ++- .../src/siblings/extensions/sibling_ext.rs | 17 +- 12 files changed, 1182 insertions(+), 56 deletions(-) create mode 100644 DevDocMD/theory/chance_sharing.md create mode 100644 DevDocMD/theory/pair_coverage_sampling.md create mode 100644 DevDocMD/theory/sibling_partition_unification.md create mode 100644 scripts/__pycache__/pseudo_alignment_nesting.cpython-314.pyc create mode 100644 scripts/pseudo_alignment_nesting.py diff --git a/DevDocMD/theory/chance_sharing.md b/DevDocMD/theory/chance_sharing.md new file mode 100644 index 00000000..181b8d6b --- /dev/null +++ b/DevDocMD/theory/chance_sharing.md @@ -0,0 +1,69 @@ +# Chance sharing of a kmer across genomes (discussion) + +Null model for "a kmer is present in $r$ of $S$ genomes": how likely is this by chance alone? Extends the single-pair estimate $P_{\mathrm{chance}}\simeq G/4^k$ (see user doc, *K-mer specificity*). + +## Homogeneous case: binomial tail + +If every genome has the same per-genome chance probability $p$, the number $R$ of genomes containing a given kmer is $\mathrm{Bin}(S,p)$: + +$$P(R\geq r)=\sum_{j=r}^{S}\binom{S}{j}p^j(1-p)^{S-j}=I_p(r,\,S-r+1)$$ + +where $I_p$ is the regularized incomplete beta function. No elementary closed form; use $I_p$ directly. + +### Numerical evaluation of the sum + +- Term recurrence, one `lgamma` start: $t_{j+1}=t_j\cdot\frac{S-j}{j+1}\cdot\frac{p}{1-p}$. +- Sum the tail **not containing the mode** ($Sp$, not $S/2$). If $r>Sp$, sum $\sum_{j\ge r}$ directly and stop when terms are negligible. The complement $1-\sum_{j=0}^{r-1}$ (note: from $j=0$) suffers catastrophic cancellation when the p-value is tiny. +- Large $S$: normal approximation with continuity correction, or Poisson($Sp$) when $p$ is small. Unreliable in extreme tails. + +## Heterogeneous case: $p_i = f(G_i)$ + +The $G_i$ (number of kmers of genome $i$) are known and differ. With $p_i\simeq G_i/4^k$, the indicators "kmer present in genome $i$ by chance" are independent Bernoulli($p_i$), so $R$ follows a **Poisson-binomial** law, not a binomial: + +$$P(R=j)=\sum_{|A|=j}\prod_{i\in A}p_i\prod_{i\notin A}(1-p_i).$$ + +Everything lives in canonical space: $G_i$ is the number of distinct canonical kmers of genome $i$, and the kmer space has size + +$$N=\frac{4^k}{2}$$ + +exactly, since $k$ is odd (no kmer equals its own reverse complement, so canonicalization pairs the $4^k$ kmers perfectly). Exact per-genome probability: $p_i=1-(1-1/N)^{G_i}\simeq 2G_i/4^k$. This is twice the $G/4^k$ of the user doc, where $G$ is a number of positions on one strand. + +### Exact computation + +Dynamic programming over genomes, $d_j$ = probability of $j$ hits so far: + +$$d'_j=d_j(1-p_i)+d_{j-1}p_i.$$ + +For the upper tail only $d_0\ldots d_{r-1}$ are needed: $P(R\ge r)=1-\sum_{j32$ only ~19 % have $H\in[0.5,1.5)$, and the 129-180 class is 0.04 % of all families. Flattening deep-pair coverage works on little informative material whatever the weights. +- Consequence for a $\mu(n)$ that grows with $n$: it would not recover these families (80 % sit below 0.5 bit); what matters at large $n$ is the share of families in the band around 1 bit, not the position of the centre. Conditional weight $w_H(f)/E[w_H\mid n_f]$ remains an option to make the entropy filter neutral to $n$; $E[w_H\mid n]$ is now available from this table. + +Limit: the table has `n_genomes_present` but not the reach of the family, so the link to specific pairs is still not measured. + +## Site concordance factors on the Primulaceae tree (measured 2026-10-12) + +Command (IQ-TREE 3.1.3): `iqtree3 -t -s -m GTR+G4+ASC --scfl 100 --cf-verbose -T 5 --prefix concord`. With `-t` plus `-m`, IQ-TREE did **not** just annotate the reference tree: it ran a full ML search started from it (102 iterations, ~59 h wall-clock). `concord.treefile` differs from the input treefile by one bipartition (Robinson-Foulds 2), so sCF refer to the re-inferred tree. Without `-m`, ModelFinder is launched and required 157 GB (first attempt failed). The log also warns that all 180 sequences contain more than 50 % gaps/ambiguity. `sN` below = informative sites averaged over the 100 sampled quartets of a branch (alignment: 5 003 414 columns). + +**Deep Primula/Androsace interface: too few informative sites (data scarcity).** + +| Node | Tips (Primula / Androsace) | sCF | sDF1 | sDF2 | `sN` | +|---|---|---|---|---|---| +| 209 | 67 (43 / 24) | 27 | 47 | 26 | 17 | +| 210 | 66 (43 / 23) | 37 | 33 | 30 | 43 | +| 208 | 74 (50 / 24) | 54 | 16 | 29 | 190 | +| 211 | 24 (2 / 22) | 64 | 18 | 18 | 226 | +| 212 | 21 (2 / 19) | 30 | 38 | 32 | 184 | +| 216 | 10 (2 / 8) | 21 | 47 | 32 | 7 | +| 217 | 9 (1 / 8) | 43 | 20 | 36 | 7 | + +Quartets crossing these nodes have tens to a few hundred informative sites out of 5 M columns, the quantitative counterpart of the pair-overlap scarcity. Primula tips are spread over several nodes mixed with Androsace (e.g. 43 + 24 at node 209); no monophyletic Primula in this tree. + +**Within Androsace: conflict, not scarcity.** Nodes 283 (72 Androsace, `sN` 17 127), 292 (49 tips, 10 335), 305 (22, 11 980), 342 (11, 6 566) have thousands of informative sites but sCF 33-37 % with both alternatives nearly as supported: sites split almost evenly among the three quartet topologies. More sites would not fix this; cause not established (a poor fit of a tree model to intra-species divergence is one hypothesis, untested). Node 185 (23 Androsace): sDF1 53.6 % > sCF 30.6 %, a majority of the 893 sites prefer an alternative (real conflict). + +**Well-supported nodes have long branches:** node 296 (45 tips, length 0.23, sCF 90 %), 341 (0.21, 91 %), 330 (0.09, 85.5 %), 181 (0.099, 89.5 %); most branches of length 0.01-0.05 have sCF 33-45 %. Degenerate: node 186 (length 0.001, `sN` 0.37) and 187 (`sN` 2.2) carry no informative site. + +Median sCF shows no gradient with clade size (about 39-48 % in every class), so the clade-size class hides the effect; the scarcity is specific to the interface nodes above. + +**Implication for sampling.** The part a sampling design can address is the interface scarcity: select families carried by genomes from the four groups around each target node (a quartet-targeted selection) rather than flattening the whole pair matrix. Not assessed: how many such families the index holds (the `sN` of nodes 209 and 216 suggests few). For the intra-Androsace conflict no sampling strategy is suggested by these measurements alone. + +## Quartet coverage (measured 2026-10-12, `--quartet-test`) + +`scripts/pseudo_alignment_nesting.py --quartet-test --quartets N` draws random genome quartets and counts exactly the columns where all four carry a call, stratified by the size of the smallest tree clade containing the four. Cost: 20 000 quartets over 5.0 M columns in about 35 s total. Exact per-family quartet weighting is out of reach ($\binom{180}{4}\approx 42.3$ M quartets; one family carried by 100 genomes covers $\binom{100}{4}\approx 3.9$ M), so quartets serve as a *diagnostic* of a sample, not as a weight. + +Result on the current sample: 4-way count quantiles 0 / 1 / 10 / 50 / 90 / 99 / 100 % = 0, 3, 68, 200, 2 329, 66 504, 194 460 (CV 3.5, 0.1 % uncovered). By depth: median 131-209 for quartets inside clades ≥ 65 genomes versus 11 580-57 387 for clades 17-64. The ratio of the 4-way count to the smallest of the six pair overlaps has median 0.20-0.29 for deep quartets (0.53 for clades 33-64; 1 would be perfect nesting): pair overlap is only a loose upper bound of 4-way coverage, so flattening pairs does not by itself flatten quartets. 88 % of random quartets fall in the deepest class (root as written; strata with < 5 quartets are noise). Co-presence is not informativeness (a column must also differ among the four). Next measurement: repeat on a sample drawn with the pair weight and check whether the ratio moves toward 1. + +## Two kinds of family loss: sequencing dropout vs divergence (2026-10-12) + +- **Sequencing dropout (genome skimming, ~1× coverage).** About $e^{-1}\approx 37\%$ of each genome is uncovered, independently per family. Predicted for a family truly present in a clade of $m$ genomes: completeness ≈ 0.63-0.70 and clade-closed fraction $c^m$, independent of clade depth. Consistent with small clades (3-4 tips: completeness 0.71, 20 % closed vs $0.65^3=27\%$, $0.65^4=18\%$). Does not predict the observed decay at larger clades (0.45 at 5-8 tips, 0.21 at 17-32, 0.08-0.12 beyond). +- **Loss by divergence (user observation).** A family is a kmer whose 30 flanking positions (15 each side for $k=31$) are identical, with a variable central base; with per-site divergence $d$ the flanks survive with probability ≈ $(1-d)^{30}\approx e^{-30d}$ (0.74, 0.21, 0.042, 0.0012 for $d$ = 0.01, 0.05, 0.10, 0.20). Phylogeny-dependent and tied to local conservation, unlike sequencing dropout. Explains the loss of completeness with clade size, and sets a floor: no weighting can sample families that no longer exist for pairs whose $e^{-30d}$ is negligible. Surviving deep families are selected for conserved flanks, hence plausibly slow-evolving (fewer variable sites for deep splits; hypothesis, untested). +- The patristic distances of the IQ-TREE tree (1 to 3.5) come from an alignment with variable sites only, hence strongly inflated; the true $d$ is unknown and these numbers cannot be plugged into $e^{-30d}$. +- **Test (user's plan), done 2026-10-12: see the next section.** + +## Overlap vs Mash distance on the full index (measured 2026-10-12) + +Inputs: `phylo --distance mash --csv --shared-kmers --nj` on the 180-genome index (distance matrix, shared-kmer matrix, NJ tree) against the full-index `family_overlap` $O_{ij}$. Mash $D$: range 0.011-0.337, median 0.20. + +| Regression | Slope | $R^2$ | +|---|---|---| +| $\log O_{ij}$ vs $D$, all pairs | -26.95 | 0.979 | +| $\log$(shared kmers) vs $D$ | -31.87 | 0.984 | +| $\log O_{ij}$ vs $D$ + additive genome effect | -25.3 | 0.9999 | + +Spearman($D$, $O$) = -0.95; Spearman($D$, shared kmers) = -0.95; Spearman($D$, IQ-TREE patristic) = 0.81. + +Reading: + +- Overlap decays log-linearly with distance, slope about -27 versus -30 predicted by flank survival (previous section). **Not an independent confirmation of the flank model**: Mash $D$ is derived from kmer sharing and $O$ counts families defined by kmers, so both measure the same shared content. The kmer slope of -32 is $-k$ by construction of Mash (implementation check only). The ratio of the two slopes, 27/32 ≈ 0.85, is below the 30/31 ≈ 0.97 expected from flank survival alone; cause unknown, the gap is modest. +- Local slopes vary: -21 for $D\in[0.011,0.07]$ ($R^2$ 0.45), -28 for [0.07, 0.20] (0.96), -35 for $D>0.23$ (0.83). Half the pairs lie in $D\in[0.20,0.23]$, where the fit explains almost nothing ($R^2$ 0.26 and 0.07): a plateau where Mash has little resolution. +- Genome effects (log scale, sd 0.21): lowest `Androsace_saussurei--PHA010690` (factor ≈ 0.39), `Androsace_triflora--PHA000558` (≈ 0.62), `Androsace_vesulensis--PHA010698` and `Primula_stricta_var_obesior--TROM_V_149128` (≈ 0.65); highest `Androsace_pubescens--PHA010697` (≈ 2.8). Three of the four lowest are also among the genomes with the fewest calls in the pseudo-alignment (saussurei, triflora, stricta), pubescens has the most; consistent with a coverage effect at the extremes, correlation over all genomes not computed. +- NJ on Mash vs the IQ-TREE tree: Robinson-Foulds 182 of 354 (0.51 normalized); the Mash plateau near 0.2 probably limits NJ resolution for most pairs. +- Consequence for normalization: the residual $\log O_{ij}-(a+bD_{ij})$ measures coverage deviation at a given distance, a normalization of $O$ that needs no per-genome call counts. Untested on a weighted sample. + +## Data-poor genomes (measured 2026-10-12) + +Per-genome share of the 5 003 414 sampled columns with a call: median 5.2 %, max 14.3 %; 8 genomes below 1 % and 23 below 2 %. Fewest calls: `Androsace_triflora--PHA000558` 1 936, `Primula_stricta_var_obesior--TROM_V_149128` 2 403, `Primula_carniolica--PHA007185` 4 937, `Androsace_saussurei--PHA010690` 6 986, `Primula_pumila--TROM_V_135924` 17 393, `Androsace_maxima--PHA000534` 17 939, `Primula_auricula_subsp_serratifolia--PHA007184` 30 621, `Androsace_pyrenaica--PHA000543` 37 725. Three of them (saussurei, serratifolia, pyrenaica PHA000543) are among the tips whose placement looked aberrant, previously attributed to possible mislabelling or contamination. A simpler alternative is too little data to place them reliably; testable by removing them (`--exclude-genome`) and watching deep-node support. + +Consequence for flattening: raw $O_{ij}$ mixes evolutionary distance and per-genome coverage; flattening on it chases pairs involving data-poor genomes, for which no family exists. Normalizing, $O_{ij}/(c_ic_j)$ with $c_i$ the number of families carried by genome $i$ (per-genome counts from `--sibling-stats`), targets distance instead of coverage; the best normalization (smoothing of $c_i$, etc.) is not established. + +## Open questions + +- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy. +- Combination with entropy bias: not independent criteria if entropy correlates with reach (see section above); options: compensate the known entropy weight inside the pair weight, or apply the Gaussian inside reach strata. +- Pairs with near-zero $O_{ij}$ (suspected badly sequenced genomes) set the ceiling for everyone; `--min-shared-family`/`--exclude-genome` should be applied first so the floor is not dictated by anecdotal genomes. +- Effect on the SNP-distance estimators: a non-uniform draw over pairs biases per-pair eligible-family counts, but each pair's $p$-distance is a ratio over its own eligible families, so unbiasedness per pair is expected to hold. To confirm before relying on it. diff --git a/DevDocMD/theory/sibling_partition_unification.md b/DevDocMD/theory/sibling_partition_unification.md new file mode 100644 index 00000000..f06b63b7 --- /dev/null +++ b/DevDocMD/theory/sibling_partition_unification.md @@ -0,0 +1,60 @@ +# Sibling partition unification and masked canonical form (discussion) + +Status: **open, nothing implemented**. Records the state of the reflection on (a) routing the 4 members of a sibling family to one partition and (b) a canonical form that makes siblings directly comparable. Measurements are throwaway (not in the repo); figures are for $k=31$, $m=11$, 256 partitions, 20 bacterial genomes (98.4 M kmer positions), unless stated. + +## Problem + +A sibling family = the 4 kmers differing only at the central base (position $c=(k-1)/2$, $k$ odd). Routing is **not** a hash of all bases: a kmer goes to `minimizer().seq_hash() & mask`, and consecutive kmers of a superkmer share their minimizer. A family splits across partitions whenever the central base falls in (or changes) the selected m-mer. + +Fraction of families whose 4 variants are not all in one partition, standard minimizer: $4m/(k+2m+1)$ (fixed candidates $k-2m+1$ versus $4m$ variable ones). Measured: 81.4 % at $m=11$ (60.2 % at $m=7$, 46.8 % at $m=5$). This is the *theoretical* rate assuming all 4 members present; for two present members it is ≈ 68 %. Real families are overwhelmingly of size 1, so it only becomes cost where something probes variants. + +## Masked canonical form (Can3) + +Notation: `CentreA(K)` = K with its central base set to A; `RC(K)` its reverse complement. + +- Can1 = `min(K, RC(K))` (current). Each sibling canonicalised alone, so some may be flipped relative to the others. +- Can2 = orient by `CentreA(K)` vs `RC(CentreA(K))`. **Not a valid canonical form**: when the flanks are reverse complements of each other ($L=\mathrm{rc}(R)$, frequency ≈ $4^{-c}$, 1/1024 at $k=11$), $K$ and $\mathrm{RC}(K)$ get different canonicals (k=5: `ACAGT` and `ACTGT` both kept). Cause: `CentreA(RC(K)) ≠ RC(CentreA(K))` (the latter has centre T). +- **Can3** (retained): + +``` +Can3(K) = K if CentreA(K) < CentreA(RC(K)) + = RC(K) if CentreA(K) > CentreA(RC(K)) + = Can1(K) otherwise // self-revcomp family +``` + +Orientation is decided on the flanks only (centre ignored), tie broken on the full kmer. Gain: the 4 canonical siblings have identical flanks in the same orientation, so they compare directly with no reverse complement. Self-revcomp families have only 2 distinct canonical members (centres A/C; T/G fold onto them). The A↔T (C↔G) substitution in that context is undetectable by any double-strand canonical set, same as under Can1 — not a loss. + +Gain is simplification only: `central_canonical_neighbors` already canonicalises each variant independently and is correct (see `evolutionary_distances.md`, per-partition sweep). The earlier "masked-flank grouping" there is the Can2 idea; Can3 is its repaired form. **No performance gain by itself**, and it breaks existing indexes (canonical orientation changes for some kmers → MPHF hashes change; rebuild). + +Prerequisite if adopted: `Minimizer = CanonicalKmerOf` shares the canonical type with index kmers; an m-mer has no sibling family and must keep Can1. Decouple with a distinct `Minimizer` type (6 files outside `kmer.rs`: `routable.rs`, `packed_seq.rs`, `obiskbuilder/{rolling_stat,stream_iter,iter}.rs`, `obifastwrite`). + +## Routing that excludes the centre (scheme B) — rejected + +Candidates restricted to m-mers not overlapping the centre ($m\le c$): $S(j)=\min$ hash over m-mers $p\in[j,j+c-m]$, minimizer$(i)=\min(S(i),S(i+c+1))$. Family- and strand-invariant by construction (checked). But unitigs are partition-local (`pipeline.md`), and the hole moves with each kmer, so runs shrink: + +| $m$ | mean run, current | mean run, B | bases/kmer, current → B | +|---|---|---|---| +| 5 | 14.9 | 6.3 | 3.02 → 5.76 | +| 11 | 11.0 | 3.0 | 3.72 → 10.96 | +| 15 | 9.0 | 1.0 | 4.32 → 30.9 | + +Proxy: run = unitig (upper bound; real unitigs may merge). Independent of partition count. In `benchmark/run/global_index_count` (786 MB) `unitigs.bin` + idx ≈ 241 MB (31 %), so tripling them ≈ +60 % total index. Hashing the masked kmer per kmer (no minimizer) would cut every path at every kmer: worse. + +Structural tension: path coherence (long unitigs) needs a key shared by consecutive kmers, family coherence needs a key independent of each kmer's own centre, which is an ordinary base for its neighbour. Decoupling sequence storage from MPHF partitioning fails because a partition can only enumerate its kmers from its own unitigs (MPHF not invertible). Redefining the family on a terminal base would cut the split rate to ≈ 17 % but is not closed under revcomp and changes the SNP semantics. Lower $m$ reduces the split rate for free but deviates from the literature $k/m\approx 2.5$ optimum (balance, bucket size), unmeasured here — $m=11$ kept. + +## What the cross-partition cost actually is + +- Phylo-time: only families of size > 1 trigger remote lookups (a small share). +- Annex construction probes the 3 variants of **every** kmer (size-1 families included), ≈ 2/3 of probes crossing partitions; batch bounded to `#kmers(p)*3` per partition (not the whole index). +- Real remaining benefit of unification: the outer sweep over source partitions could run in parallel (it is sequential today for memory and for the `q >= p` dedup rule); plus code simplification in `family_scan.rs` / `annex.rs`. Not quantified. +- Unification would also allow an MPHF keyed on the family (one lookup, 4-bit member mask, union/intersection as OR/AND); needs the family-size histogram (1–4 members) to judge storage. + +## Open measurements (index not on this machine) + +1. Family-size histogram of the real index: `obikmer phylo --sibling-hist` (needs a built annex). +2. Time and peak RSS of `--sibling-annex` on the Vaccinium/Salix index (`Stage` "sibling_annex" traces time; peak RSS from outside). +3. Only if those show the sequential sweep or the probe volume is a real bottleneck: real-index test of scheme B or of a smaller $m$. + +## Decision state + +No code change decided. Can3 + `Minimizer` decoupling is the only self-contained step and is independent of the measurements above; it forces an index rebuild, so it is worth doing only if its simplification is wanted. diff --git a/UserDocMD/molecular-ecology-modified.csl b/UserDocMD/molecular-ecology-modified.csl index ed2b1ce9..d8551650 100644 --- a/UserDocMD/molecular-ecology-modified.csl +++ b/UserDocMD/molecular-ecology-modified.csl @@ -71,6 +71,54 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -85,9 +133,7 @@ - - - + + + + + + + + + + + + + @@ -150,6 +208,14 @@ + + + + + + + + diff --git a/UserDocMD/references.bib b/UserDocMD/references.bib index f547a3f4..23ca6fe4 100644 --- a/UserDocMD/references.bib +++ b/UserDocMD/references.bib @@ -768,3 +768,92 @@ pages = {535--563}, year = {1988} } + @ARTICLE{Le-Cam1960-pe, + title = "{An approximation theorem for the Poisson binomial distribution}", + author = "Le Cam, Lucien", + journal = "Pacific Journal of Mathematics", + publisher = "Mathematical Sciences Publishers", + volume = 10, + number = 4, + pages = "1181--1197", + month = dec, + year = 1960, + url = "http://dx.doi.org/10.2140/pjm.1960.10.1181", + doi = "10.2140/pjm.1960.10.1181", + issn = "0030-8730,1945-5844" + } + + @ARTICLE{Hoeffding1956-bx, + title = "{On the distribution of the number of successes in independent + trials}", + author = "Hoeffding, Wassily", + journal = "The Annals of Mathematical Statistics", + publisher = "Institute of Mathematical Statistics", + volume = 27, + number = 3, + pages = "713--721", + abstract = "Let $S$ be the number of successes in $n$ independent trials, and + let $p_j$ denote the probability of success in the $j$th trial, + $j = 1, 2, \cdots, n$ (Poisson trials). We consider the problem + of finding the maximum and the minimum of $Eg(S),$ the expected + value of a given real-valued function of $S,$ when $ES = np$ is + fixed. It is well known that the maximum of the variance of $S$ + is attained when $p_1 = p_2 = \cdots = p_n = p.$ This can be + interpreted as showing that the variability in the number of + successes is highest when the successes are equally probable + (Bernoulli trials). This interpretation is further supported by + the following two theorems, proved in this paper. If $b$ and $c$ + are two integers, $0 \leqq b \leqq np \leqq c \leqq n,$ the + probability $P(b \leqq S \leqq c)$ attains its minimum if and + only if $p_1 = p_2 = \cdots = p_n = p,$ unless $b = 0$ and $c = + n$ (Theorem 5, a corollary of Theorem 4, which gives the maximum + and the minimum of $P(S \leqq c)$). If $g$ is a strictly convex + function, $Eg(S)$ attains its maximum if and only if $p_1 = p_2 = + \cdots = p_n = p$ (Theorem 3). These results are obtained with + the help of two theorems concerning the extrema of the expected + value of an arbitrary function $g(S)$ under the condition $ES = + np.$ Theorem 1 gives necessary conditions for the maximum and the + minimum of $Eg(S).$ Theorem 2 gives a partial characterization of + the set of points at which an extremum is attained. Corollary 2.1 + states that the maximum and the minimum are attained when $p_1, + p_2, \cdots, p_n$ take on, at most, three different values, only + one of which is distinct from 0 and 1. Applications of Theorems 3 + and 5 to problems of estimation and testing are pointed out in + Section 5.", + month = sep, + year = 1956, + url = "http://dx.doi.org/10.1214/aoms/1177728178", + doi = "10.1214/aoms/1177728178", + issn = "0003-4851,2168-8990", + language = "en" + } + @ARTICLE{Gleser1975-yu, + title = "{On the distribution of the number of successes in independent + trials}", + author = "Gleser, Leon Jay", + journal = "Annals of Probability", + publisher = "Institute of Mathematical Statistics", + volume = 3, + number = 1, + pages = "182--188", + abstract = "Let $S$ be the number of successes in $n$ independent Bernoulli + trials, where $p_j$ is the probability of success on the $j$th + trial. Let $\mathbf{p} = (p_1, p_2, \cdots, p_n)$, and for any + integer $c, 0 \leqq c \leqq n$, let $H(c \mid \mathbf{p}) = P\{S + \leqq c\}$. Let $\mathbf{p}^{(1)}$ be one possible choice of + $\mathbf{p}$ for which $E(S) = \lambda$. For any $n \times n$ + doubly stochastic matrix $\Pi$, let $\mathbf{p}^{(2)} = + \mathbf{p}^{(1)}\Pi$. Then in the present paper it is shown that + $H(c \mid \mathbf{p}^{(1)}) \leqq H(c \mid \mathbf{p}^{(2)})$ for + $0 \leqq c \leqq \lbrack\lambda - 2\rbrack$, and $H(c \mid + \mathbf{p}^{(1)}) \geqq H(c \mid \mathbf{p}^{(2)})$ for + $\lbrack\lambda + 2\rbrack \leqq c \leqq n$. These results + provide a refinement of inequalities for $H(c \mid \mathbf{p})$ + obtained by Hoeffding [3]. Their derivation is achieved by + applying consequences of the partial ordering of majorization.", + month = feb, + year = 1975, + url = "http://dx.doi.org/10.2307/2959277", + doi = "10.1214/aop/1176996461", + issn = "0091-1798,2168-894X" + } diff --git a/UserDocMD/theory/kmer_indexing/kmers.md b/UserDocMD/theory/kmer_indexing/kmers.md index 36befb07..5ce36ac2 100644 --- a/UserDocMD/theory/kmer_indexing/kmers.md +++ b/UserDocMD/theory/kmer_indexing/kmers.md @@ -22,24 +22,102 @@ This gives the hard representation limit: $k \leq 32.$ Furthermore, OBIKmer only ### K-mer specificity -A simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are +One simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are $4^k$ possible k-mers, so a particular k-mer is expected to occur once every $4^k$ positions on average. In a sequence of length $G$, the expected number of occurrences of a particular k-mer is approximately $G \cdot 4^{-k}$. When k is large enough to ensure $4^k >> G$, the chance of a second occurrence of the same k-mer becomes close to zero. Consequently, observing the same k-mer twice can only be explained by a homology relationship, except for low-complexity k-mers, which are overrepresented in genomes. + +### Estimating False discovery rate on homology inference + +OBIKmer uses k-mers long enough that their occurrence in multiple indexed genomes provides evidence of homology. The corresponding false discovery rate (FDR) can be expressed in terms of the probability of observing a k-mer in at least $R$ of the $S$ indexed genomes under a purely random model. + +#### The random model + +The random model, or null hypothesis, assumes that the $S$ genomes are unrelated and that each genome is an independent random sequence in which each nucleotide occurs with equal probability. Rejecting this hypothesis for a given k-mer indicates that its presence in multiple genomes is unlikely to be explained by chance alone, providing evidence for homology. + +For a genome of size $G$, the number of occurrences of a given canonical k-mer $K$ is approximated by a Poisson distribution with parameter + +$$ \lambda = 2\frac{G}{4^k}, $$ + +which is also the expected number of occurrences under the random model. The factor of two accounts for k-mer canonicalisation, which halves the k-mer space when $k$ is odd. + +The probability of observing at least one occurrence of this k-mer is therefore $$ -4^k +\begin{aligned} +P(n_K > 0) &= 1-e^{-\lambda} \\ +&= 1-e^{-2G/4^k}. +\end{aligned} $$ -possible k-mers, and a particular k-mer is therefore expected to occur once every $4^k$ positions on average. In a sequence of length $G$, the expected number of occurrences of a particular k-mer is approximately +Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore + $$ -E[n_K] \simeq \frac{G}{4^k}. +P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right){S-j}. $$ -When $4^k\gg G$, this quantity is also a good approximation of the probability that an observed k-mer will occur at least once by chance elsewhere in a sequence of length $G$. It therefore provides a simple measure of the risk of a chance recurrence of an observed k-mer: +This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where $$ -P_{\mathrm{chance}}(k,G) \simeq \frac{G}{4^k}. +p = 1-e^{-2G/4^k}. $$ +Different genome sizes + +When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean + +$$ \bar G = \frac{1}{S}\sum_{i=1}^{S}G_i. $$ + +This gives a common probability + +$$ \bar p = 1-e^{-2\bar G/4^k}, $$ + +and the number of genomes containing the k-mer can then be approximated by a binomial distribution: + +$$ P(X\geq r)=\sum_{j=r}^{S}\binom{S}{j}\bar p^j(1-\bar p)^{S-j}. $$ + +This approximation is conservative for the upper tail. Hoeffding [-@Hoeffding1956-bx, p. 717, Eq. 26] shows that, for a fixed mean, the binomial upper tail is at least as large as the corresponding Poisson-binomial tail for + +$$ r\geq S\bar p+1. $$ + +Using the mean genome size therefore provides a conservative estimate of the FDR. + +When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$. + +### Exact tail probability + +With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time. + +Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does: + +$$ +\left \{ +\begin{aligned} +d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\ +d_0&=1 +\end{aligned} +\right . +$$ + +before processing the first genome. + +The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then + +$$ +P(X\geq r)=1-\sum_{j=0}^{r-1}d_j. +$$ + +Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$. + +When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean + +$$ +\lambda=\sum_{i=1}^{S}p_i. +$$ + +The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe]. + + +******* + This probability decreases by a factor of four for every additional nucleotide in the k-mer. Conversely, it increases linearly with the amount of sequence searched. This simple model already shows why very short k-mers provide limited specificity for genome-scale analyses. With $k=11$, @@ -189,34 +267,150 @@ Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nomb Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$. -## FDR +## Chance sharing across genomes and the FDR of homology -C'est exactement la bonne question à se poser — ni la p-value par hit, ni la probabilité globale « y a-t-il au moins une collision quelque part », mais **la fraction de tes appels d'homologie qui sont du bruit**. C'est un problème de type FDR (*false discovery rate*), et il se construit directement à partir des deux briques qu'on a déjà : +The formulas above ask how likely a match is between two sequences. The question OBIKmer actually has to answer is different: a k-mer is found in $r$ of the $S$ indexed genomes, and one wants to know whether this sharing can be explained by chance or whether it is evidence of homology. This section builds a null model for that question and turns it into a false discovery rate (FDR) for the call "a k-mer shared by at least $r$ genomes is homologous". + +### The null model + +The null hypothesis is that the $S$ genomes are unrelated, that is, that each one is an independent random sequence. Rejecting it for a k-mer means that chance alone cannot explain why the k-mer is shared, which is the signal of homology one is looking for. + +The model only records whether a k-mer is present in a genome, not how many times. Two consequences follow. First, repeats inside a genome are irrelevant, and the size $G_i$ of genome $i$ is its number of **distinct canonical k-mers**. Second, a repeat family shared between genomes is paralogy, which is a form of homology; it is a true positive and not a false one. The only k-mers that could be shared by composition rather than by descent are low-complexity ones, which OBIKmer removes beforehand with its entropy filter. + +Everything takes place in canonical space. Because $k$ is odd, no k-mer equals its own reverse complement, so the $4^k$ k-mers pair up perfectly and the space contains exactly $$ -\widehat{FDR}(r) = \frac{\text{nombre de k-mers *attendus par hasard* avec }R\geq r}{\text{nombre de k-mers *observés* avec }R\geq r} +N=\frac{4^k}{2} $$ -**Le numérateur** (bruit attendu) : c'est la généralisation, au seuil $\geq r$ (pas seulement « partagé par exactement ces $r$ génomes »), de la quantité pont qu'on vient d'identifier — le nombre de k-mers testés $M$ multiplié par la probabilité par-mot sous le null : +canonical k-mers. A given canonical k-mer is present in genome $i$ by chance with probability $$ -E[\text{faux positifs}] \simeq M \cdot P_0(R\geq r) = M\sum_{j=r}^{S}\binom{S}{j}p^{j}(1-p)^{S-j} +p_i = 1-\left(1-\frac1N\right)^{G_i}\simeq \frac{G_i}{N}=\frac{2\,G_i}{4^k}. $$ -avec $p\simeq G/4^k$ (ou la version Poisson-binomiale à $G_i$ inégaux) et $M$ le nombre de k-mers canoniques réellement testés (pas $4^k/2$ théorique, mais le nombre de k-mers **effectivement présents** dans ton index — c'est cette correction, plutôt que le $4^k$ théorique, qui rend l'estimation réaliste, puisque la plupart des $4^k$ mots possibles n'apparaissent jamais dans de vrais génomes). +This is twice the $G/4^k$ used earlier in this page, because there $G$ is a number of positions on one strand, whereas $G_i$ here counts canonical k-mers, for which both strands are already merged. -**Le dénominateur** (bruit + vrai signal, observé) : le nombre de k-mers de ton index dont le masque de présence a une cardinalité $\geq r$ — une donnée que `FamilyMask`/`SiblingAnnex` dans `obikphylo::siblings` te donne déjà directement, sans aucun calcul supplémentaire. +Let $R$ be the number of genomes, out of $S$, that contain a given canonical k-mer by chance. Genomes are independent under the null, so $R$ is a sum of independent presence indicators, each with its own probability $p_i$. -**Ce que ça donne concrètement.** Tu n'as besoin que de trois nombres, tous déjà disponibles ou calculables : -1. $M$ = cardinalité totale de ton set de k-mers indexés (mesure directe de l'index) -2. $N_{obs}(r)$ = nombre de familles/k-mers avec présence $\geq r$ (mesure directe, déjà dans `FamilyMask`) -3. $P_0(R\geq r)$ = calculé depuis le modèle nul (le $p$ de l'approche « naïve » du tour précédent) +### Exact tail probability -et $\widehat{FDR}(r) = M\cdot P_0(R\geq r) / N_{obs}(r)$ te dit directement : « sur les k-mers que j'ai retenus comme preuve d'homologie au seuil $r$, telle fraction (%) est attendue être du pur hasard ». +When genome sizes differ, the $p_i$ differ, and $R$ follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be hopeless, since there are up to $2^S$ subsets. The distribution is instead obtained by adding the genomes one at a time. Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does: -**Pourquoi c'est mieux que ce qu'on avait avant** : -- Le $P(K_{r,s}\geq k)$ de Karlin-Ost (tour précédent) répond à « y a-t-il *au moins un* faux positif quelque part » — binaire, ne dit rien sur combien. -- La p-value par hit répond à « ce hit précis est-il suspect » — mot par mot, ne donne pas de vue d'ensemble. -- Le $\widehat{FDR}$ répond directement à ta question : **quelle proportion** de l'ensemble retenu est due au hasard — c'est la quantité qu'on utiliserait pour dire « à ce seuil $r$, mon jeu de zones d'homologie candidates est fiable à 95 % » ou pour choisir le $r$ minimal qui ramène le $\widehat{FDR}$ sous un seuil acceptable, à $k$ fixé par la contrainte de représentation (64 bits). +$$ +d'_j = d_j\,(1-p_i) + d_{j-1}\,p_i, \qquad d_0=1 \text{ before the first genome.} +$$ -Un point à trancher avec toi avant d'aller plus loin : pour $M$, préfères-tu le nombre théorique de k-mers canoniques possibles ($4^k/2$) ou le nombre réel de k-mers distincts dans ton index (qui donnerait un $\widehat{FDR}$ beaucoup plus conservateur et réaliste, mais nécessite de le lire depuis l'index au moment du calcul plutôt que de le dériver de $k$ seul) ? +The table $d$ only needs to remember how many genomes were hit, not which ones, and this brings the cost down to $O(S^2)$. The result does not depend on the order in which the genomes are processed. It does not depend on the k-mer either, so it is computed once per index. The upper tail is then + +$$ +P_0(R\geq r) = 1-\sum_{j=0}^{r-1} d_j . +$$ + +Only $d_0,\dots,d_{r-1}$ are needed, so the cost is $O(S\,r)$. + +### Approximation with equal genome sizes + +If all genomes are given the same size, taken as the mean + +$$ +\bar G=\frac1S\sum_{i=1}^{S}G_i, \qquad \bar p=\frac{2\bar G}{4^k}, +$$ + +then $R$ follows a plain binomial distribution and the tail is + +$$ +P_0(R\geq r)\simeq\sum_{j=r}^{S}\binom{S}{j}\,\bar p^{\,j}(1-\bar p)^{S-j}. +$$ + +This sum is the regularized incomplete beta function $I_{\bar p}(r,\,S-r+1)$, available in standard statistical libraries. If it is summed by hand, the terms are obtained one from the next by $t_{j+1}=t_j\,\frac{S-j}{j+1}\,\frac{\bar p}{1-\bar p}$. + +A theorem of Hoeffding [@Hoeffding1956-bx] shows that this approximation errs on the safe side. For a fixed mean, the Poisson-binomial distribution is more concentrated than the binomial one, so for $r\geq S\bar p+1$ the binomial tail is never smaller than the exact tail. Using the mean genome size therefore overestimates the chance of sharing and can only make the FDR more pessimistic. Gleser [@Gleser1975-yu] refines this result: replacing the $p_i$ by any partial average of themselves, not only by their full mean, also moves the upper tail towards the binomial one, provided $r-1\geq S\bar p+2$. Grouping genomes of similar size is therefore safe in the same sense. What it loses is power when genome sizes differ by orders of magnitude, since one large genome then weighs as much as several small ones in $\bar p$. + +When all $p_i$ are small, which is the case for large $k$, the number of hits is also well approximated by a Poisson distribution of mean $\lambda=\sum_i p_i$, with an error bounded by $\sum_i p_i^2$ [@Le-Cam1960-pe]. + +For the numerical evaluation, the tail that does not contain the mode $S\bar p$ should be summed directly. When $r>S\bar p$, computing the sum from $j=r$ upwards is accurate even for very small probabilities, whereas computing $1$ minus the sum of the lower terms loses all precision once the tail falls below the machine epsilon. + +### False discovery rate + +The FDR of the call "present in at least $r$ genomes" is the fraction of retained k-mers expected to be pure chance: + +$$ +\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P_0(R\geq r)}{n_{\geq r}}\right). +$$ + +The numerator is the number of k-mers expected to reach $r$ genomes by chance if the genomes were unrelated. Each of the $N$ canonical k-mers has probability $P_0(R\geq r)$ of doing so, and no further correction for multiple testing is needed because the number of tests is already in the product. The denominator $n_{\geq r}$ is the number of distinct k-mers actually observed in at least $r$ genomes, which is read directly from the index. The observed count mixes chance and true homology, so the ratio is an upper bound on the true FDR: it implicitly assumes that all k-mers are null, and it never underestimates the noise. The minimal $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls below the accepted level, for example 5 %, is the sharing threshold to use. + +As an illustration, take $k=21$ (so $N=2.2\times10^{12}$), $S=10$ genomes and $G_i=10^9$ distinct canonical k-mers each, hence $p=4.5\times10^{-4}$. The number of k-mers expected to be shared by chance is: + +| $r$ | $N\,P_0(R\geq r)$ | +| ---: | ---: | +| 2 | $2\times10^{7}$ | +| 3 | $2.5\times10^{4}$ | +| 4 | $\approx 20$ | +| 5 | $\approx 0.01$ | + +Chance alone produces millions of pairwise shared 21-mers at this genome size, a few tens of thousands shared by three genomes, and almost none shared by five or more. Whether a given $r$ is acceptable then depends on how many k-mers the index actually reports at that level. + +### Limits of the model + +- The FDR is computed per k-mer. Neighbouring k-mers of a shared region overlap and are not independent, so a single homologous region contributes many correlated k-mers. A per-region FDR would count regions instead of k-mers and is not covered here. +- The probability $p_i$ assumes equiprobable, independent nucleotides. The composition biases discussed earlier in this page violate this assumption, and the resulting error is not accounted for. + +--- + +Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`. + +## 1. FDR d'un ensemble de k-mers sélectionnés + +**Estimation de l'espérance du nombre de faux positifs.** Soit $M$ le nombre de k-mers canoniques testés, $M = 4^k/2$ pour k impair. Soit $q(r)=P(X\ge r)$ la queue du modèle nul. Le nombre attendu de k-mers présents par hasard dans au moins $r$ génomes vaut + +$$ +E[\mathrm{FP}(r)] = M\,q(r). +$$ + +Cette égalité est exacte par linéarité de l'espérance, même si les k-mers chevauchants sont dépendants. La dépendance n'affecte que la variance, pas l'espérance. Seule l'approximation de Poisson par génome reste une approximation. + +**FDR.** Soit $N(r)=\#\{i : r_i\ge r\}$ le nombre observé de k-mers couvrant au moins $r$ génomes. Alors + +$$ +\mathrm{FDR}(r)=\min\!\Big(1,\ \frac{M\,q(r)}{N(r)}\Big). +$$ + +Cette estimation est légèrement conservatrice. Elle suppose $\pi_0=1$, c'est-à-dire que presque tout l'espace $M$ est nul, ce qui est vrai pour k grand. + +**Ensemble hétérogène en $r_i$.** Si le jeu n'est pas défini par un seuil unique, on utilise le local fdr par classe de couverture : + +$$ +\mathrm{fdr}(r)=\frac{M\,P(X=r)}{N_r},\qquad N_r=\#\{i : r_i=r\}. +$$ + +Le FDR de l'ensemble est alors la moyenne $\frac1N\sum_i \mathrm{fdr}(r_i)$. Pour un ensemble défini par seuil $r_{\min}$, cela redonne exactement la formule précédente. + +**Remarque.** $M\,p\approx G$, d'où $E[\mathrm{FP}(r)]\approx\binom{S}{r}\,G^r/M^{r-1}$ en régime de petits $p$. Cela donne une lecture directe de la quantité. + +## 2. Markov d'ordre 0 avec fréquences inégales + +Soit $\pi_b$ la fréquence du nucléotide $b$. Pour un k-mer $w$ de composition $n=(n_A,n_C,n_G,n_T)$ avec $\sum n_b=k$, on a $P(w)=\prod_b \pi_b^{n_b}$. + +**Attendu par génome.** Le k-mer canonique $K=\{w,\overline w\}$ est vu quand le brin direct porte $w$ ou $\overline w$, donc + +$$ +\lambda_K = G\,\big(P(w)+P(\overline w)\big)=G\Big(\prod_b\pi_b^{n_b}+\pi_A^{n_T}\pi_T^{n_A}\pi_C^{n_G}\pi_G^{n_C}\Big). +$$ + +Cela ne dépend que de la composition. Si $\pi_A=\pi_T$ et $\pi_C=\pi_G$, on retrouve $2P(w)$. + +**Faux positifs.** Le test n'est plus homogène : chaque composition a sa propre $p(n)=1-e^{-\lambda(n)}$. Il faut donc sommer sur les classes de composition (un k-mer et son complément inverse partagent le même $\lambda$, d'où le facteur ½) : + +$$ +E[\mathrm{FP}(r)]=\frac12\sum_{n}\binom{k}{n_A,n_C,n_G,n_T}\,P\big(X\ge r\mid p(n)\big). +$$ + +Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Elle est donc légère avec la binomiale de la moyenne, ou avec une Poisson-binomiale par composition si les $G_i$ diffèrent. Si les $\pi_b$ varient d'un génome à l'autre (GC différent), les $p_i(n)$ se construisent par génome. + +**Effet attendu.** La moyenne de $\lambda_w$ sur tous les k-mers reste $2G/4^k$, car $\sum_w P(w)=1$. Dans le régime de petits $p$, le terme dominant $\binom{S}{r}p^r$ est convexe en $\lambda$. Par Jensen, le modèle équiprobable est donc le minimum de $E[\mathrm{FP}]$, et la déviation de composition ne peut qu'augmenter le FDR. Les k-mers riches en AT ou GC dominent la somme, ce qui rejoint ton remarque sur la faible complexité. C'est un résultat sur le terme dominant, pas une preuve pour la queue exacte. + +**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total. + +Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ? diff --git a/UserDocMD/usage/phylo.md b/UserDocMD/usage/phylo.md index 6041c20e..f4b188ed 100644 --- a/UserDocMD/usage/phylo.md +++ b/UserDocMD/usage/phylo.md @@ -111,6 +111,7 @@ Without `--subsample`, every variable family (family size ≥ 2) is used. With ` | Column | Meaning | |---|---| +| `partition` | the index partition that holds the family | | `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes | | `family_idx` | the family's position within that layer | | `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count | @@ -118,7 +119,7 @@ Without `--subsample`, every variable family (family size ≥ 2) is used. With ` | `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) | | `n_genomes_present` | how many genomes the entropy was computed over | -Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful for choosing `--entropy`/`--entropy-sd` values before a full run. +`--shannon` always scans every non-monomorphic family of the whole index; `--subsample`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample. ### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families diff --git a/mkdocs.yml b/mkdocs.yml index 6f493d68..ee06c0fe 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -36,7 +36,10 @@ nav: - Entropy filter: theory/entropy.md - Minimizer selection: theory/minimizer.md - Partitioning architecture: theory/indexing.md + - Chance sharing across genomes (discussion): theory/chance_sharing.md - Central-position SNP distance (discussion): theory/evolutionary_distances.md + - Sibling partition unification (discussion): theory/sibling_partition_unification.md + - Pair-coverage-biased family sampling (discussion): theory/pair_coverage_sampling.md - Implementation: - SuperKmer: implementation/superkmer.md - Kmer: implementation/kmer.md diff --git a/scripts/__pycache__/pseudo_alignment_nesting.cpython-314.pyc b/scripts/__pycache__/pseudo_alignment_nesting.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..b5375ee02267cc7abc1c27b39a93ba4bb6b2c344 GIT binary patch literal 29161 zcmcJ133wYvmR>h5;v@<162+@YN+coi5bw*9Oi`i^QWOPJP1&YMgCt17A_=GgN+Ns= zXEWIWWqSn4)(F}eub>$_g0uDx{beVncXmI^yE8kQ>`WFk2#W(NSx@3G$>jKATTUEL z@+JAJ8x1rl=-Az)B{sUds$RXSdiCC`SFhgobc2?{_3E)-K2y4fqW&#;5ieQn;qD9` z7AXtmrba0XeUNt3{IAR{s7cgYNS)A?QcbaHN&vf@|1)c?XOY}vowd&_Fpe`eZpP|ltaFph#Ef;)Rx>f< zaLsczrmAet=5pKTrpwHXV~VjdgZ%@H(`B2Vbkta9?9+3zwmEmLR=dwSagvFDU?vF%u`P zoOQx&;|QEN2eQmD)^V3@Zoe7 zoSbdSevX;8p8?SE1*XPf?$9#KW5V}cV*+Ypu@0!6aGha9BvDGhcGl|V?B_a|x%t^~ zAcJlYFm*9Nm9qid1Fvz1`(z9h+JuHL#F`E4?bd0>oOOoT$?T|RT-Mo#0-JEmp(Sp+ z16|=_o_37ePt5`+=gEZ`2UAlcG^ZxkoTtswrp5+RXk!dxC&~%~?y`H(t8UKfo}cYt zPCCxwyTfIhWZWlX1Bmf@SW~T_nEAO$041f|oXw`y_6zjG%-XDkqB@u<2Zs(2Fy;Hh zj#@A@q)4pCT{br}Sk27fr@Ly<%!~urNt@O+3t|B{9&(233}C$$=*+nijawizmLPp#RAs}YSP(Hn8G;|5 zBZjmKL~8D2W*ld=JX{xP;v6%{Ih>ApH-lC>t?r4F3j{@%jXPuIN6kLxvQP5O5rBbM zhhlw2P~aR6H!8>2p5^RrG=#Cw@iq6`&e|tVF;n&#(#7Zn7ls@4OmmKTCn?F3xVB0J zfKCF~Ih%d@Cd|Z>_L)i2Iiw%~!%rk9!5S1g zW1m7{()m?2HRPp+yqG(fNry0&WI9g}!XrarWjI0$POF_`s=BK8n86!d4lSQ9OkYpvrkj8GBy2Ed)dIAEIbZHP@Mb1gII z#yEl_PPnQ`DrxB4q+=FM=Q*BhejL*nG)TD25J-@x2_7pj5qt-VQ_PXEu07qLO!6z` z9XpwUs&3{vhV&T9_o2LNewLXV>teRW3z08;LH<*>w0spAm`7t)2Pk93~r*v^5+&n(n3ya<^Ca$3&joUu<>-5^6-jmtJ; zn*b~4sS-q07aytkcusK`03tCQXXiv&hT(BJNFl8NfN)GF=a{ff60wH)f>{VY3h~3Z z`FxB6q6MuWgKi@tM5O%F^Pn4WB9ZuG?+N-x!>4AfWU>;{m(mL%TwoDP)rye#X<(hU z?Zy<2eRDDWBCCmiAv0)_v!YG&b9Qje*_nk-o_iAFujbo!ACZJ=XOI)qUKqj+a)+=w zGA3wiy!S0OrvuHL)Dn>plLI0%h+%{g135vMRq+ECBa(+OWikR%Ou#V&dYHs4;_Tz| zguh^{AnSR~JU_?x4p0;Op_(ThVfc|bhLpCc=B$t|4{9e)#bJ_LzaE}dJgvOgGvIzBw9*bx;`c0?0sE(_Vt}$@QGJ_ zh^Di`(b{lAS~k{kJb{veVn;F!J8+#y${Xk8BqRSK>Nb4l9ehpv zcb=2(sI5;FmvSCboXp)C%v~2M;>$ssn8f60Cbg0bE*plm9|-hMh>9;!Ow;furzpi; z3%**@tZJ*qs$HZ;VoEDT4M=mxzLr9b9Z`r`EV9;E9!DBWXHq>zigL;<^1-wb0-e8D zSraWhRp6&Li=s4CF;$Cy7R9{Wp>}8;8a9npvl>?0s36MqjM^!;D4dEBjZluFSjwVm ziq+IfkJ0n8dHGmey>=>DCF=eb>Q=F;0i9SX@oPyFaj1-aoa`e?I;*gu4*beX@hs*k zp$63hhD2%ciB(zD)tkjP^iM|OS$q;z{!uC^C;DaMCFV@jT1z<uy_ZshS`o)VO{2V=VyNwu*pJf~l|srcE1`-| z2l?06+$k@j?6g_q>9*M1^V}R$HDevO%|J9btrJAN#|7Frksf5W3;Yr+M_k)L37wmE zpETG0mIC#>Nk^5F?ggi9sDC0xOgxH&$VjK$J&CW6DIS4%sR1QlGU>)rVgn^zp-Rfa zzgTFI+@18GVhSWuNJ)s47NIy4hj*p=KB#Cri}hh0Uyfg$60igMduu|c%PmSN3F#?| zJQYlQR}3ojRD)8i9ptM_MVgYfWu~lA#?sOntKz+qfNYVKQ)bm&Qk)}Ppx>kEhz%9v zQ81-Dncw~&TxLyFHs_3LiH0yfKQ(3JqKao6_PMBX-ahAUi^?&&QMqfu6;;^I+1*i< z6+C}#GO8v7LWshwiYkeo;X*Fk=_u4hXh80mvy3UaA9tu4qWk!)jm@P!LhNr*cz_RH30?D?W%i#5Q!?MX69Wd zqh*}UI*GL_rT!lf7UCfIs%DC~nWUB1?)8CSx{(iHLBm6X~^w>BDI)2>Yn2E|A++lTpQt z9cn+}C|nM?<&s+-ZZ5*1D=K(7_BbwX6W*g)DLR*{K}s7b@)>Ub6J2V|y0YogCV%G{F8x(*G^uWyE+#xt__}!6xZJ_Zd)yG3m12+Wp=J_+I&g*NoLL! z{YAZh=-QK4pZwm)N159`&!BShzLHbg&8S>$_%LnLuhj;x?Ik^k;MImJ?U&m9Y&dI6 zU@D}m4k@a?%0o^d_B$7mK_3+F(e2Gu{xny&w?dP|{0JlC<@a}Sw`#3AtA3G}tF;v5 zs?MQzXdz>1u5^?di8IXvtK+&^EvsYojdEh*g3MJ8H??8#_R*7nNEpubhSiAI_15kkX$g4@g}fid>&eq5|bEll%XxUB$#Q3H?hiunal|tVjw23 zP*MD1rR5AX43}steR*W_kpKQFtpz1c&wE?){hHVrVDJMkbVxPY7Q1hUK`>d)QG)}S9KWJD6I>wPM8m1eC3TlyxPRT+zT7cWNrV1q`S$~ z7j&QR;Nda(A~i@4#HV>8qb1&X&U{Be0^jdsn0(DIdi-54zYtOv@sbj1 zi`Jlt(p6EqI!c>8ijHb?XO!;n$obz+p#cgWcR?`9oen2a2^?0p*$`E@ZRgxXIzlQ& z<*@HWX?v8O;~Ifgl%9wx_;x5Kh?y@+KO>02dL(iU1bNaWf+!^NFfI-C(spzEkZG9Y zUd9OjZjt(=us9$K>l~ab2f>X~MVVMf>9EuM`Sp@n?EUr+<0t zx<2#e{azZWKFv$J*7XLT@#UWLUF(K)U-?VN|289gJ+IJjxx8>u@teGoS3Ew&x+ucd zGRxLYx!&$r8E@CGvkPB2ScG)bcc0JSR?OHcwy}H%E_qF}OmbJoqeDWQ=thz+w;YG16 zCZu3^Dw~L}nK6t+4+Fg`?k+=BTg2)% zlRBoDmqBjJdLY%Eidw2)){8W8;FEI1oC&8Ce~ZGT2l(|}q? z^`?VbSSgoJB~^zxRwd2TC}9J`!_t(LrN{4Iy0>4zd?dcqve0=_9YoRwQLc&-*{Npb zwklq}Py^_JjTbAA=hO^8A;mIz#hv`-RM%3h+9Io?>nQH8omy61OS9@7H1}UvLuIUr z$z{G)!vGnb4VQ?eQMt++bnhaYIk?l=3B; z0W8o$g2lYtsc>ct{}rz(8B*CCM&cHZ;lGmBEMg@7LMzgyq7qh>$kF>QWTcE@N%0q@p=CMq6S>*^ zte3GxY;nD;k=JbTjeiqTgm20G0OdBND7Wd}a=rgZpoHFk68rtd zJ(N)>jzr=YBWaS%U##g)pGs~u<|_0Vp0~)H@wxA0SSHz+h4-Z1lJc=p8M~=EE;%SF zsdq~6!Qkl!VSqj>C3^Tg?342{r@}5g$(@{71gyT0Y8Gh$(0!Dd$TN(Soe8JkJ^*neU zMU=+7kKtAZOX?yuMD@g^3TI+vknwnvNEP}VwWutiM5#iYOq%7eyvlw{2Ce^1=-N?P z?T#pYCQ3Vb4I7%KpkZqsgUQu)&dKeHR*;sA@d%BHcqTOMc)qIu=^|9|TObz7XkRU^ zyEy5yF4r%gey8~geTfbi*M03G+Wzn93h7JQLYmuw7v4(Fn?{Y*io5_((_UJY*cZ|KItOHNGol;IBR^8^`Z>eN;i{CaUE}L&&Fz6OgM6@G5X6ZT{Tz60@YG7*Osk z09?YKYYcR@Nc|?SFsv=QIPWWcvEkzUVs}VebSE?4U-409>5@WNE70F*S=VR0+;idZ z(qVsn;OJ|1f9;yye9yO!^ybggC}Y`|YD%BEn$iA!dr-AjTo=x059`_&_pYa9e3(|S zo}P8#)Y2(`RXDvI*3S&n%jf*`%g_4jUw*->AlAAqL3yAcIPR-j+_#>U2kp7ZU*T=~ zTWv;cS@2>nuqvxQ*5J`MPrRz~LFa!(b-l%_Sk37CID6C5UT>H8 zG!{J<4lEt;9et@6%bUyR0;QLq^X_}`z~@RTC;xL5Rk3X?ciZ)$t3&>yubjR<^kMEc zuX;%rTSLBj+<)@5Q{kL#YwGRdvhwJ6GJca^95ysu&GB^wPv1Co{nQ1k*LpQ)u_t6` zxKmUSII~t%yQZsKEvmgZ+V!1`M6s;LV2`!-!J@E^Tg9MLlYV? z74;>xztn%!pBv6(77u=GFad##yqBN#D()C_Uv>GLURn6aSnBP%ld~D!d-}EJ*UJNw zKQw=oQ|+VIx0-`nf~ViEeB;FO-j(bX>!0WS_;4t*?4sId@>#A{`5XM)dR~FQ>+)Iu z>C2wL-k|Y~{lTs`4n^{6v7&tJ@lfXDKBaHmZ}QpvmVnHE%r~=|Sst(j_XN4%zU7AH z@#W@V-)cqs`rd;-I}pn3^J)BLev5xA47qy(IsWR^%*vp7dCxMpyl#l~%t+^|wOl26^^sFy4DN{Bv5x>cXH#fb>2I+wDTARP^O?Gv2Q`p$NcxXLyv%fWh znx9$Di&V5nGTT?WZfgGga3u5K;=x}RZwc$RUY+tAeWe!;cn@5iTGee0%m?iuU290u z`t=vbXi8W7_1!`$^Rch*s;Ru~Uw@&d(i^}2SXcM;7iCmI^Vk1Ix0Sa@{9|2zgW@mC zbp5TWzid$B{?i?X{++6ybz~lqEB-1&MebYEkEm6@+H5?MuKt$=#v@sZqzziaOEFm$ zcRdPWnMl~?#w%G#gmIKgrHCiCYzI^uAy9D(kyAe0I6!LMa8;vz@xo%1nD6;(lpcwj zfQL;`c_daKsR$PpH*pKsskG3Yv{ThdbLU23IFu;g@kMAR4ADLR6pETwf_tV|bKw;d8At7v<)guV@(*xZ{aDMRZHqLvf0u)SQDFVQ6%?Qb$q^HulEsX zCaHyRix`2~UbxuoM=8%Lr7cZ@rhy^12Oq6_|1Tqk*x z!(}fj{Bk1iVH!G;aO2f3cJFYeXE6CzSFl#KVYvIeMY=^m5|Bn`Ne)NK>PCp`+{Y` z^1vbQN$i@wd=?r+dX{hRl>-+KymBa)al2I8^R{{gI z4t+4`Z@oMlm<+bQF&oKkyq(jwn$s4}=?G_ZM%10iWPtX#bR3(=Y9pZRBc|%0B4}C8 z4jz9yJyO_myP$Kmpfg;sGn}(4qTc1ztgAD~Uh|bR7tg$M?v~NB!$#Bxjocs1jTwzEh zcp!eyRgUbisK?pf@VP3g8toFVRF=kj1SfX-QaQ1(nnVp2dU%1@weY*d|H_9K zAi%XW6?f3@;W$4%7&riR*O~OmlzMF`iWCNe9MyL03c08T9UpiO_NSwr(yUGwo{Uv zIg;{X14_JNdG$%@o5fU^_mc8L+ewU@Bt^`Rel>^;U?6VH7gH$;E0RR{RK2bg*9OUY zQST20_8=)8HsZXE{CxA&3r6O8DXUL(2#3ZC&enSjGGu_G$6;~b(pIQ-WIw^$r^;oaz6 z)yPQ{O}Arg);>2zT;i(*mwzaNNSdEDlS3~MNONu+jUxWCF<`Y?c*ZKM=j<-tq{`b` z?G*SC#L^GYDvsFEp&{cmlAGitXP}ToxJyU>ZwX)QkG*mS@FG5wE{Q$=mJp@*Oc;(i zQxI>w^;Hjj8;%NYJCR*cmC)dr85ZhxOOqf44RGl^Ylj;L-(ErLDW>4I4#TZ|4qf#P z0FI`i@%CAweB&XiciF7m#7Uyf+a{5QMk39e&3h1XUQ$3qPP#!ahq)EYYAULo8xuP@ zsu@R5+9q)}iF*xcybO)13~(LYb~Sa=6odM@Zy0xFzX#Zf86)XTe2Ui-&C&| z>KA)pB7|L!sPMk2D>D~o0`-?2f%%UNHITpQ*~9|qx4mZe>wKe;^es?z(~bV7%e9g8 z(m?leHk8{ft&#LrUQhP5`S)Jl70D=zsLN7*NPo5LwyEqxQ(2%fY^uC%+P-SqzHI!n zviGars|uO6hfUoPb@zGI=Q=9A_RDlC({$nN`LjOH>kT*Be$e*To*x}}^FX+~`F45x zYI%D&ql2^uXn%*SJdC-%)BgI)^M31hOJI7;%fI5e5~&^x=M9}dKumt6#O9g%nTj&xM)V~x9P5p~tyjz!%>kp&95$5)j@@|T z`V-4tfhWRSTSBImh`tpa2CrPld57N~%vl~@OW%2uj-+=j?uRet_xJE?pmWQ{aA9L4 zt8sZaqApEN7Oh1m%Fp}e(pwCcbW4WFd+(bs zb0jVwEk=vUlJ(H~vn@H6Tua_VA#ajE&bMrS1jq$|T=)o(izJYXEhUctnHh;Mjx42? zvPURUo~UCq+p^`Mt=*c)Z>g|UK0*yuk`m?(HN+i05~K!Ig1?+J+-Wlt-^Hgu;t!dm9bBg5SD1C_)1x}BY)Ms`C*ldFNk4rkLh}cX-l0$fNjM@Gnr(uVe50Jgv4Hug!)`Xn^z>K*r#J|EN7vb69Fc4(-gArMQU6{Dq&-3%TXoAD%#Ef`Ic!b?uwxMs&nXoona zmO6_WE?@8o)Yu<0ZoXhm;*4%c2dJbmvp7<005R zz5xVY#ga)jhfQoDYfM@#7qLYHgZGw3B$^#@O|Y0P9+kDd1)42k^Ajdkwxo(mRaRu* zDKW@ri$DP-$>Yfuvn5Gb6*~!D#~y+g!}GnkMh6-%;-$;U`lMq-4}&A6YiB8IU>P`Y zC0QbzWfJ;f@5KuAV`9rJ^_B)vbK1aR%ULO3#Ln!-3lPe&ci1S-xuj9cW~H3)ex-}q zvNkNJ0i{XA;{OrVXo=bS zU}Gp@}IGE0kCB9#urst3m61>z5uR!iG}HU1iZNVt(p z`LLy3`XprhTV z2TjT4Knrb{;aih!pHj%l^PSJXp>FK~IYGlK2Jxf@N5w9%;`wD@IO<`7FEG4D`cpR*C!LY(DN?4U&VFlAL?k ziUD!B#NzlBWeZAnCnS-CJI_cm#~y)Og-99VFUs#l4P-NG!zHO`=)eABL^Ut``&&O2Y;VzvWH*oCb;{Vdvg=|lTg9q_vVnSQK}?4&)u7o zz!K?%tzwO2D|cjG%*QI=@Ln~%I^Yq%B!0moWKC(pJ6WkZ&CYGka!FtHp{*&(Vf+sB zbE03`FA^-DOVlAg0ZTKETHBArXPtAqq^2X5fmAZcax{-(w`ay_SZrq!pSQ4P_#Hu) zco2Q_{ArLiKWK)G#OD|CJz%a_hT@z?Lft#Mvy;aZKhgkU3_yI*CKp{Qr|e2_6Y7AHZ|OHiHt|By^XW zo5n3Amg7W5N%E(r(;7>uIKDy&w&oG(OA0$RF58m(QfvoLo$U?j;yWbUze+9HBy;Be zoQ!RV+kvo}e4uR({AzpyzgpZt8h9<0S{{EdReB--f!GFTp#;|_ud_u}oQeIe#J4A&d6HYSvvwZ7vbVden)&u{DLcj z8VO-jEA&&{?x$q!FXoVLB0{WR{ zQ8i~fJ#RjwY5nwdrTWa2JO zLL9awzzJ~$%{aKH;C-HN0T%&edOV^aA7PNhYeH0zF*i4F^R&dG{0ea;<6bE^Er&pf zq=wY4C_q~UFBWvQPz8EP^pLPKu(Ep0i9nYz2x58yV?JP^*`t$$r}Z>(^U^@Ff_Dse zhQ#S4{>`zNo*Uf4J!l~x@-Wt0_Qwr!%F`UHSnR@i5*#@ewpap791bCTVgeoF=l$>h z;3~P^+wEy5ucW=CPl)3RAD5PRkN_wjI=2p?u455t<3o(-i#))1phft@{DO=N85}|= zg0rUGn}yfMHkv&uA!1{{ zrzA#1pcR*!1Y@=)Q_VE0$MxB6-k;u6!Ba4ZN{F-@pF5~{0;d@fMHG&rHcmK@No?KR zb#nV5xxJ1XW}gN=FR^Gepy;zCiZT4qe}zO(R*IP2+!RSkiv{oFTJRR7wH|#u{wCsj zdnWkfk7LAjhPc#JOJA+$s^lJM(j zR|gZN>pU{1%9De*)qG%LX(Jhr>MV!o(~M_RiZ15ejy>tIX~$2T4#tx!PU!e_y9sk?$yo1?F5M)vxMXo|m#E&AYafZrh zp$t^)*fFgAL_bIzJN})#cE?md2dgM?89Rg@{&1}eCQ`(>;oYLR!+4J7%_4SqiZjO3 zG(4oRMJ${#J}|a=Vl;I8iCYu5T7S7Yd}8!#7ukCLy-XABF?1q2Z!BajIz-t?24MQ~ zM-~4;S8Ln_QfQd+7~<|=B>1}L8U9dhjGPE1CcX2dD0U~vj0~^fuU37!en;$yr^k4` zVy=6<_q1FXczO&8{sfNW(Tamp&3G~q)gG~e6D$E?g7CqUK1Ry$E@M2->G2%ZebqyR z&ReE9_#9ClT}M1}@=lLV07qzx(4uQ{CnmFEa?*iG!FYC}SpAeIopDx?N_Oo;$Yt|R zA+Avx@AxOREz1o9Au+zs+UPvUSRu=aB#thV_8PRa1jf-YM$y^EX1!S6Zu0158IMbpN{ za4u+UjN5{8Q9aIlk0CFgL-3R5kMn4SkVr@&u6QI0F9!{k;)u%%ry)0NKERzmCJM&j zIuDOW#J=UD2y#CqIm{%7Cf>)~GI`hY@3EP~m63cuBe#7dQ9;5I@~5XcVp${cDBZ&O z^QTCjA#x*bCfpy9+mpCOwexc%CMAhu$)9WI$WokRNRk3!+TGmq* zgnNg)sNuNCJ10gBPO;VAZRJp`IK~^AsfWyuGGm`o)`G$A|&^V&>=_ozRb&)R! zGjP_$+4)Bj1Pw>iCUK69ZxinN!#_Il=82V-aLt~Z^^ux=YnmP$BvYuqedL=*e7$Q5MtnZBrYI4g z_xiU5-Jy>D#UpEqBjU%tP-PGKuwVReD5wZE4w6qp;-`bJcQ5Cz=l6Xkr^;-y2Wri6+KLlUPZTeAz-UZWA#+^!niPmi7Fn;+apxGaq=paaln!H{(SAg`G<~ zU))V%9q7L8`KHI$_3anF^#V@eT~MD_hl~wt>c&sC`U~bI^KEU(sg!o=Zhv?5-O=UImj-=VzRBCUm8-dx!HP)k z4*34w%7XWAdL|C=kuatgW|wAfYYNvig#p=};%&i(AGN;O`bX{I;^tL#fmi8se}&ki zMW1eIT~V%_x#_+&^fxDdaU#6ssZ~YkVy|CwCr#_^45#INztMl@d!1`(6>-Gd*VG;H zzy-ud`P}()?`#im+V#%%AfgCdE4*45u>N7mJKI+`?Fwh?UOez=BFI314%7$6aY%4{ zxqSKbO8t-LL-{+ynY$Jb-qGm2PrO*{FO6sl7Wdpi$5frK@-=+la_#ul<7?UqbV|(G zdHi+R4fS<6JDV4K#P|iKSM&gZGyYYV5Od&_>cw8-(0lUyNyIxi^Sb56@$1Le(re+* zYcwq$`c!a^zV^)3XKowJa5lzKB^t*R4_!QTIX$2X>nrgM?Rx(F^L{qa6*!H61!sb% zm+Rl2UopNt6RPY8ZRv;1vp%h-GK=q=SPq7rh zzrP6$ZoJWPy(4H{E`8e;E^k^svs%70#O#a|?7CgBZ?$0GPbb3#y=(eDa)PkoO6#Ro zf77+jtDUcNH$2xp;ev+cp?9Bp=c%8Xe!lsG&Ee+0+s*x}&Ha&r{&3cj+ga>t7P~h5 z#O>j+)#0&l*3%L7)84dCNRWt^`^kB$%nS2N^Zrur{EN@yP-I4y@AT3FIUQ+wbpj_Z zo3CxZy8U}K;oPc-y2`6XC_P`7&+XfPc^e|yHof+Yw+S!|=@*VK9rxFJkB1FKI1!2X zfX0`P`Ho(6BWQ#B^6-_>i=#oKZ#0~U!c}4uA41+b?bRPcJK0lp4sc}F7ZBQNScucXqqeYuTNrrlO#f2hdz4g0O&XP7U(>?;Kq3w(x9 z(T-4h{c_`NP3xMbb){ilt-YXLQv34#gM<(RQ{keTh_Pl_5ivHcshjWQB05T-@}t~s zOFf7$;#$|I3-Lw#S(k_5xPR{IxwqtRbt6=>AzacD$!S^1j^yn8NWTj{>UrD0Qc~Ks zFSV35U98?e5=!3|?7pqpv8LIvyeC%sXB2JP3D&6D^(p+~x$ov9lp#V^)?YaDoijK` zdqsUw?LUYJA$|T`L1VCe`RI!A$HyYuJ6Eic?Yl$U9=}Q7EWc^}S;fsww~XtBTS?Af zBcgV+L<;Md%R>b%%hnau%IQDX-0Z$Z|Ewp{cIZ}PsP)LL)9ZzmLHf1+;NY_HZ8lQa zxZE8oXj_qerj%`d?7veo<4#1^Fco~2MrrF2#8_v-_>s{r4IIUJfQYsVlUAYeBHW6v z+qV$N4X8s^$3pDWp<~vNbv&e*s0=8*^?P)g5@L)Q=eurII+Cyg7Iw=Q!l74L6*Z`)6*Z)HU~`fd$| z+K-|p3th5aQg!Y5tIvP$8_QXdlBVUsP;onf+ezR)_9Lod9l-16Dnz4QM|8v zPq+O1&AyQK=$gXvNt*6<+9v#8P1|%QJNL?gO9%W%Umw45^7_fO>^iUJPBC-M{s;CT z{k<8l##aSqe5bT#U2pR3_dgL-{^q^>Ja#qf~?|IMjljnXp5waY=Z8@=OIT5yug{-!4 z|5Rl6RA~B{(5_RV%$fftr!!`0LW^_sr#K}&bmggwPx%f9ibAHk<*eI=<~2j}iYYb? z(3`nkB(q@;{VAfyG<^37gpA29@Sncehli}}E6o?1ue83}y1uCxfn_wlu65F37Y_t9 z!O}PM{uhErg9|Ihm7{;26RF#MvouoI9je`Xv+Ji5w@UwJ>o2zcb>&~q+#C%J3hZI@ z!(*eNv8Tfzu5nvvXeu-{eTON#q5FX@*cIf$Ov7>`h6_@sbmjDW74H|lSM-yTTUnvD z!?%w9lKY#5f3XmHf|QRuVGSP{4^2#k`lmzFb_{&+Df+)tbj}R@IikvxfZEerzBEyq zbf3!SzVhtFXMMAQonRVk>IRJcw_o`73*USJ;^THk`G*G;fz)s zi^V`9UO16GLE~$){xL*wcfb3>J1>M3T!yKkF|JZ8ERZzvzN62<0r7#fP&vWZJP@e%sKpW@uR{ ziw$Qer(#WC`EhaCMXgWav-*u9@7e2b2^b;R2LqMhKlQ=EVCAwYl;8AGW;6I)Njdmj zdMo1M6m-!`hrHds90XMLoxWV@FTY&v?Y@(hbEWN4TX1N(?2nI!3tQei9;6YD$aRhN zvu_?>Eo=#AwIU3=(FCL?E}pubQ?Z&;5zeWK7^{Lu|0HX9=%3}keS$;_=(>ZGKF2VR zIs>j?W56DKcBSe4_V?O<(s|Pr>gWYE_R*WxHx*t}UsWSMt2(@?Dwq|@s{wl}$1K*o zr}>F4($Ia={qu7loC_UeZ=MV9ACA-yhmJoL+A)ISC+M>EvhqN8AP*ccYa2MAVLNC7 z&BE!XBzo8tN^kf`-3WQ3&i%?JgQ5jhETk#?Utj2{{M}#ep~3heVnUkM-+cxCUk`>K z((L%%S1q6|OrnsX`FCGcpwln{Awvr(Rn+PlhSvWJs_0yk@~_i3FR0ZY%l4=j49b7i zSiMlH{MTFd=<)L(OWPKjmH(-D&u;wOt?Kk>l>e>RUE& zp!vF4=TT{Xr_p&fYoh8gM0tbdY|L!nqGX-1PHto+jgZXqv(5!!EnP#tk`=J9_Wdq- zx<+nf-N-K}2qdz6!Mgn*`Ig2f;1*RsCPe$##T8=Ff&Y{X`-c=J7)^h!sHGKm5$1r- z{Fu`ITgvo>s*tYvVpkho`o$T#nr{1Il+LBMebJz&3%)4Mqj!GMv{yzqeb%p_6nd{` TO`iX2h5GkR( literal 0 HcmV?d00001 diff --git a/scripts/pseudo_alignment_nesting.py b/scripts/pseudo_alignment_nesting.py new file mode 100644 index 00000000..7fe59fdc --- /dev/null +++ b/scripts/pseudo_alignment_nesting.py @@ -0,0 +1,395 @@ +#!/usr/bin/env python3 +"""Family overlap and clade-closure ("nesting") of a SNP pseudo-alignment. + +Each alignment column is one family; a genome "carries a call" at a column +when its character is not an absence symbol (default '-' and '?'). + +Outputs (prefix given by -o): + _family_overlap.csv genome x genome matrix: number of columns where + both genomes carry a call (diagonal = 0, same + convention as `obikmer phylo --family-overlap`) + _nesting.csv per clade-size stratum: how closed the families + are under the tree + +Nesting measure: for a family carried by the genome set S, let C(S) be the +smallest tree clade containing S. completeness = |S| / |C(S)|. A value of 1 +means the family is clade-closed (every genome of the clade carries it); low +values mean dropout or patchy conservation inside the clade. + +The tree is rooted as written in the Newick file unless --outgroup is given +(rooted at the neighbour of that tip, which then becomes a child of the root). + +Optional lift test (--lift-test): do families covering a rare pair (A,B) also +cover an abundant pair (A,C) more often than the margins alone imply? For each +genome A, B ranges over its --lift-frac rarest partners and C over its most +abundant ones (ranked on pair overlap). Statistics, on a random column subsample: + T = mean over triples of O_ABC / O_AB = P(C | A,B) + L = sum n_A * O_ABC / sum O_AB * O_AC (lift; 1 if B,C independent given A) +Null: curveball swap randomisation of the genome x family presence matrix, which +preserves every genome's family count and every family's size exactly. Each null +replicate re-selects the rare/abundant partners from its own overlaps, so the +selection procedure is identical to the observed one. + +Optional quartet test (--quartet-test): does flattening pair overlaps also flatten +quartet coverage? --quartets N genome quartets are drawn uniformly; for each, the +number of columns where all four genomes carry a call is counted exactly. Reported +per stratum of the quartet's depth (size of the smallest tree clade containing the +four genomes): count distribution, share of uncovered quartets, and the ratio of the +4-way count to the smallest of the six pair overlaps (1 = perfect nesting; the pair +overlap is an upper bound of the 4-way count). + +Usage +----- + pseudo_alignment_nesting.py ALIGNMENT.fasta TREEFILE -o prefix + [--outgroup LABEL] [--absent '-?'] [--chunk 131072] + [--quartet-test [--quartets 20000]] + [--lift-test [--lift-columns 200000] [--lift-frac 0.1] + [--null-reps 20] [--trades 5000] [--seed N]] +""" + +import argparse +import sys + +import numpy as np + + +def read_fasta_presence(path, absent): + """Return (labels, packed presence bits per genome, alignment length).""" + absent_lut = np.zeros(256, dtype=bool) + for ch in absent: + absent_lut[ord(ch)] = True + labels, packed, length = [], [], None + + def flush(parts): + nonlocal length + seq = np.frombuffer(b"".join(parts), dtype=np.uint8) + if length is None: + length = len(seq) + elif len(seq) != length: + sys.exit(f"{labels[-1]}: length {len(seq)} != {length}") + packed.append(np.packbits(~absent_lut[seq])) + + parts = None + with open(path, "rb") as fh: + for line in fh: + if line.startswith(b">"): + if parts is not None: + flush(parts) + labels.append(line[1:].split()[0].decode()) + parts = [] + else: + parts.append(line.strip()) + if parts is not None: + flush(parts) + return labels, np.vstack(packed), length + + +def parse_newick(text): + """Parse Newick into (names, parent, length) arrays; internal labels ignored.""" + names, parent, length = [], [], [] + stack, cur = [], None + i, n = 0, len(text) + + def new_node(): + names.append(None) + parent.append(-1) + length.append(0.0) + return len(names) - 1 + + while i < n: + c = text[i] + if c in " \n\r\t": + i += 1 + elif c == "(": + if stack: + node = stack[-1] # the placeholder child becomes this internal node + else: + node = new_node() + stack.append(node) + cur = None + i += 1 + child = new_node() + parent[child] = node + stack.append(child) + elif c == ",": + stack.pop() + child = new_node() + parent[child] = stack[-1] + stack.append(child) + i += 1 + elif c == ")": + stack.pop() + cur = stack[-1] + i += 1 + j = i + while j < n and text[j] not in ":,);": + j += 1 + i = j # internal label (support values) discarded + elif c == ":": + j = i + 1 + while j < n and text[j] not in ",);": + j += 1 + length[stack[-1] if cur is None else cur] = float(text[i + 1:j]) + cur = None + i = j + elif c == ";": + break + else: + j = i + while j < n and text[j] not in ":,);": + j += 1 + names[stack[-1]] = text[i:j] + i = j + return names, parent, length + + +def leaf_order_matrices(names, parent, length, outgroup): + """DFS leaf order plus LCA-clade-size and patristic matrices in that order.""" + adj = [[] for _ in names] + for v, p in enumerate(parent): + if p >= 0: + adj[v].append((p, length[v])) + adj[p].append((v, length[v])) + root = next(v for v, p in enumerate(parent) if p < 0) + if outgroup is not None: + tip = names.index(outgroup) + root = adj[tip][0][0] + + order, depth, children = [], {root: 0.0}, {} + stack, seen = [root], {root} + while stack: + v = stack.pop() + kids = [(u, w) for u, w in adj[v] if u not in seen] + for u, w in kids: + seen.add(u) + depth[u] = depth[v] + w + children[v] = [u for u, _ in kids] + stack.extend(reversed(children[v])) + order.append(v) + + leaves = [v for v in order if not children[v]] + rank = {v: r for r, v in enumerate(leaves)} + n = len(leaves) + span = {} + for v in reversed(order): + span[v] = (rank[v], rank[v] + 1) if not children[v] else ( + span[children[v][0]][0], span[children[v][-1]][1]) + + clade = np.ones((n, n), dtype=np.int32) + lca_depth = np.zeros((n, n)) + for v in order: + kids = children[v] + for a in range(len(kids)): + for b in range(a + 1, len(kids)): + (a0, a1), (b0, b1) = span[kids[a]], span[kids[b]] + size = span[v][1] - span[v][0] + clade[a0:a1, b0:b1] = clade[b0:b1, a0:a1] = size + lca_depth[a0:a1, b0:b1] = lca_depth[b0:b1, a0:a1] = depth[v] + leaf_depth = np.array([depth[v] for v in leaves]) + patristic = leaf_depth[:, None] + leaf_depth[None, :] - 2 * lca_depth + return [names[v] for v in leaves], clade, patristic + + +def lift_stats(pres, frac): + """(L, T) for the rare/abundant partner triples of every genome (see module doc).""" + n = pres.shape[0] + pf = pres.astype(np.float32) + overlap = pf @ pf.T + k = max(1, round(frac * (n - 1))) + num = den = ratio_sum = 0.0 + n_ratio = 0 + for a in range(n): + order = [j for j in np.argsort(overlap[a], kind="stable") if j != a] + rare, abundant = order[:k], order[-k:] + cols = pres[a] + n_a = float(cols.sum()) + if n_a == 0: + continue + o_abc = pf[rare][:, cols] @ pf[abundant][:, cols].T + o_ab, o_ac = overlap[a, rare].astype(np.float64), overlap[a, abundant].astype(np.float64) + num += n_a * float(o_abc.sum()) + den += float((o_ab[:, None] * o_ac[None, :]).sum()) + ok = o_ab > 0 + ratio_sum += float((o_abc[ok] / o_ab[ok, None]).sum()) + n_ratio += int(ok.sum()) * k + return num / den, ratio_sum / n_ratio + + +def curveball(pres, trades, rng): + """In-place curveball trades: preserve every row sum and every column sum.""" + n = pres.shape[0] + for _ in range(trades): + i, j = rng.choice(n, size=2, replace=False) + a, b = pres[i], pres[j] + diff = np.flatnonzero(a ^ b) + k = int((a & ~b).sum()) + diff = rng.permutation(diff) + a[diff] = False + b[diff] = False + a[diff[:k]] = True + b[diff[k:]] = True + + +def quartet_test(packed, overlap, tree_rank, clade, n_quartets, bin_edges, rng): + """Exact 4-way co-presence counts for random genome quartets. + + Returns (counts, ratio_to_min_pair, depth_stratum), one entry per quartet. + """ + n = packed.shape[0] + quartets = np.argpartition(rng.random((n_quartets, n)), 3, axis=1)[:, :4] + counts = np.zeros(n_quartets, dtype=np.int64) + block = 2048 # bytes of packed columns per pass: bounds the (Q, block) gathers + for a in range(0, packed.shape[1], block): + blk = packed[:, a:a + block] + acc = blk[quartets[:, 0]] & blk[quartets[:, 1]] + acc &= blk[quartets[:, 2]] + acc &= blk[quartets[:, 3]] + counts += np.bitwise_count(acc).sum(axis=1, dtype=np.int64) + + pair_min = np.full(n_quartets, np.iinfo(np.int64).max) + for x in range(4): + for y in range(x + 1, 4): + pair_min = np.minimum(pair_min, overlap[quartets[:, x], quartets[:, y]]) + ranks = tree_rank[quartets] + csize = clade[ranks.min(axis=1), ranks.max(axis=1)] + stratum = np.searchsorted(bin_edges, csize, side="right") - 1 + return counts, counts / np.maximum(pair_min, 1), stratum + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("alignment") + ap.add_argument("treefile") + ap.add_argument("-o", "--output", required=True, help="output file prefix") + ap.add_argument("--outgroup", help="tip label to root the tree on") + ap.add_argument("--absent", default="-?", help="absence symbols (default '-?')") + ap.add_argument("--chunk", type=int, default=131072, + help="columns per chunk, multiple of 8 (default 131072)") + ap.add_argument("--quartet-test", action="store_true", + help="measure exact 4-way coverage of random genome quartets") + ap.add_argument("--quartets", type=int, default=20000, + help="number of random quartets (default 20000)") + ap.add_argument("--lift-test", action="store_true", + help="run the margin-preserving lift test (see module doc)") + ap.add_argument("--lift-columns", type=int, default=200000, + help="random columns used by the lift test (default 200000)") + ap.add_argument("--lift-frac", type=float, default=0.1, + help="fraction of partners ranked rare / abundant (default 0.1)") + ap.add_argument("--null-reps", type=int, default=20) + ap.add_argument("--trades", type=int, default=5000, + help="curveball trades between replicates, also burn-in (default 5000)") + ap.add_argument("--seed", type=int, default=None) + args = ap.parse_args() + if args.chunk % 8: + sys.exit("--chunk must be a multiple of 8") + + rng = np.random.default_rng(args.seed) + labels, packed, length = read_fasta_presence(args.alignment, args.absent) + keep_p = min(1.0, args.lift_columns / length) if args.lift_test else 0.0 + sampled = [] + with open(args.treefile) as fh: + names, parent, blen = parse_newick(fh.read()) + leaf_labels, clade, patristic = leaf_order_matrices(names, parent, blen, args.outgroup) + if set(leaf_labels) != set(labels): + sys.exit("tree tips and alignment records differ") + n = len(labels) + tree_row = np.array([labels.index(l) for l in leaf_labels]) # tree rank -> alignment row + + bin_edges = np.array([2, 3, 5, 9, 17, 33, 65, 129, n + 1]) # clade-size strata + nb = len(bin_edges) - 1 + cnt = np.zeros(nb, dtype=np.int64) + closed = np.zeros(nb, dtype=np.int64) + sum_ratio = np.zeros(nb) + sum_missing = np.zeros(nb) + overlap = np.zeros((n, n), dtype=np.int64) + n_cols_lt2 = 0 + + for a in range(0, length, args.chunk): + b = min(a + args.chunk, length) + pres = np.unpackbits(packed[:, a // 8:(b + 7) // 8], axis=1)[:, :b - a] + if keep_p: + sampled.append(pres[:, rng.random(b - a) < keep_p].astype(bool)) + overlap += np.rint(pres.astype(np.float32) @ pres.astype(np.float32).T).astype(np.int64) + + t = pres[tree_row] + size = t.sum(axis=0) + keep = size >= 2 + n_cols_lt2 += int((~keep).sum()) + t, size = t[:, keep], size[keep] + lo = t.argmax(axis=0) + hi = n - 1 - t[::-1].argmax(axis=0) + csize = clade[lo, hi] + stratum = np.searchsorted(bin_edges, csize, side="right") - 1 + cnt += np.bincount(stratum, minlength=nb) + closed += np.bincount(stratum, weights=(size == csize), minlength=nb).astype(np.int64) + sum_ratio += np.bincount(stratum, weights=size / csize, minlength=nb) + sum_missing += np.bincount(stratum, weights=csize - size, minlength=nb) + + np.fill_diagonal(overlap, 0) + with open(f"{args.output}_family_overlap.csv", "w") as fh: + fh.write("genome," + ",".join(labels) + "\n") + for l, row in zip(labels, overlap): + fh.write(l + "," + ",".join(map(str, row)) + "\n") + + with open(f"{args.output}_nesting.csv", "w") as fh: + fh.write("clade_size_min,clade_size_max,n_families,mean_completeness," + "frac_clade_closed,mean_missing_in_clade\n") + for k in range(nb): + if cnt[k]: + fh.write(f"{bin_edges[k]},{bin_edges[k + 1] - 1},{cnt[k]}," + f"{sum_ratio[k] / cnt[k]:.4f},{closed[k] / cnt[k]:.4f}," + f"{sum_missing[k] / cnt[k]:.3f}\n") + + off = overlap[np.triu_indices(n, 1)] + print(f"genomes={n} columns={length} (columns with <2 calls skipped for nesting: {n_cols_lt2})") + print(f"overlap off-diagonal: min={off.min()} median={int(np.median(off))} " + f"max={off.max()} CV={off.std() / off.mean():.3f}") + tot = cnt.sum() + print(f"clade-closed families: {closed.sum()}/{tot} ({closed.sum() / tot:.3f}); " + f"mean completeness {sum_ratio.sum() / tot:.3f}") + print(f"wrote {args.output}_family_overlap.csv and {args.output}_nesting.csv") + + if args.quartet_test: + tree_rank = np.empty(n, dtype=np.int64) + tree_rank[tree_row] = np.arange(n) + counts, ratio, stratum = quartet_test( + packed, overlap, tree_rank, clade, args.quartets, bin_edges, rng) + print(f"quartet test: {args.quartets} random quartets over {length} columns") + q = np.quantile(counts, [0, .01, .1, .5, .9, .99, 1]) + print("4-way count quantiles 0/1/10/50/90/99/100 %: " + " ".join(f"{v:.0f}" for v in q) + + f"; uncovered: {(counts == 0).mean():.4f}; CV={counts.std() / max(counts.mean(), 1e-9):.2f}") + with open(f"{args.output}_quartets.csv", "w") as fh: + fh.write("depth_clade_min,depth_clade_max,n_quartets,median_count,q10_count," + "frac_uncovered,median_ratio_to_min_pair\n") + for k in range(nb): + m = stratum == k + if m.any(): + c = counts[m] + fh.write(f"{bin_edges[k]},{bin_edges[k + 1] - 1},{int(m.sum())},{np.median(c):.0f}," + f"{np.quantile(c, .1):.0f},{(c == 0).mean():.4f},{np.median(ratio[m]):.4f}\n") + print(f"wrote {args.output}_quartets.csv") + + if args.lift_test: + pres = np.ascontiguousarray(np.hstack(sampled)) + print(f"lift test on {pres.shape[1]} columns, {args.null_reps} null replicates") + rows = [("observed",) + lift_stats(pres, args.lift_frac)] + curveball(pres, args.trades, rng) # burn-in + for r in range(args.null_reps): + rows.append((f"null{r + 1}",) + lift_stats(pres, args.lift_frac)) + curveball(pres, args.trades, rng) + with open(f"{args.output}_lift_test.csv", "w") as fh: + fh.write("replicate,L,T\n") + for name, l, t in rows: + fh.write(f"{name},{l:.6f},{t:.6f}\n") + for idx, label in ((1, "L (lift)"), (2, "T = P(C|A,B)")): + obs = rows[0][idx] + null = np.array([r[idx] for r in rows[1:]]) + p = (1 + (null >= obs).sum()) / (1 + len(null)) + print(f"{label}: observed={obs:.4f} null mean={null.mean():.4f} " + f"sd={null.std(ddof=1):.4f} z={(obs - null.mean()) / null.std(ddof=1):.1f} " + f"p(null>=obs)={p:.3f}") + print(f"wrote {args.output}_lift_test.csv") + + +if __name__ == "__main__": + main() diff --git a/src/obikphylo/src/siblings/algorithms/entropy.rs b/src/obikphylo/src/siblings/algorithms/entropy.rs index 175d46ac..78c27c77 100644 --- a/src/obikphylo/src/siblings/algorithms/entropy.rs +++ b/src/obikphylo/src/siblings/algorithms/entropy.rs @@ -51,28 +51,25 @@ pub(crate) fn family_entropy(genome_mask: &[u8]) -> Option<(f64, usize)> { Some((h, present as usize)) } -/// Shannon entropy (bits) of one family's per-genome states, reduced to the -/// 4 plain nucleotide symbols instead of the 15-state space above — kept -/// alongside it (not instead of it) purely to measure how much the two -/// diverge on real data. A genome carrying several bases at once -/// (`genome_mask[g]` with more than one bit set) contributes to *each* of -/// those bases' counts — counted once per base present, not split -/// fractionally, and not folded into a combined state the way -/// [`family_entropy`] does. The denominator is therefore the total number -/// of base-occurrences summed over every present genome, not the genome -/// count. `None` under the same condition as [`family_entropy`]. +/// Shannon entropy (bits) of one family's per-genome states over the 4 plain +/// nucleotide symbols, restricted to the genomes carrying exactly one member +/// of the family (`genome_mask[g].count_ones() == 1`) — the genomes that stay +/// informative under `--no-ambiguity`, where a genome carrying several +/// members (paralogues, not read ambiguity) becomes missing data. Genomes +/// where the family is absent (`genome_mask[g] == 0`) and genomes carrying +/// several members are both left out of the denominator, which is returned +/// as the second value (`n_genomes_unambiguous`). `None` if no genome carries +/// exactly one member. pub(crate) fn family_entropy_4(genome_mask: &[u8]) -> Option<(f64, usize)> { let mut freq = [0u32; 4]; - let mut total = 0u32; + let mut n_single = 0u32; for &m in genome_mask { - for (base, count) in freq.iter_mut().enumerate() { - if m & (1 << base) != 0 { - *count += 1; - total += 1; - } + if m.count_ones() == 1 { + freq[m.trailing_zeros() as usize] += 1; + n_single += 1; } } - if total == 0 { + if n_single == 0 { return None; } let mut h = 0.0f64; @@ -80,10 +77,37 @@ pub(crate) fn family_entropy_4(genome_mask: &[u8]) -> Option<(f64, usize)> { if c == 0 { continue; } - let p = f64::from(c) / f64::from(total); + let p = f64::from(c) / f64::from(n_single); h -= p * p.log2(); } - Some((h, total as usize)) + Some((h, n_single as usize)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn entropy_4_balanced_two_states_is_one_bit() { + // two genomes carry base 0 (A), two carry base 2 (G); one is absent + let (h, n) = family_entropy_4(&[0b0001, 0b0001, 0b0100, 0b0100, 0]).unwrap(); + assert!((h - 1.0).abs() < 1e-12); + assert_eq!(n, 4); + } + + #[test] + fn entropy_4_ignores_genomes_with_several_members() { + // the two-member genomes (0b0011, 0b1100) are left out entirely + let (h, n) = family_entropy_4(&[0b0001, 0b0001, 0b0011, 0b1100]).unwrap(); + assert_eq!(h, 0.0); + assert_eq!(n, 2); + } + + #[test] + fn entropy_4_is_none_without_unambiguous_genome() { + assert!(family_entropy_4(&[0, 0b0011, 0b0110]).is_none()); + assert!(family_entropy_4(&[]).is_none()); + } } /// Builds one layer's entropy annex if it doesn't already exist, and/or diff --git a/src/obikphylo/src/siblings/extensions/sibling_ext.rs b/src/obikphylo/src/siblings/extensions/sibling_ext.rs index 7cc26ba3..b1812c1b 100644 --- a/src/obikphylo/src/siblings/extensions/sibling_ext.rs +++ b/src/obikphylo/src/siblings/extensions/sibling_ext.rs @@ -45,10 +45,13 @@ pub trait SiblingExt { fn build_sibling_annex(&self) -> OKIResult<()>; /// Write a CSV (`partition,layer,family_idx,entropy15,entropy4, - /// family_size,n_genomes_present`) of per-family Shannon entropy — both - /// [`family_entropy`] (15 non-empty states, the project's settled - /// definition) and [`family_entropy_4`] (plain 4-symbol reduction, kept - /// alongside for comparison) — one row per non-monomorphic minorant of + /// family_size,n_genomes_present,n_genomes_unambiguous`) of per-family + /// Shannon entropy — both [`family_entropy`] (15 non-empty states over + /// every genome carrying the family, the project's settled definition) + /// and [`family_entropy_4`] (4 plain bases over the genomes carrying + /// exactly one member only, i.e. what survives `--no-ambiguity`; + /// `n_genomes_unambiguous` is its denominator, `0` meaning `entropy4` + /// is not defined) — one row per non-monomorphic minorant of /// every cached layer, from an already-built sibling annex (run /// [`build_sibling_annex`](Self::build_sibling_annex) first). /// @@ -166,7 +169,7 @@ impl SiblingExt for IndexCache { let fast_mode = is_fast_mode(self); let mut f = BufWriter::new(std::fs::File::create(path).map_err(OKIError::Io)?); - writeln!(f, "partition,layer,family_idx,entropy15,entropy4,family_size,n_genomes_present") + writeln!(f, "partition,layer,family_idx,entropy15,entropy4,family_size,n_genomes_present,n_genomes_unambiguous") .map_err(OKIError::Io)?; let pb = progress_bar("shannon_entropy", self.n_partition() as u64, "partitions"); @@ -205,10 +208,10 @@ impl SiblingExt for IndexCache { let Some((h15, n_present)) = family_entropy(genome_mask) else { return; }; - let h4 = family_entropy_4(genome_mask).map_or(0.0, |(h, _)| h); + let (h4, n_unambiguous) = family_entropy_4(genome_mask).unwrap_or((0.0, 0)); if let Err(e) = writeln!( f, - "{part},{l},{family_idx},{h15:.6},{h4:.6},{},{n_present}", + "{part},{l},{family_idx},{h15:.6},{h4:.6},{},{n_present},{n_unambiguous}", mask.family_size() ) { write_err = Some(e);