Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
This commit is contained in:
1 parent
95bbc0b2ae
commit
0d12494f1a
12 files changed
+1182
-56
No files matched your search
@@ -0,0 +1,69 @@
|
||||
# Chance sharing of a kmer across genomes (discussion)
|
||||
|
||||
Null model for "a kmer is present in $r$ of $S$ genomes": how likely is this by chance alone? Extends the single-pair estimate $P_{\mathrm{chance}}\simeq G/4^k$ (see user doc, *K-mer specificity*).
|
||||
|
||||
## Homogeneous case: binomial tail
|
||||
|
||||
If every genome has the same per-genome chance probability $p$, the number $R$ of genomes containing a given kmer is $\mathrm{Bin}(S,p)$:
|
||||
|
||||
$$P(R\geq r)=\sum_{j=r}^{S}\binom{S}{j}p^j(1-p)^{S-j}=I_p(r,\,S-r+1)$$
|
||||
|
||||
where $I_p$ is the regularized incomplete beta function. No elementary closed form; use $I_p$ directly.
|
||||
|
||||
### Numerical evaluation of the sum
|
||||
|
||||
- Term recurrence, one `lgamma` start: $t_{j+1}=t_j\cdot\frac{S-j}{j+1}\cdot\frac{p}{1-p}$.
|
||||
- Sum the tail **not containing the mode** ($Sp$, not $S/2$). If $r>Sp$, sum $\sum_{j\ge r}$ directly and stop when terms are negligible. The complement $1-\sum_{j=0}^{r-1}$ (note: from $j=0$) suffers catastrophic cancellation when the p-value is tiny.
|
||||
- Large $S$: normal approximation with continuity correction, or Poisson($Sp$) when $p$ is small. Unreliable in extreme tails.
|
||||
|
||||
## Heterogeneous case: $p_i = f(G_i)$
|
||||
|
||||
The $G_i$ (number of kmers of genome $i$) are known and differ. With $p_i\simeq G_i/4^k$, the indicators "kmer present in genome $i$ by chance" are independent Bernoulli($p_i$), so $R$ follows a **Poisson-binomial** law, not a binomial:
|
||||
|
||||
$$P(R=j)=\sum_{|A|=j}\prod_{i\in A}p_i\prod_{i\notin A}(1-p_i).$$
|
||||
|
||||
Everything lives in canonical space: $G_i$ is the number of distinct canonical kmers of genome $i$, and the kmer space has size
|
||||
|
||||
$$N=\frac{4^k}{2}$$
|
||||
|
||||
exactly, since $k$ is odd (no kmer equals its own reverse complement, so canonicalization pairs the $4^k$ kmers perfectly). Exact per-genome probability: $p_i=1-(1-1/N)^{G_i}\simeq 2G_i/4^k$. This is twice the $G/4^k$ of the user doc, where $G$ is a number of positions on one strand.
|
||||
|
||||
### Exact computation
|
||||
|
||||
Dynamic programming over genomes, $d_j$ = probability of $j$ hits so far:
|
||||
|
||||
$$d'_j=d_j(1-p_i)+d_{j-1}p_i.$$
|
||||
|
||||
For the upper tail only $d_0\ldots d_{r-1}$ are needed: $P(R\ge r)=1-\sum_{j<r}d_j$, cost $O(S\,r)$, per distinct $(p_1..p_S)$ vector — independent of the kmer, so computed once per index (table of $P(R\ge r)$ for $r=1..S$), not per kmer. Cancellation caveat as above: for tiny tails compute $\Pr(R\ge r)$ by the DP on the upper side (track $d_j$ for $j\ge r$ only from a top-down recurrence) or in log space. Alternative: DFT of the characteristic function (Hong 2013), $O(S^2)$ exact.
|
||||
|
||||
### Approximations
|
||||
|
||||
- **Poisson**: $R\approx\mathrm{Poisson}(\lambda=\sum_i p_i)$ with total variation error $\le\sum_i p_i^2$ (Le Cam). Excellent when all $p_i\ll1$, i.e. $4^k\gg G_i$ (large $k$).
|
||||
- **Binomial with mean $\bar p=\frac1S\sum_i p_i$**: Hoeffding (1956) shows the Poisson-binomial is more concentrated than $\mathrm{Bin}(S,\bar p)$. Hence for $r\ge S\bar p+1$, $P_{\mathrm{PB}}(R\ge r)\le P_{\mathrm{Bin}}(R\ge r)$: using the mean $\bar G$ overestimates the chance probability, i.e. is **conservative** in the upper tail. Wrong direction for $r\le S\bar p$ (lower tail), irrelevant for significance testing.
|
||||
|
||||
Consequence: assuming a common $G$ is safe (conservative) if $G=\bar G$ (arithmetic mean), but loses power when genome sizes span orders of magnitude; the exact DP is cheap enough (once per index) to make the approximation unnecessary. Small genomes with $k=11$ saturate ($p_i\to1$) and break the Poisson approximation, not the DP.
|
||||
|
||||
## Open points
|
||||
|
||||
- Repeats are not a problem. The model is presence/absence per genome, so copy number within a genome is irrelevant, and $G_i$ is the number of **distinct** kmers of genome $i$. A repeat shared between genomes is paralogy, i.e. homology: a true positive, not a false one. Low-complexity kmers (which would share by composition, not descent) are removed upstream by the [entropy filter](entropy.md).
|
||||
- Residual violation of the equiprobability behind $p_i$: nucleotide composition bias (see Karlin–Ost in the user doc).
|
||||
- Test dependence between overlapping kmers (see FDR section).
|
||||
|
||||
## Goal: an FDR of homology
|
||||
|
||||
$H_0$: the $S$ genomes are unrelated (independent random sequences of sizes $G_i$). Rejecting $H_0$ for a kmer, i.e. sharing that chance cannot explain, is the evidence of homology. The quantity wanted is the false discovery rate of the call "shared kmer $\Rightarrow$ homologous".
|
||||
|
||||
### Empirical FDR from the count histogram
|
||||
|
||||
Let $N$ be the size of the kmer space and $n_{\ge r}$ the observed number of distinct kmers present in at least $r$ genomes (a by-product of the index). Under $H_0$ the expected number of kmers with $R\ge r$ over the whole space is exactly $N\,P(R\ge r)$, with $P$ the Poisson-binomial tail above. Hence
|
||||
|
||||
$$\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P(R\ge r)}{n_{\ge r}}\right).$$
|
||||
|
||||
- No per-kmer multiple-testing correction: the number of tests ($N$) is already in the numerator.
|
||||
- Conservative: the observed $n_{\ge r}$ mixes true homology and chance, and the true-null proportion $\pi_0\le1$ is taken as $1$ (Storey-type upper bound).
|
||||
- Take the smallest $r$ with $\widehat{\mathrm{FDR}}(r)\le\alpha$ as the sharing threshold.
|
||||
- Equivalent per-kmer view: BH on $p=P(R\ge r)$ with $N$ tests. The zero-truncation matters if one tests only kmers present in the index ($R\ge1$ by selection): use $P(R\ge r)/P(R\ge1)$, not $P(R\ge r)$, otherwise the test is too conservative by a factor $\sim1/\lambda$.
|
||||
|
||||
### Dependence
|
||||
|
||||
Kmers of one shared region overlap and are not independent tests. The FDR above holds per kmer; for a per-region FDR, count regions (unitigs / super-kmers, with the appropriate clumping of $N$ and $n_{\ge r}$) rather than kmers. Not settled.
|
||||
@@ -0,0 +1,222 @@
|
||||
# Pair-coverage-biased family sampling (discussion, 2026-10-07)
|
||||
|
||||
Idea: bias the family sample used by `--pseudo-alignment`/`--sankoff`/`snp-*` so that the `family_overlap` matrix of the *sample* is as uniform as possible, compensating the scarcity of shared families between distant species. Complements, does not replace, the existing proportional-by-partition and entropy-biased draws (see user doc, *Sampling design*).
|
||||
|
||||
## Motivation: observed on Primulaceae (180 genomes)
|
||||
|
||||
Scatter of patristic distance (IQ-TREE on the pseudo-alignment) vs. normalized `family_overlap` shows three pair classes:
|
||||
|
||||
- close intra-genus pairs: high overlap, well resolved;
|
||||
- inter-genus plus distant intra-genus pairs: overlap collapses by ~2 orders of magnitude, one large noisy cloud;
|
||||
- a small low-overlap cluster at the largest distances, plausibly a few poorly sequenced genomes (user hypothesis, unverified; the tree shows one tip, `Androsace_saussurei--PHA010690`, with a 2.9 terminal branch).
|
||||
|
||||
Uniform family draw reproduces this imbalance in the pseudo-alignment: distant pairs share few informative columns, close pairs many. The tree quality for deep splits is then limited by sampled-column count on those pairs, not by the total number of families.
|
||||
|
||||
## What defines a target
|
||||
|
||||
A family $f$ is carried by a genome set $S_f$ and contributes to every pair in $S_f \times S_f$. For a sample $\mathcal{F}$ with inclusion probabilities $\pi_f$, the expected sample overlap of pair $(i,j)$ is
|
||||
|
||||
$$\mathbb{E}[O^{\mathcal{F}}_{ij}] = \sum_{f:\, i,j\in S_f} \pi_f$$
|
||||
|
||||
The goal is a $\pi$ making this approximately constant over pairs. The full-index $O_{ij}$ (already computed by `--family-overlap`) is the pool size per pair; $\min_{ij} O_{ij}$ is a hard ceiling on the flat level reachable.
|
||||
|
||||
## Nesting hypothesis (user, unverified)
|
||||
|
||||
Families covering the poorest pairs also cover the richest ones: absent genome-skimming sampling bias (locus dropout), a family conserved across distant species is not expected to be missing between closer ones. If it holds, $S_f$ is approximately a clade closure and the family pool is nested by pair distance, which changes the design:
|
||||
|
||||
- no sequential/stateful draw needed: a family's weight depends only on its own reach (farthest pair covered, or equivalently its coverage of the poorest pairs), computable in one pass;
|
||||
- flattening reduces to a monotone reweighting by reach: keep (nearly) all long-reach families, thin the short-reach ones, until the sample overlap of close pairs drops to the level of distant ones.
|
||||
|
||||
My earlier remark that the weight is stateful (depends on what other families already covered) holds only without nesting.
|
||||
|
||||
## Measuring nesting: `scripts/pseudo_alignment_nesting.py`
|
||||
|
||||
Works on a pseudo-alignment FASTA plus the IQ-TREE treefile (columns = families; a call is any character other than `-`/`?`). Outputs the sample's `family_overlap` matrix and, per clade-size stratum, the *completeness* $|S_f|/|C(S_f)|$ where $C(S_f)$ is the smallest clade containing $S_f$. Tree rooted as written in the Newick unless `--outgroup`. Uses the already-existing families × genomes presence structure only; no `S_f` is stored (clade of a column = LCA of its first and last carrier in DFS leaf order, precomputed 180×180 matrix).
|
||||
|
||||
### Result, Primulaceae pseudo-alignment (180 genomes, 5.0 M columns, uniform draw, root as written)
|
||||
|
||||
| Clade size | Families | Mean completeness | Clade-closed |
|
||||
|---|---|---|---|
|
||||
| 2 | 145 047 | 1.00 | 100 % |
|
||||
| 3-4 | 272 918 | 0.71 | 20 % |
|
||||
| 5-8 | 503 114 | 0.45 | 0.6 % |
|
||||
| 9-16 | 637 896 | 0.35 | 0.1 % |
|
||||
| 17-32 | 1 656 229 | 0.21 | 0 % |
|
||||
| 33-64 | 299 632 | 0.18 | 0 % |
|
||||
| 65-128 | 181 126 | 0.08 | 0 % |
|
||||
| 129-180 | 1 307 452 | 0.12 | 0 % |
|
||||
|
||||
Overall 4.1 % clade-closed, mean completeness 0.275; off-diagonal overlap min 4, median 1 665, max 387 057, CV 1.57. **Strict nesting does not hold on this sample.** Caveats: unrooted tree, so the 129-180 stratum is inflated by families spanning the arbitrary root (rerun with `--outgroup` pending); this is the sample fed to IQ-TREE, not the full index; absence symbols `-`/`?` both count as absent.
|
||||
|
||||
### Lift test (`--lift-test`): margin-preserving null
|
||||
|
||||
Question: do families covering a rare pair $(A,B)$ also cover an abundant pair $(A,C)$ more often than margins imply? For each $A$, $B$ = its 10 % rarest partners, $C$ = its 10 % most abundant (ranked on pair overlap). Null: curveball swap randomisation of the genome × family matrix (exact row and column sums), re-selecting $B$/$C$ in every replicate with the same procedure. 200 754 sampled columns, 20 null replicates, 5 000 trades between replicates.
|
||||
|
||||
| Statistic | Observed | Null |
|
||||
|---|---|---|
|
||||
| $T=P(C\mid A,B)=O_{ABC}/O_{AB}$ | 0.559 | 0.498 ± 0.003 (no replicate ≥ observed, p at floor 1/21) |
|
||||
| $L=\sum n_A O_{ABC}/\sum O_{AB}O_{AC}$ | 1.18 | 1.694 ± 0.004 |
|
||||
|
||||
The two statistics disagree, not contradictorily: $T$ above null supports the "positive association beyond margins" statement; $L$ below null says that, relative to $P(C\mid A)$, knowing $B$ adds less than under the null — the observed abundant partners of $A$ are far more abundant than null ones (phylogenetic structure inflates the $L$ denominator). Unverified: mixing of the chain (is 5 000 trades enough? the tiny null sd may reflect correlation), sensitivity to the 10 % threshold and to the 200 k-column subsample.
|
||||
|
||||
Combined reading: association exists but is patchy; a reach-only weight is not enough, weighting must be per presence pattern $S_f$.
|
||||
|
||||
## Candidate weighting strategy: pair raking (proposal, not implemented)
|
||||
|
||||
Goal: inclusion probabilities $\pi_f$ with $\sum_{f\ni i,j}\pi_f\approx c$ for all pairs.
|
||||
|
||||
1. First-order weight: $w_f=\sum_{i,j\in S_f}1/O_{ij}$ (draw a pair uniformly, then a family uniformly among those covering it). With the presence matrix $P$ and $R=1/O$, $w=\tfrac12\,\mathrm{rowsum}(P\odot RP)$, one pass in the existing chunked loop, no $S_f$ storage.
|
||||
2. Iterate (raking/IPF with one multiplier per pair): update $R_{ij}$ from the overlap the current $\pi$ yields, recompute $w_f$; corrects the fact that large families are drawn for many pairs at once. $\pi_f=\min(1,c\,w_f)$, $c$ set to hit the target sample size; independent Poisson draw per family.
|
||||
3. Control: recompute the sample's `family_overlap` and its CV with the script above.
|
||||
|
||||
Pre-conditions: exclude anecdotal genomes first (e.g. `Androsace_saussurei--PHA010690`, terminal branch 2.9), else their tiny $O_{ij}$ dominate $R$. Overlap uniformity is only a proxy: information for a deep split is informative sites across it, not mere co-presence; to be checked on node support of the resulting tree. Unknown: whether long-reach families are rare because conserved-single-copy or because repeats/misassemblies (the latter would be amplified by up-weighting).
|
||||
|
||||
## Alternative / complement: site classes with their own models
|
||||
|
||||
Analogy with multigene supermatrices: loci differ in rate *and* in taxon occupancy. Families covering rare pairs (long reach, conserved, slow) vs. families covering close pairs only could be two (or a few) site classes, each with its own substitution model / branch-length scale (IQ-TREE partitions; edge-proportional vs edge-linked options to be checked in the manual). Independent of the weighting: sampling fixes scarcity, partitioning fixes rate heterogeneity; with class-specific scales, oversampling a class would not distort that class's branch lengths, removing the need for $1/\pi_f$ reweighting (IQ-TREE support for per-site weights unverified).
|
||||
|
||||
Caveats: reach is a continuum, two classes is a simplification and there is no criterion for the cut; class membership is defined from presence patterns, hence correlated with the data the model sees (assignment bias, worst with `?` missing data; magnitude not assessed); partitioning does not fix repeats/misassembly, it only gives them their own rate.
|
||||
|
||||
The pseudo-alignment already keeps only variable families, so branch lengths are not substitutions/site anyway (ascertainment bias, handled by `+ASC`, see the constant-column fix section in [evolutionary_distances.md](evolutionary_distances.md)). With partitions the ascertainment correction must be per class: each class excludes a different number of constant sites. Whether the existing fix applies per partition is unchecked.
|
||||
|
||||
Literature: see the next section.
|
||||
|
||||
## Literature on informative-site selection (2026-10-07; read from search extracts, not full papers)
|
||||
|
||||
- **Phylogenetic informativeness.** Townsend 2007 (*Syst. Biol.* 56:222-231) links each site's rate of change to its power to resolve splits at a given depth: invariant sites are useless, saturated sites misleading, with an optimum in between that depends on the age of the split. Implemented online as PhyDesign (requires an ultrametric tree). Relevance: supports an entropy centre $\mu$ that depends on depth (pair distance) rather than a single value; cf. the $\mu(n)$ discussion.
|
||||
- **Entropy-based trimming.** BMGE (Criscuolo & Gribaldo 2010, *BMC Evol. Biol.* 10:210) computes a per-character entropy-like score weighted by a similarity matrix and removes regions above a threshold. Uses entropy to *discard* over-variable characters, the opposite direction of the Gaussian band here.
|
||||
- **Saturation / fast sites.** Classical removal methods: Slow-Fast, LQP, TIGER, Observed Variability. Counter-evidence (Tree-of-Life protein datasets, PMC10609509): even the fastest sites carry reliable signal, trimming them can lower accuracy, slow sites recover fewer true bipartitions than the fastest. Argument against penalizing high entropy ($H\ge2$) without checking that those families do not carry the deep signal; domain (conserved proteins) differs from central-position SNPs, transfer unverified.
|
||||
- **Taxon-coverage theory.** Sanderson, McMahon & Steel 2010 (*BMC Evol. Biol.*, "Phylogenomics with incomplete taxon coverage: the limits to inference") characterise phylogenetically decisive collections of taxon sets via the four-way partition property: when the coverage pattern suffices to determine the tree uniquely irrespective of signal. Formal frame for pair-coverage sampling; applicability at 180 genomes not assessed.
|
||||
|
||||
Takeaways to check before weighting: (i) whether high-entropy families are the deep-signal ones (entropy vs carrier count measured, see below; the cross with reach is still open); (ii) depth-dependent entropy centre; (iii) whether a decisiveness criterion can serve as a sufficiency test for the sample's pair coverage.
|
||||
|
||||
## Simpler weight: poorest pair of the family (user preference, 2026-10-07)
|
||||
|
||||
$w_f=1/\min_{i,j\in S_f}O_{ij}$, $\pi_f=\min(1,c\,w_f)$. Equivalent in spirit to a reach weight but measured on actual overlaps (no tree needed), same $O(|S_f|^2)$ cost per family as the sum form. Limit: it ignores that large families also feed already-abundant pairs; flattening from above then relies only on dropping short-reach families, which is exact only under nesting, and nesting holds partially (see measurements). Untested.
|
||||
|
||||
### Estimating $O$ locally (per partition / per layer)
|
||||
|
||||
- **Per layer: not representative.** A layer is an increment of kmer content added to a partition (typically one `merge`), not a batch of genomes; all genome columns exist in every layer (`UserDocMD/formats/index_layout.md`). Layer content is dominated by kmers novel at merge time, so a layer's $O_{ij}$ reflects merge order, not the index-wide distribution; not an unbiased estimator of the global $O_{ij}$.
|
||||
- **Per partition: valid only if partitions are exchangeable samples of families** (hash-based; not verified, minimizer distribution may be skewed). Even then, the pairs that matter are the rare ones, with near-zero count per partition; a `min` over noisy estimates is biased downward (winner's curse), weights would go to pairs that were unlucky in that partition rather than globally poorest.
|
||||
- **Recommended:** compute $O$ globally once (the `--family-overlap` scan already exists), apply the weight locally per partition, keeping partitions independent. Without an extra pass: pilot estimate on a random subset of partitions.
|
||||
- **Sequential variant:** update the $O$ estimate as partitions are processed in random order, boosting still-undersampled pairs. Self-correcting but breaks partition independence and makes the result order-dependent.
|
||||
- **Quota consequence:** the current per-partition quota is proportional to candidate-family count; with weights it must be proportional to the partition's weight mass $\sum_f w_f$, otherwise partitions rich in high-weight families are undersampled.
|
||||
|
||||
## Does the entropy bias distort pair representation? (measured 2026-10-07)
|
||||
|
||||
Sample drawn with `--entropy 1 --entropy-sd 0.5` (μ=1 explicit, σ default 0.5): family weight $\exp(-2(H-1)^2)$, $H$ = `entropy15` in bits; weight 1 at $H=1$, 0.135 at $H=0$ or $2$, $\approx 3\!\times\!10^{-4}$ at $H=3$.
|
||||
|
||||
Comparison, per genome pair, of the full-index `family_overlap` $O^{\text{full}}$ (index scan) to the sample's overlap $O^{\text{samp}}$ (5.0 M columns), sampling fraction $r_{ij}=O^{\text{samp}}_{ij}/O^{\text{full}}_{ij}$ (uniform draw would give constant $r$):
|
||||
|
||||
| Quantity | Value |
|
||||
|---|---|
|
||||
| Spearman $O^{\text{full}}$ vs $O^{\text{samp}}$ | 0.994 (Pearson on logs 0.998): ranking preserved |
|
||||
| global fraction $\sum O^{\text{samp}}/\sum O^{\text{full}}$ | 0.52 % |
|
||||
| $r$ quantiles 5 / 50 / 95 % | 0.0025 / 0.0037 / 0.0055 |
|
||||
| slope $\log r$ vs $\log O^{\text{full}}$ | +0.12 |
|
||||
| Spearman($r$, patristic distance) | −0.71 |
|
||||
|
||||
Per decile of $O^{\text{full}}$, median $r$ rises from 0.0027 (poorest decile) to 0.0053 (three richest, plateau): the poorest pairs are represented about half as much as the richest, so the draw amplifies the imbalance. $r$ climbs slowly over deciles 1-6 (0.0027 to 0.0037), jumps at deciles 7-8 (0.0045-0.0052), where the $O^{\text{full}}$ distribution has a gap (689 k to 11.6 M, presumably close intra-clade pairs, unchecked).
|
||||
|
||||
Not established: (a) causality — a dependence of $r$ on pair richness is compatible with the entropy filter but no uniform control draw of the same size exists; (b) noise — the poorest sample pairs have tiny counts (min 4).
|
||||
|
||||
**Candidate mechanism: first hypothesis refuted, see the next section.** I first suggested that long-reach families, carried by many divergent genomes, spread over several states ($H\approx2\text{-}3$) and are downweighted from above by the Gaussian. The full entropy table (next section) shows the opposite: families with many carriers are mostly *low*-entropy, and are downweighted from below.
|
||||
|
||||
**Consequence for the pair-weight proposal:** entropy and reach are likely correlated, not independent criteria. Since the entropy weight of each family is known, the pair weight can compensate it exactly (divide by it) rather than compose with it; order of the two filters must then be defined.
|
||||
|
||||
## Entropy vs number of carrier genomes, full index (measured 2026-10-09)
|
||||
|
||||
Source: `phylo --shannon`, which is **always a full, unsampled scan** of every non-monomorphic minorant of every layer (`SiblingExt::shannon_entropy_csv(path)` takes no `subsample`/`entropy_bias` argument; `--subsample`/`--entropy` apply only to `--pseudo-alignment`/`--sankoff`/`snp-*`). Here: 19.7 GB CSV, 535 257 857 families, read in 11 s with DuckDB (no in-memory load). `UserDocMD/usage/phylo.md` line 121 ("`--subsample N --shannon` gives a bounded diagnostic sample") is stale and contradicts line 55 and the code; to fix.
|
||||
|
||||
| Carriers $n$ | Families | mean $H$ | median $H$ | $P(H<0.5)$ | $P(H\ge1.5)$ | mean weight ($\mu=1,\sigma=0.5$) |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 2 | 163.3 M | 0.66 | 1.00 | 34.5 % | 0 % | 0.70 |
|
||||
| 3-4 | 104.6 M | 0.94 | 0.92 | 0.5 % | 7.9 % | 0.93 |
|
||||
| 5-8 | 91.8 M | 0.83 | 0.72 | 0.1 % | 4.6 % | 0.80 |
|
||||
| 9-16 | 62.7 M | 0.70 | 0.50 | 43.6 % | 4.9 % | 0.66 |
|
||||
| 17-32 | 45.4 M | 0.59 | 0.35 | 58.9 % | 5.9 % | 0.53 |
|
||||
| 33-64 | 48.1 M | 0.34 | 0.19 | 79.7 % | 1.1 % | 0.40 |
|
||||
| 65-128 | 19.1 M | 0.37 | 0.18 | 77.4 % | 3.8 % | 0.37 |
|
||||
| 129-180 | 0.19 M | 0.83 | 0.47 | 53.0 % | 21.4 % | 0.46 |
|
||||
|
||||
Findings:
|
||||
|
||||
- Families with many carriers are mostly one dominant state plus rare variants ($H<0.5$ for ~77-80 % at $n\ge33$); only 1-4 % have $H\ge1.5$. The Gaussian at $\mu=1$ therefore penalizes them from the **low** side (weight 0.135 at $H=0$), not the high side.
|
||||
- Relative to a uniform draw, expected inclusion is about ×1.32 at 3-4 carriers and ×0.53-0.56 at 33-128 carriers: a factor ~2.4 between small and large $n$. Same order as the ~2× pair-level undersampling of poor pairs measured above; compatible, but a family-level ratio is not a pair-level one, so causality remains unproven (no uniform control draw).
|
||||
- The filter does what it is designed for (drop near-autapomorphic variants), but those low-entropy high-$n$ families are also the ones covering distant pairs, hence the side effect.
|
||||
- Informative high-$n$ families are scarce: at $n>32$ only ~19 % have $H\in[0.5,1.5)$, and the 129-180 class is 0.04 % of all families. Flattening deep-pair coverage works on little informative material whatever the weights.
|
||||
- Consequence for a $\mu(n)$ that grows with $n$: it would not recover these families (80 % sit below 0.5 bit); what matters at large $n$ is the share of families in the band around 1 bit, not the position of the centre. Conditional weight $w_H(f)/E[w_H\mid n_f]$ remains an option to make the entropy filter neutral to $n$; $E[w_H\mid n]$ is now available from this table.
|
||||
|
||||
Limit: the table has `n_genomes_present` but not the reach of the family, so the link to specific pairs is still not measured.
|
||||
|
||||
## Site concordance factors on the Primulaceae tree (measured 2026-10-12)
|
||||
|
||||
Command (IQ-TREE 3.1.3): `iqtree3 -t <treefile> -s <alignment> -m GTR+G4+ASC --scfl 100 --cf-verbose -T 5 --prefix concord`. With `-t` plus `-m`, IQ-TREE did **not** just annotate the reference tree: it ran a full ML search started from it (102 iterations, ~59 h wall-clock). `concord.treefile` differs from the input treefile by one bipartition (Robinson-Foulds 2), so sCF refer to the re-inferred tree. Without `-m`, ModelFinder is launched and required 157 GB (first attempt failed). The log also warns that all 180 sequences contain more than 50 % gaps/ambiguity. `sN` below = informative sites averaged over the 100 sampled quartets of a branch (alignment: 5 003 414 columns).
|
||||
|
||||
**Deep Primula/Androsace interface: too few informative sites (data scarcity).**
|
||||
|
||||
| Node | Tips (Primula / Androsace) | sCF | sDF1 | sDF2 | `sN` |
|
||||
|---|---|---|---|---|---|
|
||||
| 209 | 67 (43 / 24) | 27 | 47 | 26 | 17 |
|
||||
| 210 | 66 (43 / 23) | 37 | 33 | 30 | 43 |
|
||||
| 208 | 74 (50 / 24) | 54 | 16 | 29 | 190 |
|
||||
| 211 | 24 (2 / 22) | 64 | 18 | 18 | 226 |
|
||||
| 212 | 21 (2 / 19) | 30 | 38 | 32 | 184 |
|
||||
| 216 | 10 (2 / 8) | 21 | 47 | 32 | 7 |
|
||||
| 217 | 9 (1 / 8) | 43 | 20 | 36 | 7 |
|
||||
|
||||
Quartets crossing these nodes have tens to a few hundred informative sites out of 5 M columns, the quantitative counterpart of the pair-overlap scarcity. Primula tips are spread over several nodes mixed with Androsace (e.g. 43 + 24 at node 209); no monophyletic Primula in this tree.
|
||||
|
||||
**Within Androsace: conflict, not scarcity.** Nodes 283 (72 Androsace, `sN` 17 127), 292 (49 tips, 10 335), 305 (22, 11 980), 342 (11, 6 566) have thousands of informative sites but sCF 33-37 % with both alternatives nearly as supported: sites split almost evenly among the three quartet topologies. More sites would not fix this; cause not established (a poor fit of a tree model to intra-species divergence is one hypothesis, untested). Node 185 (23 Androsace): sDF1 53.6 % > sCF 30.6 %, a majority of the 893 sites prefer an alternative (real conflict).
|
||||
|
||||
**Well-supported nodes have long branches:** node 296 (45 tips, length 0.23, sCF 90 %), 341 (0.21, 91 %), 330 (0.09, 85.5 %), 181 (0.099, 89.5 %); most branches of length 0.01-0.05 have sCF 33-45 %. Degenerate: node 186 (length 0.001, `sN` 0.37) and 187 (`sN` 2.2) carry no informative site.
|
||||
|
||||
Median sCF shows no gradient with clade size (about 39-48 % in every class), so the clade-size class hides the effect; the scarcity is specific to the interface nodes above.
|
||||
|
||||
**Implication for sampling.** The part a sampling design can address is the interface scarcity: select families carried by genomes from the four groups around each target node (a quartet-targeted selection) rather than flattening the whole pair matrix. Not assessed: how many such families the index holds (the `sN` of nodes 209 and 216 suggests few). For the intra-Androsace conflict no sampling strategy is suggested by these measurements alone.
|
||||
|
||||
## Quartet coverage (measured 2026-10-12, `--quartet-test`)
|
||||
|
||||
`scripts/pseudo_alignment_nesting.py --quartet-test --quartets N` draws random genome quartets and counts exactly the columns where all four carry a call, stratified by the size of the smallest tree clade containing the four. Cost: 20 000 quartets over 5.0 M columns in about 35 s total. Exact per-family quartet weighting is out of reach ($\binom{180}{4}\approx 42.3$ M quartets; one family carried by 100 genomes covers $\binom{100}{4}\approx 3.9$ M), so quartets serve as a *diagnostic* of a sample, not as a weight.
|
||||
|
||||
Result on the current sample: 4-way count quantiles 0 / 1 / 10 / 50 / 90 / 99 / 100 % = 0, 3, 68, 200, 2 329, 66 504, 194 460 (CV 3.5, 0.1 % uncovered). By depth: median 131-209 for quartets inside clades ≥ 65 genomes versus 11 580-57 387 for clades 17-64. The ratio of the 4-way count to the smallest of the six pair overlaps has median 0.20-0.29 for deep quartets (0.53 for clades 33-64; 1 would be perfect nesting): pair overlap is only a loose upper bound of 4-way coverage, so flattening pairs does not by itself flatten quartets. 88 % of random quartets fall in the deepest class (root as written; strata with < 5 quartets are noise). Co-presence is not informativeness (a column must also differ among the four). Next measurement: repeat on a sample drawn with the pair weight and check whether the ratio moves toward 1.
|
||||
|
||||
## Two kinds of family loss: sequencing dropout vs divergence (2026-10-12)
|
||||
|
||||
- **Sequencing dropout (genome skimming, ~1× coverage).** About $e^{-1}\approx 37\%$ of each genome is uncovered, independently per family. Predicted for a family truly present in a clade of $m$ genomes: completeness ≈ 0.63-0.70 and clade-closed fraction $c^m$, independent of clade depth. Consistent with small clades (3-4 tips: completeness 0.71, 20 % closed vs $0.65^3=27\%$, $0.65^4=18\%$). Does not predict the observed decay at larger clades (0.45 at 5-8 tips, 0.21 at 17-32, 0.08-0.12 beyond).
|
||||
- **Loss by divergence (user observation).** A family is a kmer whose 30 flanking positions (15 each side for $k=31$) are identical, with a variable central base; with per-site divergence $d$ the flanks survive with probability ≈ $(1-d)^{30}\approx e^{-30d}$ (0.74, 0.21, 0.042, 0.0012 for $d$ = 0.01, 0.05, 0.10, 0.20). Phylogeny-dependent and tied to local conservation, unlike sequencing dropout. Explains the loss of completeness with clade size, and sets a floor: no weighting can sample families that no longer exist for pairs whose $e^{-30d}$ is negligible. Surviving deep families are selected for conserved flanks, hence plausibly slow-evolving (fewer variable sites for deep splits; hypothesis, untested).
|
||||
- The patristic distances of the IQ-TREE tree (1 to 3.5) come from an alignment with variable sites only, hence strongly inflated; the true $d$ is unknown and these numbers cannot be plugged into $e^{-30d}$.
|
||||
- **Test (user's plan), done 2026-10-12: see the next section.**
|
||||
|
||||
## Overlap vs Mash distance on the full index (measured 2026-10-12)
|
||||
|
||||
Inputs: `phylo --distance mash --csv --shared-kmers --nj` on the 180-genome index (distance matrix, shared-kmer matrix, NJ tree) against the full-index `family_overlap` $O_{ij}$. Mash $D$: range 0.011-0.337, median 0.20.
|
||||
|
||||
| Regression | Slope | $R^2$ |
|
||||
|---|---|---|
|
||||
| $\log O_{ij}$ vs $D$, all pairs | -26.95 | 0.979 |
|
||||
| $\log$(shared kmers) vs $D$ | -31.87 | 0.984 |
|
||||
| $\log O_{ij}$ vs $D$ + additive genome effect | -25.3 | 0.9999 |
|
||||
|
||||
Spearman($D$, $O$) = -0.95; Spearman($D$, shared kmers) = -0.95; Spearman($D$, IQ-TREE patristic) = 0.81.
|
||||
|
||||
Reading:
|
||||
|
||||
- Overlap decays log-linearly with distance, slope about -27 versus -30 predicted by flank survival (previous section). **Not an independent confirmation of the flank model**: Mash $D$ is derived from kmer sharing and $O$ counts families defined by kmers, so both measure the same shared content. The kmer slope of -32 is $-k$ by construction of Mash (implementation check only). The ratio of the two slopes, 27/32 ≈ 0.85, is below the 30/31 ≈ 0.97 expected from flank survival alone; cause unknown, the gap is modest.
|
||||
- Local slopes vary: -21 for $D\in[0.011,0.07]$ ($R^2$ 0.45), -28 for [0.07, 0.20] (0.96), -35 for $D>0.23$ (0.83). Half the pairs lie in $D\in[0.20,0.23]$, where the fit explains almost nothing ($R^2$ 0.26 and 0.07): a plateau where Mash has little resolution.
|
||||
- Genome effects (log scale, sd 0.21): lowest `Androsace_saussurei--PHA010690` (factor ≈ 0.39), `Androsace_triflora--PHA000558` (≈ 0.62), `Androsace_vesulensis--PHA010698` and `Primula_stricta_var_obesior--TROM_V_149128` (≈ 0.65); highest `Androsace_pubescens--PHA010697` (≈ 2.8). Three of the four lowest are also among the genomes with the fewest calls in the pseudo-alignment (saussurei, triflora, stricta), pubescens has the most; consistent with a coverage effect at the extremes, correlation over all genomes not computed.
|
||||
- NJ on Mash vs the IQ-TREE tree: Robinson-Foulds 182 of 354 (0.51 normalized); the Mash plateau near 0.2 probably limits NJ resolution for most pairs.
|
||||
- Consequence for normalization: the residual $\log O_{ij}-(a+bD_{ij})$ measures coverage deviation at a given distance, a normalization of $O$ that needs no per-genome call counts. Untested on a weighted sample.
|
||||
|
||||
## Data-poor genomes (measured 2026-10-12)
|
||||
|
||||
Per-genome share of the 5 003 414 sampled columns with a call: median 5.2 %, max 14.3 %; 8 genomes below 1 % and 23 below 2 %. Fewest calls: `Androsace_triflora--PHA000558` 1 936, `Primula_stricta_var_obesior--TROM_V_149128` 2 403, `Primula_carniolica--PHA007185` 4 937, `Androsace_saussurei--PHA010690` 6 986, `Primula_pumila--TROM_V_135924` 17 393, `Androsace_maxima--PHA000534` 17 939, `Primula_auricula_subsp_serratifolia--PHA007184` 30 621, `Androsace_pyrenaica--PHA000543` 37 725. Three of them (saussurei, serratifolia, pyrenaica PHA000543) are among the tips whose placement looked aberrant, previously attributed to possible mislabelling or contamination. A simpler alternative is too little data to place them reliably; testable by removing them (`--exclude-genome`) and watching deep-node support.
|
||||
|
||||
Consequence for flattening: raw $O_{ij}$ mixes evolutionary distance and per-genome coverage; flattening on it chases pairs involving data-poor genomes, for which no family exists. Normalizing, $O_{ij}/(c_ic_j)$ with $c_i$ the number of families carried by genome $i$ (per-genome counts from `--sibling-stats`), targets distance instead of coverage; the best normalization (smoothing of $c_i$, etc.) is not established.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
|
||||
- Combination with entropy bias: not independent criteria if entropy correlates with reach (see section above); options: compensate the known entropy weight inside the pair weight, or apply the Gaussian inside reach strata.
|
||||
- Pairs with near-zero $O_{ij}$ (suspected badly sequenced genomes) set the ceiling for everyone; `--min-shared-family`/`--exclude-genome` should be applied first so the floor is not dictated by anecdotal genomes.
|
||||
- Effect on the SNP-distance estimators: a non-uniform draw over pairs biases per-pair eligible-family counts, but each pair's $p$-distance is a ratio over its own eligible families, so unbiasedness per pair is expected to hold. To confirm before relying on it.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Sibling partition unification and masked canonical form (discussion)
|
||||
|
||||
Status: **open, nothing implemented**. Records the state of the reflection on (a) routing the 4 members of a sibling family to one partition and (b) a canonical form that makes siblings directly comparable. Measurements are throwaway (not in the repo); figures are for $k=31$, $m=11$, 256 partitions, 20 bacterial genomes (98.4 M kmer positions), unless stated.
|
||||
|
||||
## Problem
|
||||
|
||||
A sibling family = the 4 kmers differing only at the central base (position $c=(k-1)/2$, $k$ odd). Routing is **not** a hash of all bases: a kmer goes to `minimizer().seq_hash() & mask`, and consecutive kmers of a superkmer share their minimizer. A family splits across partitions whenever the central base falls in (or changes) the selected m-mer.
|
||||
|
||||
Fraction of families whose 4 variants are not all in one partition, standard minimizer: $4m/(k+2m+1)$ (fixed candidates $k-2m+1$ versus $4m$ variable ones). Measured: 81.4 % at $m=11$ (60.2 % at $m=7$, 46.8 % at $m=5$). This is the *theoretical* rate assuming all 4 members present; for two present members it is ≈ 68 %. Real families are overwhelmingly of size 1, so it only becomes cost where something probes variants.
|
||||
|
||||
## Masked canonical form (Can3)
|
||||
|
||||
Notation: `CentreA(K)` = K with its central base set to A; `RC(K)` its reverse complement.
|
||||
|
||||
- Can1 = `min(K, RC(K))` (current). Each sibling canonicalised alone, so some may be flipped relative to the others.
|
||||
- Can2 = orient by `CentreA(K)` vs `RC(CentreA(K))`. **Not a valid canonical form**: when the flanks are reverse complements of each other ($L=\mathrm{rc}(R)$, frequency ≈ $4^{-c}$, 1/1024 at $k=11$), $K$ and $\mathrm{RC}(K)$ get different canonicals (k=5: `ACAGT` and `ACTGT` both kept). Cause: `CentreA(RC(K)) ≠ RC(CentreA(K))` (the latter has centre T).
|
||||
- **Can3** (retained):
|
||||
|
||||
```
|
||||
Can3(K) = K if CentreA(K) < CentreA(RC(K))
|
||||
= RC(K) if CentreA(K) > CentreA(RC(K))
|
||||
= Can1(K) otherwise // self-revcomp family
|
||||
```
|
||||
|
||||
Orientation is decided on the flanks only (centre ignored), tie broken on the full kmer. Gain: the 4 canonical siblings have identical flanks in the same orientation, so they compare directly with no reverse complement. Self-revcomp families have only 2 distinct canonical members (centres A/C; T/G fold onto them). The A↔T (C↔G) substitution in that context is undetectable by any double-strand canonical set, same as under Can1 — not a loss.
|
||||
|
||||
Gain is simplification only: `central_canonical_neighbors` already canonicalises each variant independently and is correct (see `evolutionary_distances.md`, per-partition sweep). The earlier "masked-flank grouping" there is the Can2 idea; Can3 is its repaired form. **No performance gain by itself**, and it breaks existing indexes (canonical orientation changes for some kmers → MPHF hashes change; rebuild).
|
||||
|
||||
Prerequisite if adopted: `Minimizer = CanonicalKmerOf<MLen>` shares the canonical type with index kmers; an m-mer has no sibling family and must keep Can1. Decouple with a distinct `Minimizer` type (6 files outside `kmer.rs`: `routable.rs`, `packed_seq.rs`, `obiskbuilder/{rolling_stat,stream_iter,iter}.rs`, `obifastwrite`).
|
||||
|
||||
## Routing that excludes the centre (scheme B) — rejected
|
||||
|
||||
Candidates restricted to m-mers not overlapping the centre ($m\le c$): $S(j)=\min$ hash over m-mers $p\in[j,j+c-m]$, minimizer$(i)=\min(S(i),S(i+c+1))$. Family- and strand-invariant by construction (checked). But unitigs are partition-local (`pipeline.md`), and the hole moves with each kmer, so runs shrink:
|
||||
|
||||
| $m$ | mean run, current | mean run, B | bases/kmer, current → B |
|
||||
|---|---|---|---|
|
||||
| 5 | 14.9 | 6.3 | 3.02 → 5.76 |
|
||||
| 11 | 11.0 | 3.0 | 3.72 → 10.96 |
|
||||
| 15 | 9.0 | 1.0 | 4.32 → 30.9 |
|
||||
|
||||
Proxy: run = unitig (upper bound; real unitigs may merge). Independent of partition count. In `benchmark/run/global_index_count` (786 MB) `unitigs.bin` + idx ≈ 241 MB (31 %), so tripling them ≈ +60 % total index. Hashing the masked kmer per kmer (no minimizer) would cut every path at every kmer: worse.
|
||||
|
||||
Structural tension: path coherence (long unitigs) needs a key shared by consecutive kmers, family coherence needs a key independent of each kmer's own centre, which is an ordinary base for its neighbour. Decoupling sequence storage from MPHF partitioning fails because a partition can only enumerate its kmers from its own unitigs (MPHF not invertible). Redefining the family on a terminal base would cut the split rate to ≈ 17 % but is not closed under revcomp and changes the SNP semantics. Lower $m$ reduces the split rate for free but deviates from the literature $k/m\approx 2.5$ optimum (balance, bucket size), unmeasured here — $m=11$ kept.
|
||||
|
||||
## What the cross-partition cost actually is
|
||||
|
||||
- Phylo-time: only families of size > 1 trigger remote lookups (a small share).
|
||||
- Annex construction probes the 3 variants of **every** kmer (size-1 families included), ≈ 2/3 of probes crossing partitions; batch bounded to `#kmers(p)*3` per partition (not the whole index).
|
||||
- Real remaining benefit of unification: the outer sweep over source partitions could run in parallel (it is sequential today for memory and for the `q >= p` dedup rule); plus code simplification in `family_scan.rs` / `annex.rs`. Not quantified.
|
||||
- Unification would also allow an MPHF keyed on the family (one lookup, 4-bit member mask, union/intersection as OR/AND); needs the family-size histogram (1–4 members) to judge storage.
|
||||
|
||||
## Open measurements (index not on this machine)
|
||||
|
||||
1. Family-size histogram of the real index: `obikmer phylo --sibling-hist` (needs a built annex).
|
||||
2. Time and peak RSS of `--sibling-annex` on the Vaccinium/Salix index (`Stage` "sibling_annex" traces time; peak RSS from outside).
|
||||
3. Only if those show the sequential sweep or the probe volume is a real bottleneck: real-index test of scheme B or of a smaller $m$.
|
||||
|
||||
## Decision state
|
||||
|
||||
No code change decided. Can3 + `Minimizer` decoupling is the only self-contained step and is independent of the measurements above; it forces an index rebuild, so it is worth doing only if its simplification is wanted.
|
||||
@@ -71,6 +71,54 @@
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="access-detail">
|
||||
<group delimiter=" ">
|
||||
<text variable="URL" prefix="Available at: "/>
|
||||
<group prefix="(" suffix=")" delimiter=" ">
|
||||
<text term="accessed"/>
|
||||
<date variable="accessed">
|
||||
<date-part name="day" suffix=" "/>
|
||||
<date-part name="month" form="long" suffix=" "/>
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</group>
|
||||
</group>
|
||||
</macro>
|
||||
<macro name="access">
|
||||
<choose>
|
||||
<if type="webpage post post-weblog entry entry-encyclopedia dataset software report manuscript document" match="any">
|
||||
<text macro="access-detail"/>
|
||||
</if>
|
||||
<else-if variable="accessed">
|
||||
<text macro="access-detail"/>
|
||||
</else-if>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="access-fallback">
|
||||
<group delimiter=" ">
|
||||
<choose>
|
||||
<if variable="URL">
|
||||
<text macro="access-detail" suffix="."/>
|
||||
</if>
|
||||
<else-if variable="publisher">
|
||||
<text variable="publisher" prefix="Available at: " suffix="."/>
|
||||
</else-if>
|
||||
</choose>
|
||||
<text variable="note" suffix="."/>
|
||||
</group>
|
||||
</macro>
|
||||
<macro name="issued-year">
|
||||
<choose>
|
||||
<if variable="issued">
|
||||
<date variable="issued">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
</if>
|
||||
<else>
|
||||
<text term="no date" form="short"/>
|
||||
</else>
|
||||
</choose>
|
||||
</macro>
|
||||
<macro name="publisher">
|
||||
<group delimiter=", ">
|
||||
<text variable="publisher"/>
|
||||
@@ -85,9 +133,7 @@
|
||||
<layout prefix="(" suffix=")" delimiter="; ">
|
||||
<group delimiter=" ">
|
||||
<text macro="author-short"/>
|
||||
<date variable="issued">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
<text macro="issued-year"/>
|
||||
<group>
|
||||
<label variable="locator" form="short"/>
|
||||
<text variable="locator" prefix=" "/>
|
||||
@@ -102,9 +148,7 @@
|
||||
</sort>
|
||||
<layout>
|
||||
<text macro="author" suffix=" "/>
|
||||
<date variable="issued" prefix="(" suffix=")">
|
||||
<date-part name="year"/>
|
||||
</date>
|
||||
<text macro="issued-year" prefix="(" suffix=")"/>
|
||||
<choose>
|
||||
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
|
||||
<group suffix=".">
|
||||
@@ -112,6 +156,7 @@
|
||||
<text macro="editor-translator" prefix=" "/>
|
||||
</group>
|
||||
<text prefix=" " suffix="." macro="publisher"/>
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</if>
|
||||
<else-if type="thesis" match="any">
|
||||
<group suffix=".">
|
||||
@@ -119,6 +164,7 @@
|
||||
<text variable="genre" prefix=" " suffix=" Thesis."/>
|
||||
<text prefix=" " macro="publisher"/>
|
||||
</group>
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</else-if>
|
||||
<else-if type="chapter paper-conference" match="any">
|
||||
<text macro="title" prefix=" "/>
|
||||
@@ -138,6 +184,18 @@
|
||||
</group>
|
||||
</group>
|
||||
</else-if>
|
||||
<else-if type="webpage post post-weblog entry entry-encyclopedia dataset software" match="any">
|
||||
<group suffix=".">
|
||||
<text macro="title" prefix=" "/>
|
||||
<text macro="editor-translator" prefix=" "/>
|
||||
</group>
|
||||
<group prefix=" " suffix=".">
|
||||
<text variable="container-title" text-case="title" font-style="italic"/>
|
||||
<text variable="genre" prefix=", "/>
|
||||
<text macro="publisher" prefix=". "/>
|
||||
</group>
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</else-if>
|
||||
<else>
|
||||
<group suffix=".">
|
||||
<text macro="title" prefix=" "/>
|
||||
@@ -150,6 +208,14 @@
|
||||
</group>
|
||||
<text variable="page" prefix=", "/>
|
||||
</group>
|
||||
<choose>
|
||||
<if variable="container-title">
|
||||
<text macro="access" prefix=" " suffix="."/>
|
||||
</if>
|
||||
<else>
|
||||
<text macro="access-fallback" prefix=" "/>
|
||||
</else>
|
||||
</choose>
|
||||
</else>
|
||||
</choose>
|
||||
</layout>
|
||||
|
||||
@@ -768,3 +768,92 @@
|
||||
pages = {535--563},
|
||||
year = {1988}
|
||||
}
|
||||
@ARTICLE{Le-Cam1960-pe,
|
||||
title = "{An approximation theorem for the Poisson binomial distribution}",
|
||||
author = "Le Cam, Lucien",
|
||||
journal = "Pacific Journal of Mathematics",
|
||||
publisher = "Mathematical Sciences Publishers",
|
||||
volume = 10,
|
||||
number = 4,
|
||||
pages = "1181--1197",
|
||||
month = dec,
|
||||
year = 1960,
|
||||
url = "http://dx.doi.org/10.2140/pjm.1960.10.1181",
|
||||
doi = "10.2140/pjm.1960.10.1181",
|
||||
issn = "0030-8730,1945-5844"
|
||||
}
|
||||
|
||||
@ARTICLE{Hoeffding1956-bx,
|
||||
title = "{On the distribution of the number of successes in independent
|
||||
trials}",
|
||||
author = "Hoeffding, Wassily",
|
||||
journal = "The Annals of Mathematical Statistics",
|
||||
publisher = "Institute of Mathematical Statistics",
|
||||
volume = 27,
|
||||
number = 3,
|
||||
pages = "713--721",
|
||||
abstract = "Let $S$ be the number of successes in $n$ independent trials, and
|
||||
let $p_j$ denote the probability of success in the $j$th trial,
|
||||
$j = 1, 2, \cdots, n$ (Poisson trials). We consider the problem
|
||||
of finding the maximum and the minimum of $Eg(S),$ the expected
|
||||
value of a given real-valued function of $S,$ when $ES = np$ is
|
||||
fixed. It is well known that the maximum of the variance of $S$
|
||||
is attained when $p_1 = p_2 = \cdots = p_n = p.$ This can be
|
||||
interpreted as showing that the variability in the number of
|
||||
successes is highest when the successes are equally probable
|
||||
(Bernoulli trials). This interpretation is further supported by
|
||||
the following two theorems, proved in this paper. If $b$ and $c$
|
||||
are two integers, $0 \leqq b \leqq np \leqq c \leqq n,$ the
|
||||
probability $P(b \leqq S \leqq c)$ attains its minimum if and
|
||||
only if $p_1 = p_2 = \cdots = p_n = p,$ unless $b = 0$ and $c =
|
||||
n$ (Theorem 5, a corollary of Theorem 4, which gives the maximum
|
||||
and the minimum of $P(S \leqq c)$). If $g$ is a strictly convex
|
||||
function, $Eg(S)$ attains its maximum if and only if $p_1 = p_2 =
|
||||
\cdots = p_n = p$ (Theorem 3). These results are obtained with
|
||||
the help of two theorems concerning the extrema of the expected
|
||||
value of an arbitrary function $g(S)$ under the condition $ES =
|
||||
np.$ Theorem 1 gives necessary conditions for the maximum and the
|
||||
minimum of $Eg(S).$ Theorem 2 gives a partial characterization of
|
||||
the set of points at which an extremum is attained. Corollary 2.1
|
||||
states that the maximum and the minimum are attained when $p_1,
|
||||
p_2, \cdots, p_n$ take on, at most, three different values, only
|
||||
one of which is distinct from 0 and 1. Applications of Theorems 3
|
||||
and 5 to problems of estimation and testing are pointed out in
|
||||
Section 5.",
|
||||
month = sep,
|
||||
year = 1956,
|
||||
url = "http://dx.doi.org/10.1214/aoms/1177728178",
|
||||
doi = "10.1214/aoms/1177728178",
|
||||
issn = "0003-4851,2168-8990",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Gleser1975-yu,
|
||||
title = "{On the distribution of the number of successes in independent
|
||||
trials}",
|
||||
author = "Gleser, Leon Jay",
|
||||
journal = "Annals of Probability",
|
||||
publisher = "Institute of Mathematical Statistics",
|
||||
volume = 3,
|
||||
number = 1,
|
||||
pages = "182--188",
|
||||
abstract = "Let $S$ be the number of successes in $n$ independent Bernoulli
|
||||
trials, where $p_j$ is the probability of success on the $j$th
|
||||
trial. Let $\mathbf{p} = (p_1, p_2, \cdots, p_n)$, and for any
|
||||
integer $c, 0 \leqq c \leqq n$, let $H(c \mid \mathbf{p}) = P\{S
|
||||
\leqq c\}$. Let $\mathbf{p}^{(1)}$ be one possible choice of
|
||||
$\mathbf{p}$ for which $E(S) = \lambda$. For any $n \times n$
|
||||
doubly stochastic matrix $\Pi$, let $\mathbf{p}^{(2)} =
|
||||
\mathbf{p}^{(1)}\Pi$. Then in the present paper it is shown that
|
||||
$H(c \mid \mathbf{p}^{(1)}) \leqq H(c \mid \mathbf{p}^{(2)})$ for
|
||||
$0 \leqq c \leqq \lbrack\lambda - 2\rbrack$, and $H(c \mid
|
||||
\mathbf{p}^{(1)}) \geqq H(c \mid \mathbf{p}^{(2)})$ for
|
||||
$\lbrack\lambda + 2\rbrack \leqq c \leqq n$. These results
|
||||
provide a refinement of inequalities for $H(c \mid \mathbf{p})$
|
||||
obtained by Hoeffding [3]. Their derivation is achieved by
|
||||
applying consequences of the partial ordering of majorization.",
|
||||
month = feb,
|
||||
year = 1975,
|
||||
url = "http://dx.doi.org/10.2307/2959277",
|
||||
doi = "10.1214/aop/1176996461",
|
||||
issn = "0091-1798,2168-894X"
|
||||
}
|
||||
@@ -22,24 +22,102 @@ This gives the hard representation limit: $k \leq 32.$ Furthermore, OBIKmer only
|
||||
|
||||
### K-mer specificity
|
||||
|
||||
A simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are
|
||||
One simple way to illustrate the effect of k-mer size is to consider a random DNA sequence in which the four nucleotides are independent and equally probable. There are $4^k$ possible k-mers, so a particular k-mer is expected to occur once every $4^k$ positions on average. In a sequence of length $G$, the expected number of occurrences of a particular k-mer is approximately $G \cdot 4^{-k}$. When k is large enough to ensure $4^k >> G$, the chance of a second occurrence of the same k-mer becomes close to zero. Consequently, observing the same k-mer twice can only be explained by a homology relationship, except for low-complexity k-mers, which are overrepresented in genomes.
|
||||
|
||||
### Estimating False discovery rate on homology inference
|
||||
|
||||
OBIKmer uses k-mers long enough that their occurrence in multiple indexed genomes provides evidence of homology. The corresponding false discovery rate (FDR) can be expressed in terms of the probability of observing a k-mer in at least $R$ of the $S$ indexed genomes under a purely random model.
|
||||
|
||||
#### The random model
|
||||
|
||||
The random model, or null hypothesis, assumes that the $S$ genomes are unrelated and that each genome is an independent random sequence in which each nucleotide occurs with equal probability. Rejecting this hypothesis for a given k-mer indicates that its presence in multiple genomes is unlikely to be explained by chance alone, providing evidence for homology.
|
||||
|
||||
For a genome of size $G$, the number of occurrences of a given canonical k-mer $K$ is approximated by a Poisson distribution with parameter
|
||||
|
||||
$$ \lambda = 2\frac{G}{4^k}, $$
|
||||
|
||||
which is also the expected number of occurrences under the random model. The factor of two accounts for k-mer canonicalisation, which halves the k-mer space when $k$ is odd.
|
||||
|
||||
The probability of observing at least one occurrence of this k-mer is therefore
|
||||
|
||||
$$
|
||||
4^k
|
||||
\begin{aligned}
|
||||
P(n_K > 0) &= 1-e^{-\lambda} \\
|
||||
&= 1-e^{-2G/4^k}.
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
possible k-mers, and a particular k-mer is therefore expected to occur once every $4^k$ positions on average. In a sequence of length $G$, the expected number of occurrences of a particular k-mer is approximately
|
||||
Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
|
||||
|
||||
|
||||
$$
|
||||
E[n_K] \simeq \frac{G}{4^k}.
|
||||
P(X \geq r)=\sum_{j=r}^{S}\binom{S}{j}\left(1-e^{-2G/4^k}\right)^j\left(e^{-2G/4^k}\right){S-j}.
|
||||
$$
|
||||
|
||||
When $4^k\gg G$, this quantity is also a good approximation of the probability that an observed k-mer will occur at least once by chance elsewhere in a sequence of length $G$. It therefore provides a simple measure of the risk of a chance recurrence of an observed k-mer:
|
||||
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
|
||||
|
||||
$$
|
||||
P_{\mathrm{chance}}(k,G) \simeq \frac{G}{4^k}.
|
||||
p = 1-e^{-2G/4^k}.
|
||||
$$
|
||||
|
||||
Different genome sizes
|
||||
|
||||
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
|
||||
|
||||
$$ \bar G = \frac{1}{S}\sum_{i=1}^{S}G_i. $$
|
||||
|
||||
This gives a common probability
|
||||
|
||||
$$ \bar p = 1-e^{-2\bar G/4^k}, $$
|
||||
|
||||
and the number of genomes containing the k-mer can then be approximated by a binomial distribution:
|
||||
|
||||
$$ P(X\geq r)=\sum_{j=r}^{S}\binom{S}{j}\bar p^j(1-\bar p)^{S-j}. $$
|
||||
|
||||
This approximation is conservative for the upper tail. Hoeffding [-@Hoeffding1956-bx, p. 717, Eq. 26] shows that, for a fixed mean, the binomial upper tail is at least as large as the corresponding Poisson-binomial tail for
|
||||
|
||||
$$ r\geq S\bar p+1. $$
|
||||
|
||||
Using the mean genome size therefore provides a conservative estimate of the FDR.
|
||||
|
||||
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
|
||||
|
||||
### Exact tail probability
|
||||
|
||||
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
|
||||
|
||||
Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
|
||||
|
||||
$$
|
||||
\left \{
|
||||
\begin{aligned}
|
||||
d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\
|
||||
d_0&=1
|
||||
\end{aligned}
|
||||
\right .
|
||||
$$
|
||||
|
||||
before processing the first genome.
|
||||
|
||||
The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then
|
||||
|
||||
$$
|
||||
P(X\geq r)=1-\sum_{j=0}^{r-1}d_j.
|
||||
$$
|
||||
|
||||
Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$.
|
||||
|
||||
When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean
|
||||
|
||||
$$
|
||||
\lambda=\sum_{i=1}^{S}p_i.
|
||||
$$
|
||||
|
||||
The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe].
|
||||
|
||||
|
||||
*******
|
||||
|
||||
This probability decreases by a factor of four for every additional nucleotide in the k-mer. Conversely, it increases linearly with the amount of sequence searched.
|
||||
|
||||
This simple model already shows why very short k-mers provide limited specificity for genome-scale analyses. With $k=11$,
|
||||
@@ -189,34 +267,150 @@ Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nomb
|
||||
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
|
||||
|
||||
|
||||
## FDR
|
||||
## Chance sharing across genomes and the FDR of homology
|
||||
|
||||
C'est exactement la bonne question à se poser — ni la p-value par hit, ni la probabilité globale « y a-t-il au moins une collision quelque part », mais **la fraction de tes appels d'homologie qui sont du bruit**. C'est un problème de type FDR (*false discovery rate*), et il se construit directement à partir des deux briques qu'on a déjà :
|
||||
The formulas above ask how likely a match is between two sequences. The question OBIKmer actually has to answer is different: a k-mer is found in $r$ of the $S$ indexed genomes, and one wants to know whether this sharing can be explained by chance or whether it is evidence of homology. This section builds a null model for that question and turns it into a false discovery rate (FDR) for the call "a k-mer shared by at least $r$ genomes is homologous".
|
||||
|
||||
### The null model
|
||||
|
||||
The null hypothesis is that the $S$ genomes are unrelated, that is, that each one is an independent random sequence. Rejecting it for a k-mer means that chance alone cannot explain why the k-mer is shared, which is the signal of homology one is looking for.
|
||||
|
||||
The model only records whether a k-mer is present in a genome, not how many times. Two consequences follow. First, repeats inside a genome are irrelevant, and the size $G_i$ of genome $i$ is its number of **distinct canonical k-mers**. Second, a repeat family shared between genomes is paralogy, which is a form of homology; it is a true positive and not a false one. The only k-mers that could be shared by composition rather than by descent are low-complexity ones, which OBIKmer removes beforehand with its entropy filter.
|
||||
|
||||
Everything takes place in canonical space. Because $k$ is odd, no k-mer equals its own reverse complement, so the $4^k$ k-mers pair up perfectly and the space contains exactly
|
||||
|
||||
$$
|
||||
\widehat{FDR}(r) = \frac{\text{nombre de k-mers *attendus par hasard* avec }R\geq r}{\text{nombre de k-mers *observés* avec }R\geq r}
|
||||
N=\frac{4^k}{2}
|
||||
$$
|
||||
|
||||
**Le numérateur** (bruit attendu) : c'est la généralisation, au seuil $\geq r$ (pas seulement « partagé par exactement ces $r$ génomes »), de la quantité pont qu'on vient d'identifier — le nombre de k-mers testés $M$ multiplié par la probabilité par-mot sous le null :
|
||||
canonical k-mers. A given canonical k-mer is present in genome $i$ by chance with probability
|
||||
|
||||
$$
|
||||
E[\text{faux positifs}] \simeq M \cdot P_0(R\geq r) = M\sum_{j=r}^{S}\binom{S}{j}p^{j}(1-p)^{S-j}
|
||||
p_i = 1-\left(1-\frac1N\right)^{G_i}\simeq \frac{G_i}{N}=\frac{2\,G_i}{4^k}.
|
||||
$$
|
||||
|
||||
avec $p\simeq G/4^k$ (ou la version Poisson-binomiale à $G_i$ inégaux) et $M$ le nombre de k-mers canoniques réellement testés (pas $4^k/2$ théorique, mais le nombre de k-mers **effectivement présents** dans ton index — c'est cette correction, plutôt que le $4^k$ théorique, qui rend l'estimation réaliste, puisque la plupart des $4^k$ mots possibles n'apparaissent jamais dans de vrais génomes).
|
||||
This is twice the $G/4^k$ used earlier in this page, because there $G$ is a number of positions on one strand, whereas $G_i$ here counts canonical k-mers, for which both strands are already merged.
|
||||
|
||||
**Le dénominateur** (bruit + vrai signal, observé) : le nombre de k-mers de ton index dont le masque de présence a une cardinalité $\geq r$ — une donnée que `FamilyMask`/`SiblingAnnex` dans `obikphylo::siblings` te donne déjà directement, sans aucun calcul supplémentaire.
|
||||
Let $R$ be the number of genomes, out of $S$, that contain a given canonical k-mer by chance. Genomes are independent under the null, so $R$ is a sum of independent presence indicators, each with its own probability $p_i$.
|
||||
|
||||
**Ce que ça donne concrètement.** Tu n'as besoin que de trois nombres, tous déjà disponibles ou calculables :
|
||||
1. $M$ = cardinalité totale de ton set de k-mers indexés (mesure directe de l'index)
|
||||
2. $N_{obs}(r)$ = nombre de familles/k-mers avec présence $\geq r$ (mesure directe, déjà dans `FamilyMask`)
|
||||
3. $P_0(R\geq r)$ = calculé depuis le modèle nul (le $p$ de l'approche « naïve » du tour précédent)
|
||||
### Exact tail probability
|
||||
|
||||
et $\widehat{FDR}(r) = M\cdot P_0(R\geq r) / N_{obs}(r)$ te dit directement : « sur les k-mers que j'ai retenus comme preuve d'homologie au seuil $r$, telle fraction (%) est attendue être du pur hasard ».
|
||||
When genome sizes differ, the $p_i$ differ, and $R$ follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be hopeless, since there are up to $2^S$ subsets. The distribution is instead obtained by adding the genomes one at a time. Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
|
||||
|
||||
**Pourquoi c'est mieux que ce qu'on avait avant** :
|
||||
- Le $P(K_{r,s}\geq k)$ de Karlin-Ost (tour précédent) répond à « y a-t-il *au moins un* faux positif quelque part » — binaire, ne dit rien sur combien.
|
||||
- La p-value par hit répond à « ce hit précis est-il suspect » — mot par mot, ne donne pas de vue d'ensemble.
|
||||
- Le $\widehat{FDR}$ répond directement à ta question : **quelle proportion** de l'ensemble retenu est due au hasard — c'est la quantité qu'on utiliserait pour dire « à ce seuil $r$, mon jeu de zones d'homologie candidates est fiable à 95 % » ou pour choisir le $r$ minimal qui ramène le $\widehat{FDR}$ sous un seuil acceptable, à $k$ fixé par la contrainte de représentation (64 bits).
|
||||
$$
|
||||
d'_j = d_j\,(1-p_i) + d_{j-1}\,p_i, \qquad d_0=1 \text{ before the first genome.}
|
||||
$$
|
||||
|
||||
Un point à trancher avec toi avant d'aller plus loin : pour $M$, préfères-tu le nombre théorique de k-mers canoniques possibles ($4^k/2$) ou le nombre réel de k-mers distincts dans ton index (qui donnerait un $\widehat{FDR}$ beaucoup plus conservateur et réaliste, mais nécessite de le lire depuis l'index au moment du calcul plutôt que de le dériver de $k$ seul) ?
|
||||
The table $d$ only needs to remember how many genomes were hit, not which ones, and this brings the cost down to $O(S^2)$. The result does not depend on the order in which the genomes are processed. It does not depend on the k-mer either, so it is computed once per index. The upper tail is then
|
||||
|
||||
$$
|
||||
P_0(R\geq r) = 1-\sum_{j=0}^{r-1} d_j .
|
||||
$$
|
||||
|
||||
Only $d_0,\dots,d_{r-1}$ are needed, so the cost is $O(S\,r)$.
|
||||
|
||||
### Approximation with equal genome sizes
|
||||
|
||||
If all genomes are given the same size, taken as the mean
|
||||
|
||||
$$
|
||||
\bar G=\frac1S\sum_{i=1}^{S}G_i, \qquad \bar p=\frac{2\bar G}{4^k},
|
||||
$$
|
||||
|
||||
then $R$ follows a plain binomial distribution and the tail is
|
||||
|
||||
$$
|
||||
P_0(R\geq r)\simeq\sum_{j=r}^{S}\binom{S}{j}\,\bar p^{\,j}(1-\bar p)^{S-j}.
|
||||
$$
|
||||
|
||||
This sum is the regularized incomplete beta function $I_{\bar p}(r,\,S-r+1)$, available in standard statistical libraries. If it is summed by hand, the terms are obtained one from the next by $t_{j+1}=t_j\,\frac{S-j}{j+1}\,\frac{\bar p}{1-\bar p}$.
|
||||
|
||||
A theorem of Hoeffding [@Hoeffding1956-bx] shows that this approximation errs on the safe side. For a fixed mean, the Poisson-binomial distribution is more concentrated than the binomial one, so for $r\geq S\bar p+1$ the binomial tail is never smaller than the exact tail. Using the mean genome size therefore overestimates the chance of sharing and can only make the FDR more pessimistic. Gleser [@Gleser1975-yu] refines this result: replacing the $p_i$ by any partial average of themselves, not only by their full mean, also moves the upper tail towards the binomial one, provided $r-1\geq S\bar p+2$. Grouping genomes of similar size is therefore safe in the same sense. What it loses is power when genome sizes differ by orders of magnitude, since one large genome then weighs as much as several small ones in $\bar p$.
|
||||
|
||||
When all $p_i$ are small, which is the case for large $k$, the number of hits is also well approximated by a Poisson distribution of mean $\lambda=\sum_i p_i$, with an error bounded by $\sum_i p_i^2$ [@Le-Cam1960-pe].
|
||||
|
||||
For the numerical evaluation, the tail that does not contain the mode $S\bar p$ should be summed directly. When $r>S\bar p$, computing the sum from $j=r$ upwards is accurate even for very small probabilities, whereas computing $1$ minus the sum of the lower terms loses all precision once the tail falls below the machine epsilon.
|
||||
|
||||
### False discovery rate
|
||||
|
||||
The FDR of the call "present in at least $r$ genomes" is the fraction of retained k-mers expected to be pure chance:
|
||||
|
||||
$$
|
||||
\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P_0(R\geq r)}{n_{\geq r}}\right).
|
||||
$$
|
||||
|
||||
The numerator is the number of k-mers expected to reach $r$ genomes by chance if the genomes were unrelated. Each of the $N$ canonical k-mers has probability $P_0(R\geq r)$ of doing so, and no further correction for multiple testing is needed because the number of tests is already in the product. The denominator $n_{\geq r}$ is the number of distinct k-mers actually observed in at least $r$ genomes, which is read directly from the index. The observed count mixes chance and true homology, so the ratio is an upper bound on the true FDR: it implicitly assumes that all k-mers are null, and it never underestimates the noise. The minimal $r$ for which $\widehat{\mathrm{FDR}}(r)$ falls below the accepted level, for example 5 %, is the sharing threshold to use.
|
||||
|
||||
As an illustration, take $k=21$ (so $N=2.2\times10^{12}$), $S=10$ genomes and $G_i=10^9$ distinct canonical k-mers each, hence $p=4.5\times10^{-4}$. The number of k-mers expected to be shared by chance is:
|
||||
|
||||
| $r$ | $N\,P_0(R\geq r)$ |
|
||||
| ---: | ---: |
|
||||
| 2 | $2\times10^{7}$ |
|
||||
| 3 | $2.5\times10^{4}$ |
|
||||
| 4 | $\approx 20$ |
|
||||
| 5 | $\approx 0.01$ |
|
||||
|
||||
Chance alone produces millions of pairwise shared 21-mers at this genome size, a few tens of thousands shared by three genomes, and almost none shared by five or more. Whether a given $r$ is acceptable then depends on how many k-mers the index actually reports at that level.
|
||||
|
||||
### Limits of the model
|
||||
|
||||
- The FDR is computed per k-mer. Neighbouring k-mers of a shared region overlap and are not independent, so a single homologous region contributes many correlated k-mers. A per-region FDR would count regions instead of k-mers and is not covered here.
|
||||
- The probability $p_i$ assumes equiprobable, independent nucleotides. The composition biases discussed earlier in this page violate this assumption, and the resulting error is not accounted for.
|
||||
|
||||
---
|
||||
|
||||
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
|
||||
|
||||
## 1. FDR d'un ensemble de k-mers sélectionnés
|
||||
|
||||
**Estimation de l'espérance du nombre de faux positifs.** Soit $M$ le nombre de k-mers canoniques testés, $M = 4^k/2$ pour k impair. Soit $q(r)=P(X\ge r)$ la queue du modèle nul. Le nombre attendu de k-mers présents par hasard dans au moins $r$ génomes vaut
|
||||
|
||||
$$
|
||||
E[\mathrm{FP}(r)] = M\,q(r).
|
||||
$$
|
||||
|
||||
Cette égalité est exacte par linéarité de l'espérance, même si les k-mers chevauchants sont dépendants. La dépendance n'affecte que la variance, pas l'espérance. Seule l'approximation de Poisson par génome reste une approximation.
|
||||
|
||||
**FDR.** Soit $N(r)=\#\{i : r_i\ge r\}$ le nombre observé de k-mers couvrant au moins $r$ génomes. Alors
|
||||
|
||||
$$
|
||||
\mathrm{FDR}(r)=\min\!\Big(1,\ \frac{M\,q(r)}{N(r)}\Big).
|
||||
$$
|
||||
|
||||
Cette estimation est légèrement conservatrice. Elle suppose $\pi_0=1$, c'est-à-dire que presque tout l'espace $M$ est nul, ce qui est vrai pour k grand.
|
||||
|
||||
**Ensemble hétérogène en $r_i$.** Si le jeu n'est pas défini par un seuil unique, on utilise le local fdr par classe de couverture :
|
||||
|
||||
$$
|
||||
\mathrm{fdr}(r)=\frac{M\,P(X=r)}{N_r},\qquad N_r=\#\{i : r_i=r\}.
|
||||
$$
|
||||
|
||||
Le FDR de l'ensemble est alors la moyenne $\frac1N\sum_i \mathrm{fdr}(r_i)$. Pour un ensemble défini par seuil $r_{\min}$, cela redonne exactement la formule précédente.
|
||||
|
||||
**Remarque.** $M\,p\approx G$, d'où $E[\mathrm{FP}(r)]\approx\binom{S}{r}\,G^r/M^{r-1}$ en régime de petits $p$. Cela donne une lecture directe de la quantité.
|
||||
|
||||
## 2. Markov d'ordre 0 avec fréquences inégales
|
||||
|
||||
Soit $\pi_b$ la fréquence du nucléotide $b$. Pour un k-mer $w$ de composition $n=(n_A,n_C,n_G,n_T)$ avec $\sum n_b=k$, on a $P(w)=\prod_b \pi_b^{n_b}$.
|
||||
|
||||
**Attendu par génome.** Le k-mer canonique $K=\{w,\overline w\}$ est vu quand le brin direct porte $w$ ou $\overline w$, donc
|
||||
|
||||
$$
|
||||
\lambda_K = G\,\big(P(w)+P(\overline w)\big)=G\Big(\prod_b\pi_b^{n_b}+\pi_A^{n_T}\pi_T^{n_A}\pi_C^{n_G}\pi_G^{n_C}\Big).
|
||||
$$
|
||||
|
||||
Cela ne dépend que de la composition. Si $\pi_A=\pi_T$ et $\pi_C=\pi_G$, on retrouve $2P(w)$.
|
||||
|
||||
**Faux positifs.** Le test n'est plus homogène : chaque composition a sa propre $p(n)=1-e^{-\lambda(n)}$. Il faut donc sommer sur les classes de composition (un k-mer et son complément inverse partagent le même $\lambda$, d'où le facteur ½) :
|
||||
|
||||
$$
|
||||
E[\mathrm{FP}(r)]=\frac12\sum_{n}\binom{k}{n_A,n_C,n_G,n_T}\,P\big(X\ge r\mid p(n)\big).
|
||||
$$
|
||||
|
||||
Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Elle est donc légère avec la binomiale de la moyenne, ou avec une Poisson-binomiale par composition si les $G_i$ diffèrent. Si les $\pi_b$ varient d'un génome à l'autre (GC différent), les $p_i(n)$ se construisent par génome.
|
||||
|
||||
**Effet attendu.** La moyenne de $\lambda_w$ sur tous les k-mers reste $2G/4^k$, car $\sum_w P(w)=1$. Dans le régime de petits $p$, le terme dominant $\binom{S}{r}p^r$ est convexe en $\lambda$. Par Jensen, le modèle équiprobable est donc le minimum de $E[\mathrm{FP}]$, et la déviation de composition ne peut qu'augmenter le FDR. Les k-mers riches en AT ou GC dominent la somme, ce qui rejoint ton remarque sur la faible complexité. C'est un résultat sur le terme dominant, pas une preuve pour la queue exacte.
|
||||
|
||||
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
|
||||
|
||||
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
|
||||
@@ -111,6 +111,7 @@ Without `--subsample`, every variable family (family size ≥ 2) is used. With `
|
||||
|
||||
| Column | Meaning |
|
||||
|---|---|
|
||||
| `partition` | the index partition that holds the family |
|
||||
| `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
|
||||
| `family_idx` | the family's position within that layer |
|
||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||
@@ -118,7 +119,7 @@ Without `--subsample`, every variable family (family size ≥ 2) is used. With `
|
||||
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
|
||||
| `n_genomes_present` | how many genomes the entropy was computed over |
|
||||
|
||||
Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of a full-index pass — useful for choosing `--entropy`/`--entropy-sd` values before a full run.
|
||||
`--shannon` always scans every non-monomorphic family of the whole index; `--subsample`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
||||
|
||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||
|
||||
|
||||
@@ -36,7 +36,10 @@ nav:
|
||||
- Entropy filter: theory/entropy.md
|
||||
- Minimizer selection: theory/minimizer.md
|
||||
- Partitioning architecture: theory/indexing.md
|
||||
- Chance sharing across genomes (discussion): theory/chance_sharing.md
|
||||
- Central-position SNP distance (discussion): theory/evolutionary_distances.md
|
||||
- Sibling partition unification (discussion): theory/sibling_partition_unification.md
|
||||
- Pair-coverage-biased family sampling (discussion): theory/pair_coverage_sampling.md
|
||||
- Implementation:
|
||||
- SuperKmer: implementation/superkmer.md
|
||||
- Kmer: implementation/kmer.md
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,395 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Family overlap and clade-closure ("nesting") of a SNP pseudo-alignment.
|
||||
|
||||
Each alignment column is one family; a genome "carries a call" at a column
|
||||
when its character is not an absence symbol (default '-' and '?').
|
||||
|
||||
Outputs (prefix given by -o):
|
||||
<prefix>_family_overlap.csv genome x genome matrix: number of columns where
|
||||
both genomes carry a call (diagonal = 0, same
|
||||
convention as `obikmer phylo --family-overlap`)
|
||||
<prefix>_nesting.csv per clade-size stratum: how closed the families
|
||||
are under the tree
|
||||
|
||||
Nesting measure: for a family carried by the genome set S, let C(S) be the
|
||||
smallest tree clade containing S. completeness = |S| / |C(S)|. A value of 1
|
||||
means the family is clade-closed (every genome of the clade carries it); low
|
||||
values mean dropout or patchy conservation inside the clade.
|
||||
|
||||
The tree is rooted as written in the Newick file unless --outgroup is given
|
||||
(rooted at the neighbour of that tip, which then becomes a child of the root).
|
||||
|
||||
Optional lift test (--lift-test): do families covering a rare pair (A,B) also
|
||||
cover an abundant pair (A,C) more often than the margins alone imply? For each
|
||||
genome A, B ranges over its --lift-frac rarest partners and C over its most
|
||||
abundant ones (ranked on pair overlap). Statistics, on a random column subsample:
|
||||
T = mean over triples of O_ABC / O_AB = P(C | A,B)
|
||||
L = sum n_A * O_ABC / sum O_AB * O_AC (lift; 1 if B,C independent given A)
|
||||
Null: curveball swap randomisation of the genome x family presence matrix, which
|
||||
preserves every genome's family count and every family's size exactly. Each null
|
||||
replicate re-selects the rare/abundant partners from its own overlaps, so the
|
||||
selection procedure is identical to the observed one.
|
||||
|
||||
Optional quartet test (--quartet-test): does flattening pair overlaps also flatten
|
||||
quartet coverage? --quartets N genome quartets are drawn uniformly; for each, the
|
||||
number of columns where all four genomes carry a call is counted exactly. Reported
|
||||
per stratum of the quartet's depth (size of the smallest tree clade containing the
|
||||
four genomes): count distribution, share of uncovered quartets, and the ratio of the
|
||||
4-way count to the smallest of the six pair overlaps (1 = perfect nesting; the pair
|
||||
overlap is an upper bound of the 4-way count).
|
||||
|
||||
Usage
|
||||
-----
|
||||
pseudo_alignment_nesting.py ALIGNMENT.fasta TREEFILE -o prefix
|
||||
[--outgroup LABEL] [--absent '-?'] [--chunk 131072]
|
||||
[--quartet-test [--quartets 20000]]
|
||||
[--lift-test [--lift-columns 200000] [--lift-frac 0.1]
|
||||
[--null-reps 20] [--trades 5000] [--seed N]]
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
def read_fasta_presence(path, absent):
|
||||
"""Return (labels, packed presence bits per genome, alignment length)."""
|
||||
absent_lut = np.zeros(256, dtype=bool)
|
||||
for ch in absent:
|
||||
absent_lut[ord(ch)] = True
|
||||
labels, packed, length = [], [], None
|
||||
|
||||
def flush(parts):
|
||||
nonlocal length
|
||||
seq = np.frombuffer(b"".join(parts), dtype=np.uint8)
|
||||
if length is None:
|
||||
length = len(seq)
|
||||
elif len(seq) != length:
|
||||
sys.exit(f"{labels[-1]}: length {len(seq)} != {length}")
|
||||
packed.append(np.packbits(~absent_lut[seq]))
|
||||
|
||||
parts = None
|
||||
with open(path, "rb") as fh:
|
||||
for line in fh:
|
||||
if line.startswith(b">"):
|
||||
if parts is not None:
|
||||
flush(parts)
|
||||
labels.append(line[1:].split()[0].decode())
|
||||
parts = []
|
||||
else:
|
||||
parts.append(line.strip())
|
||||
if parts is not None:
|
||||
flush(parts)
|
||||
return labels, np.vstack(packed), length
|
||||
|
||||
|
||||
def parse_newick(text):
|
||||
"""Parse Newick into (names, parent, length) arrays; internal labels ignored."""
|
||||
names, parent, length = [], [], []
|
||||
stack, cur = [], None
|
||||
i, n = 0, len(text)
|
||||
|
||||
def new_node():
|
||||
names.append(None)
|
||||
parent.append(-1)
|
||||
length.append(0.0)
|
||||
return len(names) - 1
|
||||
|
||||
while i < n:
|
||||
c = text[i]
|
||||
if c in " \n\r\t":
|
||||
i += 1
|
||||
elif c == "(":
|
||||
if stack:
|
||||
node = stack[-1] # the placeholder child becomes this internal node
|
||||
else:
|
||||
node = new_node()
|
||||
stack.append(node)
|
||||
cur = None
|
||||
i += 1
|
||||
child = new_node()
|
||||
parent[child] = node
|
||||
stack.append(child)
|
||||
elif c == ",":
|
||||
stack.pop()
|
||||
child = new_node()
|
||||
parent[child] = stack[-1]
|
||||
stack.append(child)
|
||||
i += 1
|
||||
elif c == ")":
|
||||
stack.pop()
|
||||
cur = stack[-1]
|
||||
i += 1
|
||||
j = i
|
||||
while j < n and text[j] not in ":,);":
|
||||
j += 1
|
||||
i = j # internal label (support values) discarded
|
||||
elif c == ":":
|
||||
j = i + 1
|
||||
while j < n and text[j] not in ",);":
|
||||
j += 1
|
||||
length[stack[-1] if cur is None else cur] = float(text[i + 1:j])
|
||||
cur = None
|
||||
i = j
|
||||
elif c == ";":
|
||||
break
|
||||
else:
|
||||
j = i
|
||||
while j < n and text[j] not in ":,);":
|
||||
j += 1
|
||||
names[stack[-1]] = text[i:j]
|
||||
i = j
|
||||
return names, parent, length
|
||||
|
||||
|
||||
def leaf_order_matrices(names, parent, length, outgroup):
|
||||
"""DFS leaf order plus LCA-clade-size and patristic matrices in that order."""
|
||||
adj = [[] for _ in names]
|
||||
for v, p in enumerate(parent):
|
||||
if p >= 0:
|
||||
adj[v].append((p, length[v]))
|
||||
adj[p].append((v, length[v]))
|
||||
root = next(v for v, p in enumerate(parent) if p < 0)
|
||||
if outgroup is not None:
|
||||
tip = names.index(outgroup)
|
||||
root = adj[tip][0][0]
|
||||
|
||||
order, depth, children = [], {root: 0.0}, {}
|
||||
stack, seen = [root], {root}
|
||||
while stack:
|
||||
v = stack.pop()
|
||||
kids = [(u, w) for u, w in adj[v] if u not in seen]
|
||||
for u, w in kids:
|
||||
seen.add(u)
|
||||
depth[u] = depth[v] + w
|
||||
children[v] = [u for u, _ in kids]
|
||||
stack.extend(reversed(children[v]))
|
||||
order.append(v)
|
||||
|
||||
leaves = [v for v in order if not children[v]]
|
||||
rank = {v: r for r, v in enumerate(leaves)}
|
||||
n = len(leaves)
|
||||
span = {}
|
||||
for v in reversed(order):
|
||||
span[v] = (rank[v], rank[v] + 1) if not children[v] else (
|
||||
span[children[v][0]][0], span[children[v][-1]][1])
|
||||
|
||||
clade = np.ones((n, n), dtype=np.int32)
|
||||
lca_depth = np.zeros((n, n))
|
||||
for v in order:
|
||||
kids = children[v]
|
||||
for a in range(len(kids)):
|
||||
for b in range(a + 1, len(kids)):
|
||||
(a0, a1), (b0, b1) = span[kids[a]], span[kids[b]]
|
||||
size = span[v][1] - span[v][0]
|
||||
clade[a0:a1, b0:b1] = clade[b0:b1, a0:a1] = size
|
||||
lca_depth[a0:a1, b0:b1] = lca_depth[b0:b1, a0:a1] = depth[v]
|
||||
leaf_depth = np.array([depth[v] for v in leaves])
|
||||
patristic = leaf_depth[:, None] + leaf_depth[None, :] - 2 * lca_depth
|
||||
return [names[v] for v in leaves], clade, patristic
|
||||
|
||||
|
||||
def lift_stats(pres, frac):
|
||||
"""(L, T) for the rare/abundant partner triples of every genome (see module doc)."""
|
||||
n = pres.shape[0]
|
||||
pf = pres.astype(np.float32)
|
||||
overlap = pf @ pf.T
|
||||
k = max(1, round(frac * (n - 1)))
|
||||
num = den = ratio_sum = 0.0
|
||||
n_ratio = 0
|
||||
for a in range(n):
|
||||
order = [j for j in np.argsort(overlap[a], kind="stable") if j != a]
|
||||
rare, abundant = order[:k], order[-k:]
|
||||
cols = pres[a]
|
||||
n_a = float(cols.sum())
|
||||
if n_a == 0:
|
||||
continue
|
||||
o_abc = pf[rare][:, cols] @ pf[abundant][:, cols].T
|
||||
o_ab, o_ac = overlap[a, rare].astype(np.float64), overlap[a, abundant].astype(np.float64)
|
||||
num += n_a * float(o_abc.sum())
|
||||
den += float((o_ab[:, None] * o_ac[None, :]).sum())
|
||||
ok = o_ab > 0
|
||||
ratio_sum += float((o_abc[ok] / o_ab[ok, None]).sum())
|
||||
n_ratio += int(ok.sum()) * k
|
||||
return num / den, ratio_sum / n_ratio
|
||||
|
||||
|
||||
def curveball(pres, trades, rng):
|
||||
"""In-place curveball trades: preserve every row sum and every column sum."""
|
||||
n = pres.shape[0]
|
||||
for _ in range(trades):
|
||||
i, j = rng.choice(n, size=2, replace=False)
|
||||
a, b = pres[i], pres[j]
|
||||
diff = np.flatnonzero(a ^ b)
|
||||
k = int((a & ~b).sum())
|
||||
diff = rng.permutation(diff)
|
||||
a[diff] = False
|
||||
b[diff] = False
|
||||
a[diff[:k]] = True
|
||||
b[diff[k:]] = True
|
||||
|
||||
|
||||
def quartet_test(packed, overlap, tree_rank, clade, n_quartets, bin_edges, rng):
|
||||
"""Exact 4-way co-presence counts for random genome quartets.
|
||||
|
||||
Returns (counts, ratio_to_min_pair, depth_stratum), one entry per quartet.
|
||||
"""
|
||||
n = packed.shape[0]
|
||||
quartets = np.argpartition(rng.random((n_quartets, n)), 3, axis=1)[:, :4]
|
||||
counts = np.zeros(n_quartets, dtype=np.int64)
|
||||
block = 2048 # bytes of packed columns per pass: bounds the (Q, block) gathers
|
||||
for a in range(0, packed.shape[1], block):
|
||||
blk = packed[:, a:a + block]
|
||||
acc = blk[quartets[:, 0]] & blk[quartets[:, 1]]
|
||||
acc &= blk[quartets[:, 2]]
|
||||
acc &= blk[quartets[:, 3]]
|
||||
counts += np.bitwise_count(acc).sum(axis=1, dtype=np.int64)
|
||||
|
||||
pair_min = np.full(n_quartets, np.iinfo(np.int64).max)
|
||||
for x in range(4):
|
||||
for y in range(x + 1, 4):
|
||||
pair_min = np.minimum(pair_min, overlap[quartets[:, x], quartets[:, y]])
|
||||
ranks = tree_rank[quartets]
|
||||
csize = clade[ranks.min(axis=1), ranks.max(axis=1)]
|
||||
stratum = np.searchsorted(bin_edges, csize, side="right") - 1
|
||||
return counts, counts / np.maximum(pair_min, 1), stratum
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
ap.add_argument("alignment")
|
||||
ap.add_argument("treefile")
|
||||
ap.add_argument("-o", "--output", required=True, help="output file prefix")
|
||||
ap.add_argument("--outgroup", help="tip label to root the tree on")
|
||||
ap.add_argument("--absent", default="-?", help="absence symbols (default '-?')")
|
||||
ap.add_argument("--chunk", type=int, default=131072,
|
||||
help="columns per chunk, multiple of 8 (default 131072)")
|
||||
ap.add_argument("--quartet-test", action="store_true",
|
||||
help="measure exact 4-way coverage of random genome quartets")
|
||||
ap.add_argument("--quartets", type=int, default=20000,
|
||||
help="number of random quartets (default 20000)")
|
||||
ap.add_argument("--lift-test", action="store_true",
|
||||
help="run the margin-preserving lift test (see module doc)")
|
||||
ap.add_argument("--lift-columns", type=int, default=200000,
|
||||
help="random columns used by the lift test (default 200000)")
|
||||
ap.add_argument("--lift-frac", type=float, default=0.1,
|
||||
help="fraction of partners ranked rare / abundant (default 0.1)")
|
||||
ap.add_argument("--null-reps", type=int, default=20)
|
||||
ap.add_argument("--trades", type=int, default=5000,
|
||||
help="curveball trades between replicates, also burn-in (default 5000)")
|
||||
ap.add_argument("--seed", type=int, default=None)
|
||||
args = ap.parse_args()
|
||||
if args.chunk % 8:
|
||||
sys.exit("--chunk must be a multiple of 8")
|
||||
|
||||
rng = np.random.default_rng(args.seed)
|
||||
labels, packed, length = read_fasta_presence(args.alignment, args.absent)
|
||||
keep_p = min(1.0, args.lift_columns / length) if args.lift_test else 0.0
|
||||
sampled = []
|
||||
with open(args.treefile) as fh:
|
||||
names, parent, blen = parse_newick(fh.read())
|
||||
leaf_labels, clade, patristic = leaf_order_matrices(names, parent, blen, args.outgroup)
|
||||
if set(leaf_labels) != set(labels):
|
||||
sys.exit("tree tips and alignment records differ")
|
||||
n = len(labels)
|
||||
tree_row = np.array([labels.index(l) for l in leaf_labels]) # tree rank -> alignment row
|
||||
|
||||
bin_edges = np.array([2, 3, 5, 9, 17, 33, 65, 129, n + 1]) # clade-size strata
|
||||
nb = len(bin_edges) - 1
|
||||
cnt = np.zeros(nb, dtype=np.int64)
|
||||
closed = np.zeros(nb, dtype=np.int64)
|
||||
sum_ratio = np.zeros(nb)
|
||||
sum_missing = np.zeros(nb)
|
||||
overlap = np.zeros((n, n), dtype=np.int64)
|
||||
n_cols_lt2 = 0
|
||||
|
||||
for a in range(0, length, args.chunk):
|
||||
b = min(a + args.chunk, length)
|
||||
pres = np.unpackbits(packed[:, a // 8:(b + 7) // 8], axis=1)[:, :b - a]
|
||||
if keep_p:
|
||||
sampled.append(pres[:, rng.random(b - a) < keep_p].astype(bool))
|
||||
overlap += np.rint(pres.astype(np.float32) @ pres.astype(np.float32).T).astype(np.int64)
|
||||
|
||||
t = pres[tree_row]
|
||||
size = t.sum(axis=0)
|
||||
keep = size >= 2
|
||||
n_cols_lt2 += int((~keep).sum())
|
||||
t, size = t[:, keep], size[keep]
|
||||
lo = t.argmax(axis=0)
|
||||
hi = n - 1 - t[::-1].argmax(axis=0)
|
||||
csize = clade[lo, hi]
|
||||
stratum = np.searchsorted(bin_edges, csize, side="right") - 1
|
||||
cnt += np.bincount(stratum, minlength=nb)
|
||||
closed += np.bincount(stratum, weights=(size == csize), minlength=nb).astype(np.int64)
|
||||
sum_ratio += np.bincount(stratum, weights=size / csize, minlength=nb)
|
||||
sum_missing += np.bincount(stratum, weights=csize - size, minlength=nb)
|
||||
|
||||
np.fill_diagonal(overlap, 0)
|
||||
with open(f"{args.output}_family_overlap.csv", "w") as fh:
|
||||
fh.write("genome," + ",".join(labels) + "\n")
|
||||
for l, row in zip(labels, overlap):
|
||||
fh.write(l + "," + ",".join(map(str, row)) + "\n")
|
||||
|
||||
with open(f"{args.output}_nesting.csv", "w") as fh:
|
||||
fh.write("clade_size_min,clade_size_max,n_families,mean_completeness,"
|
||||
"frac_clade_closed,mean_missing_in_clade\n")
|
||||
for k in range(nb):
|
||||
if cnt[k]:
|
||||
fh.write(f"{bin_edges[k]},{bin_edges[k + 1] - 1},{cnt[k]},"
|
||||
f"{sum_ratio[k] / cnt[k]:.4f},{closed[k] / cnt[k]:.4f},"
|
||||
f"{sum_missing[k] / cnt[k]:.3f}\n")
|
||||
|
||||
off = overlap[np.triu_indices(n, 1)]
|
||||
print(f"genomes={n} columns={length} (columns with <2 calls skipped for nesting: {n_cols_lt2})")
|
||||
print(f"overlap off-diagonal: min={off.min()} median={int(np.median(off))} "
|
||||
f"max={off.max()} CV={off.std() / off.mean():.3f}")
|
||||
tot = cnt.sum()
|
||||
print(f"clade-closed families: {closed.sum()}/{tot} ({closed.sum() / tot:.3f}); "
|
||||
f"mean completeness {sum_ratio.sum() / tot:.3f}")
|
||||
print(f"wrote {args.output}_family_overlap.csv and {args.output}_nesting.csv")
|
||||
|
||||
if args.quartet_test:
|
||||
tree_rank = np.empty(n, dtype=np.int64)
|
||||
tree_rank[tree_row] = np.arange(n)
|
||||
counts, ratio, stratum = quartet_test(
|
||||
packed, overlap, tree_rank, clade, args.quartets, bin_edges, rng)
|
||||
print(f"quartet test: {args.quartets} random quartets over {length} columns")
|
||||
q = np.quantile(counts, [0, .01, .1, .5, .9, .99, 1])
|
||||
print("4-way count quantiles 0/1/10/50/90/99/100 %: " + " ".join(f"{v:.0f}" for v in q)
|
||||
+ f"; uncovered: {(counts == 0).mean():.4f}; CV={counts.std() / max(counts.mean(), 1e-9):.2f}")
|
||||
with open(f"{args.output}_quartets.csv", "w") as fh:
|
||||
fh.write("depth_clade_min,depth_clade_max,n_quartets,median_count,q10_count,"
|
||||
"frac_uncovered,median_ratio_to_min_pair\n")
|
||||
for k in range(nb):
|
||||
m = stratum == k
|
||||
if m.any():
|
||||
c = counts[m]
|
||||
fh.write(f"{bin_edges[k]},{bin_edges[k + 1] - 1},{int(m.sum())},{np.median(c):.0f},"
|
||||
f"{np.quantile(c, .1):.0f},{(c == 0).mean():.4f},{np.median(ratio[m]):.4f}\n")
|
||||
print(f"wrote {args.output}_quartets.csv")
|
||||
|
||||
if args.lift_test:
|
||||
pres = np.ascontiguousarray(np.hstack(sampled))
|
||||
print(f"lift test on {pres.shape[1]} columns, {args.null_reps} null replicates")
|
||||
rows = [("observed",) + lift_stats(pres, args.lift_frac)]
|
||||
curveball(pres, args.trades, rng) # burn-in
|
||||
for r in range(args.null_reps):
|
||||
rows.append((f"null{r + 1}",) + lift_stats(pres, args.lift_frac))
|
||||
curveball(pres, args.trades, rng)
|
||||
with open(f"{args.output}_lift_test.csv", "w") as fh:
|
||||
fh.write("replicate,L,T\n")
|
||||
for name, l, t in rows:
|
||||
fh.write(f"{name},{l:.6f},{t:.6f}\n")
|
||||
for idx, label in ((1, "L (lift)"), (2, "T = P(C|A,B)")):
|
||||
obs = rows[0][idx]
|
||||
null = np.array([r[idx] for r in rows[1:]])
|
||||
p = (1 + (null >= obs).sum()) / (1 + len(null))
|
||||
print(f"{label}: observed={obs:.4f} null mean={null.mean():.4f} "
|
||||
f"sd={null.std(ddof=1):.4f} z={(obs - null.mean()) / null.std(ddof=1):.1f} "
|
||||
f"p(null>=obs)={p:.3f}")
|
||||
print(f"wrote {args.output}_lift_test.csv")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -51,28 +51,25 @@ pub(crate) fn family_entropy(genome_mask: &[u8]) -> Option<(f64, usize)> {
|
||||
Some((h, present as usize))
|
||||
}
|
||||
|
||||
/// Shannon entropy (bits) of one family's per-genome states, reduced to the
|
||||
/// 4 plain nucleotide symbols instead of the 15-state space above — kept
|
||||
/// alongside it (not instead of it) purely to measure how much the two
|
||||
/// diverge on real data. A genome carrying several bases at once
|
||||
/// (`genome_mask[g]` with more than one bit set) contributes to *each* of
|
||||
/// those bases' counts — counted once per base present, not split
|
||||
/// fractionally, and not folded into a combined state the way
|
||||
/// [`family_entropy`] does. The denominator is therefore the total number
|
||||
/// of base-occurrences summed over every present genome, not the genome
|
||||
/// count. `None` under the same condition as [`family_entropy`].
|
||||
/// Shannon entropy (bits) of one family's per-genome states over the 4 plain
|
||||
/// nucleotide symbols, restricted to the genomes carrying exactly one member
|
||||
/// of the family (`genome_mask[g].count_ones() == 1`) — the genomes that stay
|
||||
/// informative under `--no-ambiguity`, where a genome carrying several
|
||||
/// members (paralogues, not read ambiguity) becomes missing data. Genomes
|
||||
/// where the family is absent (`genome_mask[g] == 0`) and genomes carrying
|
||||
/// several members are both left out of the denominator, which is returned
|
||||
/// as the second value (`n_genomes_unambiguous`). `None` if no genome carries
|
||||
/// exactly one member.
|
||||
pub(crate) fn family_entropy_4(genome_mask: &[u8]) -> Option<(f64, usize)> {
|
||||
let mut freq = [0u32; 4];
|
||||
let mut total = 0u32;
|
||||
let mut n_single = 0u32;
|
||||
for &m in genome_mask {
|
||||
for (base, count) in freq.iter_mut().enumerate() {
|
||||
if m & (1 << base) != 0 {
|
||||
*count += 1;
|
||||
total += 1;
|
||||
if m.count_ones() == 1 {
|
||||
freq[m.trailing_zeros() as usize] += 1;
|
||||
n_single += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
if total == 0 {
|
||||
if n_single == 0 {
|
||||
return None;
|
||||
}
|
||||
let mut h = 0.0f64;
|
||||
@@ -80,10 +77,37 @@ pub(crate) fn family_entropy_4(genome_mask: &[u8]) -> Option<(f64, usize)> {
|
||||
if c == 0 {
|
||||
continue;
|
||||
}
|
||||
let p = f64::from(c) / f64::from(total);
|
||||
let p = f64::from(c) / f64::from(n_single);
|
||||
h -= p * p.log2();
|
||||
}
|
||||
Some((h, total as usize))
|
||||
Some((h, n_single as usize))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn entropy_4_balanced_two_states_is_one_bit() {
|
||||
// two genomes carry base 0 (A), two carry base 2 (G); one is absent
|
||||
let (h, n) = family_entropy_4(&[0b0001, 0b0001, 0b0100, 0b0100, 0]).unwrap();
|
||||
assert!((h - 1.0).abs() < 1e-12);
|
||||
assert_eq!(n, 4);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entropy_4_ignores_genomes_with_several_members() {
|
||||
// the two-member genomes (0b0011, 0b1100) are left out entirely
|
||||
let (h, n) = family_entropy_4(&[0b0001, 0b0001, 0b0011, 0b1100]).unwrap();
|
||||
assert_eq!(h, 0.0);
|
||||
assert_eq!(n, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entropy_4_is_none_without_unambiguous_genome() {
|
||||
assert!(family_entropy_4(&[0, 0b0011, 0b0110]).is_none());
|
||||
assert!(family_entropy_4(&[]).is_none());
|
||||
}
|
||||
}
|
||||
|
||||
/// Builds one layer's entropy annex if it doesn't already exist, and/or
|
||||
|
||||
@@ -45,10 +45,13 @@ pub trait SiblingExt {
|
||||
fn build_sibling_annex(&self) -> OKIResult<()>;
|
||||
|
||||
/// Write a CSV (`partition,layer,family_idx,entropy15,entropy4,
|
||||
/// family_size,n_genomes_present`) of per-family Shannon entropy — both
|
||||
/// [`family_entropy`] (15 non-empty states, the project's settled
|
||||
/// definition) and [`family_entropy_4`] (plain 4-symbol reduction, kept
|
||||
/// alongside for comparison) — one row per non-monomorphic minorant of
|
||||
/// family_size,n_genomes_present,n_genomes_unambiguous`) of per-family
|
||||
/// Shannon entropy — both [`family_entropy`] (15 non-empty states over
|
||||
/// every genome carrying the family, the project's settled definition)
|
||||
/// and [`family_entropy_4`] (4 plain bases over the genomes carrying
|
||||
/// exactly one member only, i.e. what survives `--no-ambiguity`;
|
||||
/// `n_genomes_unambiguous` is its denominator, `0` meaning `entropy4`
|
||||
/// is not defined) — one row per non-monomorphic minorant of
|
||||
/// every cached layer, from an already-built sibling annex (run
|
||||
/// [`build_sibling_annex`](Self::build_sibling_annex) first).
|
||||
///
|
||||
@@ -166,7 +169,7 @@ impl SiblingExt for IndexCache {
|
||||
let fast_mode = is_fast_mode(self);
|
||||
|
||||
let mut f = BufWriter::new(std::fs::File::create(path).map_err(OKIError::Io)?);
|
||||
writeln!(f, "partition,layer,family_idx,entropy15,entropy4,family_size,n_genomes_present")
|
||||
writeln!(f, "partition,layer,family_idx,entropy15,entropy4,family_size,n_genomes_present,n_genomes_unambiguous")
|
||||
.map_err(OKIError::Io)?;
|
||||
|
||||
let pb = progress_bar("shannon_entropy", self.n_partition() as u64, "partitions");
|
||||
@@ -205,10 +208,10 @@ impl SiblingExt for IndexCache {
|
||||
let Some((h15, n_present)) = family_entropy(genome_mask) else {
|
||||
return;
|
||||
};
|
||||
let h4 = family_entropy_4(genome_mask).map_or(0.0, |(h, _)| h);
|
||||
let (h4, n_unambiguous) = family_entropy_4(genome_mask).unwrap_or((0.0, 0));
|
||||
if let Err(e) = writeln!(
|
||||
f,
|
||||
"{part},{l},{family_idx},{h15:.6},{h4:.6},{},{n_present}",
|
||||
"{part},{l},{family_idx},{h15:.6},{h4:.6},{},{n_present},{n_unambiguous}",
|
||||
mask.family_size()
|
||||
) {
|
||||
write_err = Some(e);
|
||||
|
||||
Reference in new issue
Block a user