Refactor indexing, sampling, and entropy documentation

This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
This commit is contained in:
Eric Coissac committed 2026-10-13 08:54:21 +02:00
1 parent 95bbc0b2ae
commit 0d12494f1a
12 files changed
+1182 -56

No files matched your search

+69
View File
@@ -0,0 +1,69 @@
# Chance sharing of a kmer across genomes (discussion)
Null model for "a kmer is present in $r$ of $S$ genomes": how likely is this by chance alone? Extends the single-pair estimate $P_{\mathrm{chance}}\simeq G/4^k$ (see user doc, *K-mer specificity*).
## Homogeneous case: binomial tail
If every genome has the same per-genome chance probability $p$, the number $R$ of genomes containing a given kmer is $\mathrm{Bin}(S,p)$:
$$P(R\geq r)=\sum_{j=r}^{S}\binom{S}{j}p^j(1-p)^{S-j}=I_p(r,\,S-r+1)$$
where $I_p$ is the regularized incomplete beta function. No elementary closed form; use $I_p$ directly.
### Numerical evaluation of the sum
- Term recurrence, one `lgamma` start: $t_{j+1}=t_j\cdot\frac{S-j}{j+1}\cdot\frac{p}{1-p}$.
- Sum the tail **not containing the mode** ($Sp$, not $S/2$). If $r>Sp$, sum $\sum_{j\ge r}$ directly and stop when terms are negligible. The complement $1-\sum_{j=0}^{r-1}$ (note: from $j=0$) suffers catastrophic cancellation when the p-value is tiny.
- Large $S$: normal approximation with continuity correction, or Poisson($Sp$) when $p$ is small. Unreliable in extreme tails.
## Heterogeneous case: $p_i = f(G_i)$
The $G_i$ (number of kmers of genome $i$) are known and differ. With $p_i\simeq G_i/4^k$, the indicators "kmer present in genome $i$ by chance" are independent Bernoulli($p_i$), so $R$ follows a **Poisson-binomial** law, not a binomial:
$$P(R=j)=\sum_{|A|=j}\prod_{i\in A}p_i\prod_{i\notin A}(1-p_i).$$
Everything lives in canonical space: $G_i$ is the number of distinct canonical kmers of genome $i$, and the kmer space has size
$$N=\frac{4^k}{2}$$
exactly, since $k$ is odd (no kmer equals its own reverse complement, so canonicalization pairs the $4^k$ kmers perfectly). Exact per-genome probability: $p_i=1-(1-1/N)^{G_i}\simeq 2G_i/4^k$. This is twice the $G/4^k$ of the user doc, where $G$ is a number of positions on one strand.
### Exact computation
Dynamic programming over genomes, $d_j$ = probability of $j$ hits so far:
$$d'_j=d_j(1-p_i)+d_{j-1}p_i.$$
For the upper tail only $d_0\ldots d_{r-1}$ are needed: $P(R\ge r)=1-\sum_{j<r}d_j$, cost $O(S\,r)$, per distinct $(p_1..p_S)$ vector — independent of the kmer, so computed once per index (table of $P(R\ge r)$ for $r=1..S$), not per kmer. Cancellation caveat as above: for tiny tails compute $\Pr(R\ge r)$ by the DP on the upper side (track $d_j$ for $j\ge r$ only from a top-down recurrence) or in log space. Alternative: DFT of the characteristic function (Hong 2013), $O(S^2)$ exact.
### Approximations
- **Poisson**: $R\approx\mathrm{Poisson}(\lambda=\sum_i p_i)$ with total variation error $\le\sum_i p_i^2$ (Le Cam). Excellent when all $p_i\ll1$, i.e. $4^k\gg G_i$ (large $k$).
- **Binomial with mean $\bar p=\frac1S\sum_i p_i$**: Hoeffding (1956) shows the Poisson-binomial is more concentrated than $\mathrm{Bin}(S,\bar p)$. Hence for $r\ge S\bar p+1$, $P_{\mathrm{PB}}(R\ge r)\le P_{\mathrm{Bin}}(R\ge r)$: using the mean $\bar G$ overestimates the chance probability, i.e. is **conservative** in the upper tail. Wrong direction for $r\le S\bar p$ (lower tail), irrelevant for significance testing.
Consequence: assuming a common $G$ is safe (conservative) if $G=\bar G$ (arithmetic mean), but loses power when genome sizes span orders of magnitude; the exact DP is cheap enough (once per index) to make the approximation unnecessary. Small genomes with $k=11$ saturate ($p_i\to1$) and break the Poisson approximation, not the DP.
## Open points
- Repeats are not a problem. The model is presence/absence per genome, so copy number within a genome is irrelevant, and $G_i$ is the number of **distinct** kmers of genome $i$. A repeat shared between genomes is paralogy, i.e. homology: a true positive, not a false one. Low-complexity kmers (which would share by composition, not descent) are removed upstream by the [entropy filter](entropy.md).
- Residual violation of the equiprobability behind $p_i$: nucleotide composition bias (see Karlin–Ost in the user doc).
- Test dependence between overlapping kmers (see FDR section).
## Goal: an FDR of homology
$H_0$: the $S$ genomes are unrelated (independent random sequences of sizes $G_i$). Rejecting $H_0$ for a kmer, i.e. sharing that chance cannot explain, is the evidence of homology. The quantity wanted is the false discovery rate of the call "shared kmer $\Rightarrow$ homologous".
### Empirical FDR from the count histogram
Let $N$ be the size of the kmer space and $n_{\ge r}$ the observed number of distinct kmers present in at least $r$ genomes (a by-product of the index). Under $H_0$ the expected number of kmers with $R\ge r$ over the whole space is exactly $N\,P(R\ge r)$, with $P$ the Poisson-binomial tail above. Hence
$$\widehat{\mathrm{FDR}}(r)=\min\!\left(1,\ \frac{N\,P(R\ge r)}{n_{\ge r}}\right).$$
- No per-kmer multiple-testing correction: the number of tests ($N$) is already in the numerator.
- Conservative: the observed $n_{\ge r}$ mixes true homology and chance, and the true-null proportion $\pi_0\le1$ is taken as $1$ (Storey-type upper bound).
- Take the smallest $r$ with $\widehat{\mathrm{FDR}}(r)\le\alpha$ as the sharing threshold.
- Equivalent per-kmer view: BH on $p=P(R\ge r)$ with $N$ tests. The zero-truncation matters if one tests only kmers present in the index ($R\ge1$ by selection): use $P(R\ge r)/P(R\ge1)$, not $P(R\ge r)$, otherwise the test is too conservative by a factor $\sim1/\lambda$.
### Dependence
Kmers of one shared region overlap and are not independent tests. The FDR above holds per kmer; for a per-region FDR, count regions (unitigs / super-kmers, with the appropriate clumping of $N$ and $n_{\ge r}$) rather than kmers. Not settled.
+222
View File
@@ -0,0 +1,222 @@
# Pair-coverage-biased family sampling (discussion, 2026-10-07)
Idea: bias the family sample used by `--pseudo-alignment`/`--sankoff`/`snp-*` so that the `family_overlap` matrix of the *sample* is as uniform as possible, compensating the scarcity of shared families between distant species. Complements, does not replace, the existing proportional-by-partition and entropy-biased draws (see user doc, *Sampling design*).
## Motivation: observed on Primulaceae (180 genomes)
Scatter of patristic distance (IQ-TREE on the pseudo-alignment) vs. normalized `family_overlap` shows three pair classes:
- close intra-genus pairs: high overlap, well resolved;
- inter-genus plus distant intra-genus pairs: overlap collapses by ~2 orders of magnitude, one large noisy cloud;
- a small low-overlap cluster at the largest distances, plausibly a few poorly sequenced genomes (user hypothesis, unverified; the tree shows one tip, `Androsace_saussurei--PHA010690`, with a 2.9 terminal branch).
Uniform family draw reproduces this imbalance in the pseudo-alignment: distant pairs share few informative columns, close pairs many. The tree quality for deep splits is then limited by sampled-column count on those pairs, not by the total number of families.
## What defines a target
A family $f$ is carried by a genome set $S_f$ and contributes to every pair in $S_f \times S_f$. For a sample $\mathcal{F}$ with inclusion probabilities $\pi_f$, the expected sample overlap of pair $(i,j)$ is
$$\mathbb{E}[O^{\mathcal{F}}_{ij}] = \sum_{f:\, i,j\in S_f} \pi_f$$
The goal is a $\pi$ making this approximately constant over pairs. The full-index $O_{ij}$ (already computed by `--family-overlap`) is the pool size per pair; $\min_{ij} O_{ij}$ is a hard ceiling on the flat level reachable.
## Nesting hypothesis (user, unverified)
Families covering the poorest pairs also cover the richest ones: absent genome-skimming sampling bias (locus dropout), a family conserved across distant species is not expected to be missing between closer ones. If it holds, $S_f$ is approximately a clade closure and the family pool is nested by pair distance, which changes the design:
- no sequential/stateful draw needed: a family's weight depends only on its own reach (farthest pair covered, or equivalently its coverage of the poorest pairs), computable in one pass;
- flattening reduces to a monotone reweighting by reach: keep (nearly) all long-reach families, thin the short-reach ones, until the sample overlap of close pairs drops to the level of distant ones.
My earlier remark that the weight is stateful (depends on what other families already covered) holds only without nesting.
## Measuring nesting: `scripts/pseudo_alignment_nesting.py`
Works on a pseudo-alignment FASTA plus the IQ-TREE treefile (columns = families; a call is any character other than `-`/`?`). Outputs the sample's `family_overlap` matrix and, per clade-size stratum, the *completeness* $|S_f|/|C(S_f)|$ where $C(S_f)$ is the smallest clade containing $S_f$. Tree rooted as written in the Newick unless `--outgroup`. Uses the already-existing families × genomes presence structure only; no `S_f` is stored (clade of a column = LCA of its first and last carrier in DFS leaf order, precomputed 180×180 matrix).
### Result, Primulaceae pseudo-alignment (180 genomes, 5.0 M columns, uniform draw, root as written)
| Clade size | Families | Mean completeness | Clade-closed |
|---|---|---|---|
| 2 | 145 047 | 1.00 | 100 % |
| 3-4 | 272 918 | 0.71 | 20 % |
| 5-8 | 503 114 | 0.45 | 0.6 % |
| 9-16 | 637 896 | 0.35 | 0.1 % |
| 17-32 | 1 656 229 | 0.21 | 0 % |
| 33-64 | 299 632 | 0.18 | 0 % |
| 65-128 | 181 126 | 0.08 | 0 % |
| 129-180 | 1 307 452 | 0.12 | 0 % |
Overall 4.1 % clade-closed, mean completeness 0.275; off-diagonal overlap min 4, median 1 665, max 387 057, CV 1.57. **Strict nesting does not hold on this sample.** Caveats: unrooted tree, so the 129-180 stratum is inflated by families spanning the arbitrary root (rerun with `--outgroup` pending); this is the sample fed to IQ-TREE, not the full index; absence symbols `-`/`?` both count as absent.
### Lift test (`--lift-test`): margin-preserving null
Question: do families covering a rare pair $(A,B)$ also cover an abundant pair $(A,C)$ more often than margins imply? For each $A$, $B$ = its 10 % rarest partners, $C$ = its 10 % most abundant (ranked on pair overlap). Null: curveball swap randomisation of the genome × family matrix (exact row and column sums), re-selecting $B$/$C$ in every replicate with the same procedure. 200 754 sampled columns, 20 null replicates, 5 000 trades between replicates.
| Statistic | Observed | Null |
|---|---|---|
| $T=P(C\mid A,B)=O_{ABC}/O_{AB}$ | 0.559 | 0.498 ± 0.003 (no replicate ≥ observed, p at floor 1/21) |
| $L=\sum n_A O_{ABC}/\sum O_{AB}O_{AC}$ | 1.18 | 1.694 ± 0.004 |
The two statistics disagree, not contradictorily: $T$ above null supports the "positive association beyond margins" statement; $L$ below null says that, relative to $P(C\mid A)$, knowing $B$ adds less than under the null — the observed abundant partners of $A$ are far more abundant than null ones (phylogenetic structure inflates the $L$ denominator). Unverified: mixing of the chain (is 5 000 trades enough? the tiny null sd may reflect correlation), sensitivity to the 10 % threshold and to the 200 k-column subsample.
Combined reading: association exists but is patchy; a reach-only weight is not enough, weighting must be per presence pattern $S_f$.
## Candidate weighting strategy: pair raking (proposal, not implemented)
Goal: inclusion probabilities $\pi_f$ with $\sum_{f\ni i,j}\pi_f\approx c$ for all pairs.
1. First-order weight: $w_f=\sum_{i,j\in S_f}1/O_{ij}$ (draw a pair uniformly, then a family uniformly among those covering it). With the presence matrix $P$ and $R=1/O$, $w=\tfrac12\,\mathrm{rowsum}(P\odot RP)$, one pass in the existing chunked loop, no $S_f$ storage.
2. Iterate (raking/IPF with one multiplier per pair): update $R_{ij}$ from the overlap the current $\pi$ yields, recompute $w_f$; corrects the fact that large families are drawn for many pairs at once. $\pi_f=\min(1,c\,w_f)$, $c$ set to hit the target sample size; independent Poisson draw per family.
3. Control: recompute the sample's `family_overlap` and its CV with the script above.
Pre-conditions: exclude anecdotal genomes first (e.g. `Androsace_saussurei--PHA010690`, terminal branch 2.9), else their tiny $O_{ij}$ dominate $R$. Overlap uniformity is only a proxy: information for a deep split is informative sites across it, not mere co-presence; to be checked on node support of the resulting tree. Unknown: whether long-reach families are rare because conserved-single-copy or because repeats/misassemblies (the latter would be amplified by up-weighting).
## Alternative / complement: site classes with their own models
Analogy with multigene supermatrices: loci differ in rate *and* in taxon occupancy. Families covering rare pairs (long reach, conserved, slow) vs. families covering close pairs only could be two (or a few) site classes, each with its own substitution model / branch-length scale (IQ-TREE partitions; edge-proportional vs edge-linked options to be checked in the manual). Independent of the weighting: sampling fixes scarcity, partitioning fixes rate heterogeneity; with class-specific scales, oversampling a class would not distort that class's branch lengths, removing the need for $1/\pi_f$ reweighting (IQ-TREE support for per-site weights unverified).
Caveats: reach is a continuum, two classes is a simplification and there is no criterion for the cut; class membership is defined from presence patterns, hence correlated with the data the model sees (assignment bias, worst with `?` missing data; magnitude not assessed); partitioning does not fix repeats/misassembly, it only gives them their own rate.
The pseudo-alignment already keeps only variable families, so branch lengths are not substitutions/site anyway (ascertainment bias, handled by `+ASC`, see the constant-column fix section in [evolutionary_distances.md](evolutionary_distances.md)). With partitions the ascertainment correction must be per class: each class excludes a different number of constant sites. Whether the existing fix applies per partition is unchecked.
Literature: see the next section.
## Literature on informative-site selection (2026-10-07; read from search extracts, not full papers)
- **Phylogenetic informativeness.** Townsend 2007 (*Syst. Biol.* 56:222-231) links each site's rate of change to its power to resolve splits at a given depth: invariant sites are useless, saturated sites misleading, with an optimum in between that depends on the age of the split. Implemented online as PhyDesign (requires an ultrametric tree). Relevance: supports an entropy centre $\mu$ that depends on depth (pair distance) rather than a single value; cf. the $\mu(n)$ discussion.
- **Entropy-based trimming.** BMGE (Criscuolo & Gribaldo 2010, *BMC Evol. Biol.* 10:210) computes a per-character entropy-like score weighted by a similarity matrix and removes regions above a threshold. Uses entropy to *discard* over-variable characters, the opposite direction of the Gaussian band here.
- **Saturation / fast sites.** Classical removal methods: Slow-Fast, LQP, TIGER, Observed Variability. Counter-evidence (Tree-of-Life protein datasets, PMC10609509): even the fastest sites carry reliable signal, trimming them can lower accuracy, slow sites recover fewer true bipartitions than the fastest. Argument against penalizing high entropy ($H\ge2$) without checking that those families do not carry the deep signal; domain (conserved proteins) differs from central-position SNPs, transfer unverified.
- **Taxon-coverage theory.** Sanderson, McMahon & Steel 2010 (*BMC Evol. Biol.*, "Phylogenomics with incomplete taxon coverage: the limits to inference") characterise phylogenetically decisive collections of taxon sets via the four-way partition property: when the coverage pattern suffices to determine the tree uniquely irrespective of signal. Formal frame for pair-coverage sampling; applicability at 180 genomes not assessed.
Takeaways to check before weighting: (i) whether high-entropy families are the deep-signal ones (entropy vs carrier count measured, see below; the cross with reach is still open); (ii) depth-dependent entropy centre; (iii) whether a decisiveness criterion can serve as a sufficiency test for the sample's pair coverage.
## Simpler weight: poorest pair of the family (user preference, 2026-10-07)
$w_f=1/\min_{i,j\in S_f}O_{ij}$, $\pi_f=\min(1,c\,w_f)$. Equivalent in spirit to a reach weight but measured on actual overlaps (no tree needed), same $O(|S_f|^2)$ cost per family as the sum form. Limit: it ignores that large families also feed already-abundant pairs; flattening from above then relies only on dropping short-reach families, which is exact only under nesting, and nesting holds partially (see measurements). Untested.
### Estimating $O$ locally (per partition / per layer)
- **Per layer: not representative.** A layer is an increment of kmer content added to a partition (typically one `merge`), not a batch of genomes; all genome columns exist in every layer (`UserDocMD/formats/index_layout.md`). Layer content is dominated by kmers novel at merge time, so a layer's $O_{ij}$ reflects merge order, not the index-wide distribution; not an unbiased estimator of the global $O_{ij}$.
- **Per partition: valid only if partitions are exchangeable samples of families** (hash-based; not verified, minimizer distribution may be skewed). Even then, the pairs that matter are the rare ones, with near-zero count per partition; a `min` over noisy estimates is biased downward (winner's curse), weights would go to pairs that were unlucky in that partition rather than globally poorest.
- **Recommended:** compute $O$ globally once (the `--family-overlap` scan already exists), apply the weight locally per partition, keeping partitions independent. Without an extra pass: pilot estimate on a random subset of partitions.
- **Sequential variant:** update the $O$ estimate as partitions are processed in random order, boosting still-undersampled pairs. Self-correcting but breaks partition independence and makes the result order-dependent.
- **Quota consequence:** the current per-partition quota is proportional to candidate-family count; with weights it must be proportional to the partition's weight mass $\sum_f w_f$, otherwise partitions rich in high-weight families are undersampled.
## Does the entropy bias distort pair representation? (measured 2026-10-07)
Sample drawn with `--entropy 1 --entropy-sd 0.5` (μ=1 explicit, σ default 0.5): family weight $\exp(-2(H-1)^2)$, $H$ = `entropy15` in bits; weight 1 at $H=1$, 0.135 at $H=0$ or $2$, $\approx 3\!\times\!10^{-4}$ at $H=3$.
Comparison, per genome pair, of the full-index `family_overlap` $O^{\text{full}}$ (index scan) to the sample's overlap $O^{\text{samp}}$ (5.0 M columns), sampling fraction $r_{ij}=O^{\text{samp}}_{ij}/O^{\text{full}}_{ij}$ (uniform draw would give constant $r$):
| Quantity | Value |
|---|---|
| Spearman $O^{\text{full}}$ vs $O^{\text{samp}}$ | 0.994 (Pearson on logs 0.998): ranking preserved |
| global fraction $\sum O^{\text{samp}}/\sum O^{\text{full}}$ | 0.52 % |
| $r$ quantiles 5 / 50 / 95 % | 0.0025 / 0.0037 / 0.0055 |
| slope $\log r$ vs $\log O^{\text{full}}$ | +0.12 |
| Spearman($r$, patristic distance) | −0.71 |
Per decile of $O^{\text{full}}$, median $r$ rises from 0.0027 (poorest decile) to 0.0053 (three richest, plateau): the poorest pairs are represented about half as much as the richest, so the draw amplifies the imbalance. $r$ climbs slowly over deciles 1-6 (0.0027 to 0.0037), jumps at deciles 7-8 (0.0045-0.0052), where the $O^{\text{full}}$ distribution has a gap (689 k to 11.6 M, presumably close intra-clade pairs, unchecked).
Not established: (a) causality — a dependence of $r$ on pair richness is compatible with the entropy filter but no uniform control draw of the same size exists; (b) noise — the poorest sample pairs have tiny counts (min 4).
**Candidate mechanism: first hypothesis refuted, see the next section.** I first suggested that long-reach families, carried by many divergent genomes, spread over several states ($H\approx2\text{-}3$) and are downweighted from above by the Gaussian. The full entropy table (next section) shows the opposite: families with many carriers are mostly *low*-entropy, and are downweighted from below.
**Consequence for the pair-weight proposal:** entropy and reach are likely correlated, not independent criteria. Since the entropy weight of each family is known, the pair weight can compensate it exactly (divide by it) rather than compose with it; order of the two filters must then be defined.
## Entropy vs number of carrier genomes, full index (measured 2026-10-09)
Source: `phylo --shannon`, which is **always a full, unsampled scan** of every non-monomorphic minorant of every layer (`SiblingExt::shannon_entropy_csv(path)` takes no `subsample`/`entropy_bias` argument; `--subsample`/`--entropy` apply only to `--pseudo-alignment`/`--sankoff`/`snp-*`). Here: 19.7 GB CSV, 535 257 857 families, read in 11 s with DuckDB (no in-memory load). `UserDocMD/usage/phylo.md` line 121 ("`--subsample N --shannon` gives a bounded diagnostic sample") is stale and contradicts line 55 and the code; to fix.
| Carriers $n$ | Families | mean $H$ | median $H$ | $P(H<0.5)$ | $P(H\ge1.5)$ | mean weight ($\mu=1,\sigma=0.5$) |
|---|---|---|---|---|---|---|
| 2 | 163.3 M | 0.66 | 1.00 | 34.5 % | 0 % | 0.70 |
| 3-4 | 104.6 M | 0.94 | 0.92 | 0.5 % | 7.9 % | 0.93 |
| 5-8 | 91.8 M | 0.83 | 0.72 | 0.1 % | 4.6 % | 0.80 |
| 9-16 | 62.7 M | 0.70 | 0.50 | 43.6 % | 4.9 % | 0.66 |
| 17-32 | 45.4 M | 0.59 | 0.35 | 58.9 % | 5.9 % | 0.53 |
| 33-64 | 48.1 M | 0.34 | 0.19 | 79.7 % | 1.1 % | 0.40 |
| 65-128 | 19.1 M | 0.37 | 0.18 | 77.4 % | 3.8 % | 0.37 |
| 129-180 | 0.19 M | 0.83 | 0.47 | 53.0 % | 21.4 % | 0.46 |
Findings:
- Families with many carriers are mostly one dominant state plus rare variants ($H<0.5$ for ~77-80 % at $n\ge33$); only 1-4 % have $H\ge1.5$. The Gaussian at $\mu=1$ therefore penalizes them from the **low** side (weight 0.135 at $H=0$), not the high side.
- Relative to a uniform draw, expected inclusion is about ×1.32 at 3-4 carriers and ×0.53-0.56 at 33-128 carriers: a factor ~2.4 between small and large $n$. Same order as the ~2× pair-level undersampling of poor pairs measured above; compatible, but a family-level ratio is not a pair-level one, so causality remains unproven (no uniform control draw).
- The filter does what it is designed for (drop near-autapomorphic variants), but those low-entropy high-$n$ families are also the ones covering distant pairs, hence the side effect.
- Informative high-$n$ families are scarce: at $n>32$ only ~19 % have $H\in[0.5,1.5)$, and the 129-180 class is 0.04 % of all families. Flattening deep-pair coverage works on little informative material whatever the weights.
- Consequence for a $\mu(n)$ that grows with $n$: it would not recover these families (80 % sit below 0.5 bit); what matters at large $n$ is the share of families in the band around 1 bit, not the position of the centre. Conditional weight $w_H(f)/E[w_H\mid n_f]$ remains an option to make the entropy filter neutral to $n$; $E[w_H\mid n]$ is now available from this table.
Limit: the table has `n_genomes_present` but not the reach of the family, so the link to specific pairs is still not measured.
## Site concordance factors on the Primulaceae tree (measured 2026-10-12)
Command (IQ-TREE 3.1.3): `iqtree3 -t <treefile> -s <alignment> -m GTR+G4+ASC --scfl 100 --cf-verbose -T 5 --prefix concord`. With `-t` plus `-m`, IQ-TREE did **not** just annotate the reference tree: it ran a full ML search started from it (102 iterations, ~59 h wall-clock). `concord.treefile` differs from the input treefile by one bipartition (Robinson-Foulds 2), so sCF refer to the re-inferred tree. Without `-m`, ModelFinder is launched and required 157 GB (first attempt failed). The log also warns that all 180 sequences contain more than 50 % gaps/ambiguity. `sN` below = informative sites averaged over the 100 sampled quartets of a branch (alignment: 5 003 414 columns).
**Deep Primula/Androsace interface: too few informative sites (data scarcity).**
| Node | Tips (Primula / Androsace) | sCF | sDF1 | sDF2 | `sN` |
|---|---|---|---|---|---|
| 209 | 67 (43 / 24) | 27 | 47 | 26 | 17 |
| 210 | 66 (43 / 23) | 37 | 33 | 30 | 43 |
| 208 | 74 (50 / 24) | 54 | 16 | 29 | 190 |
| 211 | 24 (2 / 22) | 64 | 18 | 18 | 226 |
| 212 | 21 (2 / 19) | 30 | 38 | 32 | 184 |
| 216 | 10 (2 / 8) | 21 | 47 | 32 | 7 |
| 217 | 9 (1 / 8) | 43 | 20 | 36 | 7 |
Quartets crossing these nodes have tens to a few hundred informative sites out of 5 M columns, the quantitative counterpart of the pair-overlap scarcity. Primula tips are spread over several nodes mixed with Androsace (e.g. 43 + 24 at node 209); no monophyletic Primula in this tree.
**Within Androsace: conflict, not scarcity.** Nodes 283 (72 Androsace, `sN` 17 127), 292 (49 tips, 10 335), 305 (22, 11 980), 342 (11, 6 566) have thousands of informative sites but sCF 33-37 % with both alternatives nearly as supported: sites split almost evenly among the three quartet topologies. More sites would not fix this; cause not established (a poor fit of a tree model to intra-species divergence is one hypothesis, untested). Node 185 (23 Androsace): sDF1 53.6 % > sCF 30.6 %, a majority of the 893 sites prefer an alternative (real conflict).
**Well-supported nodes have long branches:** node 296 (45 tips, length 0.23, sCF 90 %), 341 (0.21, 91 %), 330 (0.09, 85.5 %), 181 (0.099, 89.5 %); most branches of length 0.01-0.05 have sCF 33-45 %. Degenerate: node 186 (length 0.001, `sN` 0.37) and 187 (`sN` 2.2) carry no informative site.
Median sCF shows no gradient with clade size (about 39-48 % in every class), so the clade-size class hides the effect; the scarcity is specific to the interface nodes above.
**Implication for sampling.** The part a sampling design can address is the interface scarcity: select families carried by genomes from the four groups around each target node (a quartet-targeted selection) rather than flattening the whole pair matrix. Not assessed: how many such families the index holds (the `sN` of nodes 209 and 216 suggests few). For the intra-Androsace conflict no sampling strategy is suggested by these measurements alone.
## Quartet coverage (measured 2026-10-12, `--quartet-test`)
`scripts/pseudo_alignment_nesting.py --quartet-test --quartets N` draws random genome quartets and counts exactly the columns where all four carry a call, stratified by the size of the smallest tree clade containing the four. Cost: 20 000 quartets over 5.0 M columns in about 35 s total. Exact per-family quartet weighting is out of reach ($\binom{180}{4}\approx 42.3$ M quartets; one family carried by 100 genomes covers $\binom{100}{4}\approx 3.9$ M), so quartets serve as a *diagnostic* of a sample, not as a weight.
Result on the current sample: 4-way count quantiles 0 / 1 / 10 / 50 / 90 / 99 / 100 % = 0, 3, 68, 200, 2 329, 66 504, 194 460 (CV 3.5, 0.1 % uncovered). By depth: median 131-209 for quartets inside clades ≥ 65 genomes versus 11 580-57 387 for clades 17-64. The ratio of the 4-way count to the smallest of the six pair overlaps has median 0.20-0.29 for deep quartets (0.53 for clades 33-64; 1 would be perfect nesting): pair overlap is only a loose upper bound of 4-way coverage, so flattening pairs does not by itself flatten quartets. 88 % of random quartets fall in the deepest class (root as written; strata with < 5 quartets are noise). Co-presence is not informativeness (a column must also differ among the four). Next measurement: repeat on a sample drawn with the pair weight and check whether the ratio moves toward 1.
## Two kinds of family loss: sequencing dropout vs divergence (2026-10-12)
- **Sequencing dropout (genome skimming, ~1× coverage).** About $e^{-1}\approx 37\%$ of each genome is uncovered, independently per family. Predicted for a family truly present in a clade of $m$ genomes: completeness ≈ 0.63-0.70 and clade-closed fraction $c^m$, independent of clade depth. Consistent with small clades (3-4 tips: completeness 0.71, 20 % closed vs $0.65^3=27\%$, $0.65^4=18\%$). Does not predict the observed decay at larger clades (0.45 at 5-8 tips, 0.21 at 17-32, 0.08-0.12 beyond).
- **Loss by divergence (user observation).** A family is a kmer whose 30 flanking positions (15 each side for $k=31$) are identical, with a variable central base; with per-site divergence $d$ the flanks survive with probability ≈ $(1-d)^{30}\approx e^{-30d}$ (0.74, 0.21, 0.042, 0.0012 for $d$ = 0.01, 0.05, 0.10, 0.20). Phylogeny-dependent and tied to local conservation, unlike sequencing dropout. Explains the loss of completeness with clade size, and sets a floor: no weighting can sample families that no longer exist for pairs whose $e^{-30d}$ is negligible. Surviving deep families are selected for conserved flanks, hence plausibly slow-evolving (fewer variable sites for deep splits; hypothesis, untested).
- The patristic distances of the IQ-TREE tree (1 to 3.5) come from an alignment with variable sites only, hence strongly inflated; the true $d$ is unknown and these numbers cannot be plugged into $e^{-30d}$.
- **Test (user's plan), done 2026-10-12: see the next section.**
## Overlap vs Mash distance on the full index (measured 2026-10-12)
Inputs: `phylo --distance mash --csv --shared-kmers --nj` on the 180-genome index (distance matrix, shared-kmer matrix, NJ tree) against the full-index `family_overlap` $O_{ij}$. Mash $D$: range 0.011-0.337, median 0.20.
| Regression | Slope | $R^2$ |
|---|---|---|
| $\log O_{ij}$ vs $D$, all pairs | -26.95 | 0.979 |
| $\log$(shared kmers) vs $D$ | -31.87 | 0.984 |
| $\log O_{ij}$ vs $D$ + additive genome effect | -25.3 | 0.9999 |
Spearman($D$, $O$) = -0.95; Spearman($D$, shared kmers) = -0.95; Spearman($D$, IQ-TREE patristic) = 0.81.
Reading:
- Overlap decays log-linearly with distance, slope about -27 versus -30 predicted by flank survival (previous section). **Not an independent confirmation of the flank model**: Mash $D$ is derived from kmer sharing and $O$ counts families defined by kmers, so both measure the same shared content. The kmer slope of -32 is $-k$ by construction of Mash (implementation check only). The ratio of the two slopes, 27/32 ≈ 0.85, is below the 30/31 ≈ 0.97 expected from flank survival alone; cause unknown, the gap is modest.
- Local slopes vary: -21 for $D\in[0.011,0.07]$ ($R^2$ 0.45), -28 for [0.07, 0.20] (0.96), -35 for $D>0.23$ (0.83). Half the pairs lie in $D\in[0.20,0.23]$, where the fit explains almost nothing ($R^2$ 0.26 and 0.07): a plateau where Mash has little resolution.
- Genome effects (log scale, sd 0.21): lowest `Androsace_saussurei--PHA010690` (factor ≈ 0.39), `Androsace_triflora--PHA000558` (≈ 0.62), `Androsace_vesulensis--PHA010698` and `Primula_stricta_var_obesior--TROM_V_149128` (≈ 0.65); highest `Androsace_pubescens--PHA010697` (≈ 2.8). Three of the four lowest are also among the genomes with the fewest calls in the pseudo-alignment (saussurei, triflora, stricta), pubescens has the most; consistent with a coverage effect at the extremes, correlation over all genomes not computed.
- NJ on Mash vs the IQ-TREE tree: Robinson-Foulds 182 of 354 (0.51 normalized); the Mash plateau near 0.2 probably limits NJ resolution for most pairs.
- Consequence for normalization: the residual $\log O_{ij}-(a+bD_{ij})$ measures coverage deviation at a given distance, a normalization of $O$ that needs no per-genome call counts. Untested on a weighted sample.
## Data-poor genomes (measured 2026-10-12)
Per-genome share of the 5 003 414 sampled columns with a call: median 5.2 %, max 14.3 %; 8 genomes below 1 % and 23 below 2 %. Fewest calls: `Androsace_triflora--PHA000558` 1 936, `Primula_stricta_var_obesior--TROM_V_149128` 2 403, `Primula_carniolica--PHA007185` 4 937, `Androsace_saussurei--PHA010690` 6 986, `Primula_pumila--TROM_V_135924` 17 393, `Androsace_maxima--PHA000534` 17 939, `Primula_auricula_subsp_serratifolia--PHA007184` 30 621, `Androsace_pyrenaica--PHA000543` 37 725. Three of them (saussurei, serratifolia, pyrenaica PHA000543) are among the tips whose placement looked aberrant, previously attributed to possible mislabelling or contamination. A simpler alternative is too little data to place them reliably; testable by removing them (`--exclude-genome`) and watching deep-node support.
Consequence for flattening: raw $O_{ij}$ mixes evolutionary distance and per-genome coverage; flattening on it chases pairs involving data-poor genomes, for which no family exists. Normalizing, $O_{ij}/(c_ic_j)$ with $c_i$ the number of families carried by genome $i$ (per-genome counts from `--sibling-stats`), targets distance instead of coverage; the best normalization (smoothing of $c_i$, etc.) is not established.
## Open questions
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
- Combination with entropy bias: not independent criteria if entropy correlates with reach (see section above); options: compensate the known entropy weight inside the pair weight, or apply the Gaussian inside reach strata.
- Pairs with near-zero $O_{ij}$ (suspected badly sequenced genomes) set the ceiling for everyone; `--min-shared-family`/`--exclude-genome` should be applied first so the floor is not dictated by anecdotal genomes.
- Effect on the SNP-distance estimators: a non-uniform draw over pairs biases per-pair eligible-family counts, but each pair's $p$-distance is a ratio over its own eligible families, so unbiasedness per pair is expected to hold. To confirm before relying on it.
@@ -0,0 +1,60 @@
# Sibling partition unification and masked canonical form (discussion)
Status: **open, nothing implemented**. Records the state of the reflection on (a) routing the 4 members of a sibling family to one partition and (b) a canonical form that makes siblings directly comparable. Measurements are throwaway (not in the repo); figures are for $k=31$, $m=11$, 256 partitions, 20 bacterial genomes (98.4 M kmer positions), unless stated.
## Problem
A sibling family = the 4 kmers differing only at the central base (position $c=(k-1)/2$, $k$ odd). Routing is **not** a hash of all bases: a kmer goes to `minimizer().seq_hash() & mask`, and consecutive kmers of a superkmer share their minimizer. A family splits across partitions whenever the central base falls in (or changes) the selected m-mer.
Fraction of families whose 4 variants are not all in one partition, standard minimizer: $4m/(k+2m+1)$ (fixed candidates $k-2m+1$ versus $4m$ variable ones). Measured: 81.4 % at $m=11$ (60.2 % at $m=7$, 46.8 % at $m=5$). This is the *theoretical* rate assuming all 4 members present; for two present members it is ≈ 68 %. Real families are overwhelmingly of size 1, so it only becomes cost where something probes variants.
## Masked canonical form (Can3)
Notation: `CentreA(K)` = K with its central base set to A; `RC(K)` its reverse complement.
- Can1 = `min(K, RC(K))` (current). Each sibling canonicalised alone, so some may be flipped relative to the others.
- Can2 = orient by `CentreA(K)` vs `RC(CentreA(K))`. **Not a valid canonical form**: when the flanks are reverse complements of each other ($L=\mathrm{rc}(R)$, frequency ≈ $4^{-c}$, 1/1024 at $k=11$), $K$ and $\mathrm{RC}(K)$ get different canonicals (k=5: `ACAGT` and `ACTGT` both kept). Cause: `CentreA(RC(K)) ≠ RC(CentreA(K))` (the latter has centre T).
- **Can3** (retained):
```
Can3(K) = K if CentreA(K) < CentreA(RC(K))
= RC(K) if CentreA(K) > CentreA(RC(K))
= Can1(K) otherwise // self-revcomp family
```
Orientation is decided on the flanks only (centre ignored), tie broken on the full kmer. Gain: the 4 canonical siblings have identical flanks in the same orientation, so they compare directly with no reverse complement. Self-revcomp families have only 2 distinct canonical members (centres A/C; T/G fold onto them). The A↔T (C↔G) substitution in that context is undetectable by any double-strand canonical set, same as under Can1 — not a loss.
Gain is simplification only: `central_canonical_neighbors` already canonicalises each variant independently and is correct (see `evolutionary_distances.md`, per-partition sweep). The earlier "masked-flank grouping" there is the Can2 idea; Can3 is its repaired form. **No performance gain by itself**, and it breaks existing indexes (canonical orientation changes for some kmers → MPHF hashes change; rebuild).
Prerequisite if adopted: `Minimizer = CanonicalKmerOf<MLen>` shares the canonical type with index kmers; an m-mer has no sibling family and must keep Can1. Decouple with a distinct `Minimizer` type (6 files outside `kmer.rs`: `routable.rs`, `packed_seq.rs`, `obiskbuilder/{rolling_stat,stream_iter,iter}.rs`, `obifastwrite`).
## Routing that excludes the centre (scheme B) — rejected
Candidates restricted to m-mers not overlapping the centre ($m\le c$): $S(j)=\min$ hash over m-mers $p\in[j,j+c-m]$, minimizer$(i)=\min(S(i),S(i+c+1))$. Family- and strand-invariant by construction (checked). But unitigs are partition-local (`pipeline.md`), and the hole moves with each kmer, so runs shrink:
| $m$ | mean run, current | mean run, B | bases/kmer, current → B |
|---|---|---|---|
| 5 | 14.9 | 6.3 | 3.02 → 5.76 |
| 11 | 11.0 | 3.0 | 3.72 → 10.96 |
| 15 | 9.0 | 1.0 | 4.32 → 30.9 |
Proxy: run = unitig (upper bound; real unitigs may merge). Independent of partition count. In `benchmark/run/global_index_count` (786 MB) `unitigs.bin` + idx ≈ 241 MB (31 %), so tripling them ≈ +60 % total index. Hashing the masked kmer per kmer (no minimizer) would cut every path at every kmer: worse.
Structural tension: path coherence (long unitigs) needs a key shared by consecutive kmers, family coherence needs a key independent of each kmer's own centre, which is an ordinary base for its neighbour. Decoupling sequence storage from MPHF partitioning fails because a partition can only enumerate its kmers from its own unitigs (MPHF not invertible). Redefining the family on a terminal base would cut the split rate to ≈ 17 % but is not closed under revcomp and changes the SNP semantics. Lower $m$ reduces the split rate for free but deviates from the literature $k/m\approx 2.5$ optimum (balance, bucket size), unmeasured here — $m=11$ kept.
## What the cross-partition cost actually is
- Phylo-time: only families of size > 1 trigger remote lookups (a small share).
- Annex construction probes the 3 variants of **every** kmer (size-1 families included), ≈ 2/3 of probes crossing partitions; batch bounded to `#kmers(p)*3` per partition (not the whole index).
- Real remaining benefit of unification: the outer sweep over source partitions could run in parallel (it is sequential today for memory and for the `q >= p` dedup rule); plus code simplification in `family_scan.rs` / `annex.rs`. Not quantified.
- Unification would also allow an MPHF keyed on the family (one lookup, 4-bit member mask, union/intersection as OR/AND); needs the family-size histogram (1–4 members) to judge storage.
## Open measurements (index not on this machine)
1. Family-size histogram of the real index: `obikmer phylo --sibling-hist` (needs a built annex).
2. Time and peak RSS of `--sibling-annex` on the Vaccinium/Salix index (`Stage` "sibling_annex" traces time; peak RSS from outside).
3. Only if those show the sequential sweep or the probe volume is a real bottleneck: real-index test of scheme B or of a smaller $m$.
## Decision state
No code change decided. Can3 + `Minimizer` decoupling is the only self-contained step and is independent of the measurements above; it forces an index rebuild, so it is worth doing only if its simplification is wanted.