Files
obikmer/UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md
T
Eric Coissac 33d7a59dd9 docs: restructure documentation for kmer indexing and phylogeny theory
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
2026-11-19 17:47:59 +01:00

3.1 KiB

Gamma rate heterogeneity

Real substitution rates vary across sites rather than being uniform, which a plain substitution correction (see Central-position SNP model) ignores. The standard +\Gamma correction models this by replacing each -\ln(x) term in a correction formula with \alpha\left(x^{-1/\alpha}-1\right) — the same x, the same weight, under the assumption that the per-site rate is $\mathrm{Gamma}(\alpha,\alpha)$-distributed (mean 1) rather than fixed. A small \alpha indicates strong among-site rate heterogeneity (many near-invariant sites, a few fast ones); as \alpha grows large the correction converges to the uncorrected formula.

Automatic \alpha estimation

\alpha can be estimated automatically from the index itself, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.

The estimator pools substitution counts by partition rather than by genome pair: for partition i, let n_i be the total number of substitutions observed across every genome pair, and L_i the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:

R_i = \frac{n_i}{L_i}

Under a single shared substitution rate with no among-site heterogeneity, each R_i would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as in the correction itself, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:

\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}

where \mu is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own L_i removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:

\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}} \hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}

If the measured variance V doesn't exceed the Poisson floor \hat\mu/\bar L (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), \alpha is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged.

This is not Jin & Nei's (1990) original estimator. Their publication defines the +\Gamma distance formula itself and, absent an estimate, recommends the fixed default \alpha = 1 — it does not propose a way to estimate \alpha from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is obikmer's own addition, not part of the cited correction.

See phylo for how to select a fixed \alpha or request automatic estimation.