Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models. - Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic. - Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation. - Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers. - Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details. - Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics. - Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture. - Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model. - Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections. - Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation. - Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design. - Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure. - Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations. - Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`. - Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013). - Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation. - Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance. - Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990). - Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section. - Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`. - Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
3.1 KiB
Gamma rate heterogeneity
Real substitution rates vary across sites rather than being uniform, which a plain substitution correction (see Central-position SNP model) ignores. The standard +\Gamma correction models this by replacing each -\ln(x) term in a correction formula with \alpha\left(x^{-1/\alpha}-1\right) — the same x, the same weight, under the assumption that the per-site rate is $\mathrm{Gamma}(\alpha,\alpha)$-distributed (mean 1) rather than fixed. A small \alpha indicates strong among-site rate heterogeneity (many near-invariant sites, a few fast ones); as \alpha grows large the correction converges to the uncorrected formula.
Automatic \alpha estimation
\alpha can be estimated automatically from the index itself, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.
The estimator pools substitution counts by partition rather than by genome pair: for partition i, let n_i be the total number of substitutions observed across every genome pair, and L_i the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:
R_i = \frac{n_i}{L_i}
Under a single shared substitution rate with no among-site heterogeneity, each R_i would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as in the correction itself, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:
\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}
where \mu is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own L_i removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:
\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}}
\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}
If the measured variance V doesn't exceed the Poisson floor \hat\mu/\bar L (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), \alpha is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged.
This is not Jin & Nei's (1990) original estimator. Their publication defines the +\Gamma distance formula itself and, absent an estimate, recommends the fixed default \alpha = 1 — it does not propose a way to estimate \alpha from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is obikmer's own addition, not part of the cited correction.
See phylo for how to select a fixed \alpha or request automatic estimation.