Files
obikmer/UserDocMD/theory/phylogeny/snp_based/sampling_design.md
T
Eric Coissac 33d7a59dd9 docs: restructure documentation for kmer indexing and phylogeny theory
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
2026-11-19 17:47:59 +01:00

3.6 KiB

Sampling design

Exhaustive computation over every non-monomorphic family in an index (see Central-position SNP model) is the default, but can be bounded to approximately N families instead. Sampling is designed around two properties: staying representative of the whole index, and letting the user favor families that actually carry signal.

Proportional sampling

Families are drawn in proportion to how many candidate families each part of the index actually holds, rather than uniformly across index partitions — this keeps the sample's composition representative of the whole index regardless of how unevenly candidate families happen to be distributed across partitions.

Per-family informativeness: Shannon entropy

A family's informativeness is measured directly rather than assumed uniform. For a family, entropy is computed over its observed states across genomes:

Quantity Definition
entropy15 Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of {A,C,G,T}), genomes absent from the family excluded from the count
entropy4 the same over the 4 plain bases, restricted to the genomes carrying exactly one member of the family (those that stay informative when multi-member genomes are treated as missing data)

With --no-ambiguity the entropy is entropy4, computed over the genomes carrying exactly one member of the family, so that the weighting sees the same genomes as the final alignment; without it, entropy15 is used. Each is stored in its own annex.

A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).

Quartet restriction

A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. --quartet removes them before the draw. The test counts the states of the alignment: without --free-loss, absence is a state like any other and can form one of the two pairs; with --free-loss it is missing data and never does.

Pair balance

How well a pair of genomes is represented in a sample depends on how many families the pair shares: pairs of distant genomes share few, so a uniform draw gives them few sites, and the deep branches of a tree — which rest on exactly those pairs — are the least supported. --pair-balance weights each family by the mean, over the pairs of genomes carrying it, of 1 / (families shared by the pair), computed on the whole index. Averaging rather than summing keeps the weight from growing with the number of carriers, which would favour the families carried by most genomes regardless of which pairs they inform. Layers are allotted their share of the draw in proportion to the total weight of their families.

Entropy-biased sampling

By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy \mu with a Gaussian curve of width \sigma: a family's probability of being kept scales with \exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right) — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.

See phylo for how to bound and bias the sample.