Files
Eric Coissac f29dcbcfb8
Release / create-release (push) Successful in 2m35s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 2m3s
ci.yml / build (pull_request) Successful in 4m8s
Implement advanced sampling and family annex logic
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data.

This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
2026-10-13 08:57:18 +02:00

3.6 KiB

Sampling design

Exhaustive computation over every non-monomorphic family in an index (see Central-position SNP model) is the default, but can be bounded to approximately N families instead. Sampling is designed around two properties: staying representative of the whole index, and letting the user favor families that actually carry signal.

Proportional sampling

Families are drawn in proportion to how many candidate families each part of the index actually holds, rather than uniformly across index partitions — this keeps the sample's composition representative of the whole index regardless of how unevenly candidate families happen to be distributed across partitions.

Per-family informativeness: Shannon entropy

A family's informativeness is measured directly rather than assumed uniform. For a family, entropy is computed over its observed states across genomes:

Quantity Definition
entropy15 Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of {A,C,G,T}), genomes absent from the family excluded from the count
entropy4 the same over the 4 plain bases, restricted to the genomes carrying exactly one member of the family (those that stay informative when multi-member genomes are treated as missing data)

With --no-ambiguity the entropy is entropy4, computed over the genomes carrying exactly one member of the family, so that the weighting sees the same genomes as the final alignment; without it, entropy15 is used. Each is stored in its own annex.

A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).

Quartet restriction

A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. --quartet removes them before the draw. The test counts called states only, so absence never forms one of the two pairs.

Pair balance

How well a pair of genomes is represented in a sample depends on how many families the pair shares: pairs of distant genomes share few, so a uniform draw gives them few sites, and the deep branches of a tree — which rest on exactly those pairs — are the least supported. --pair-balance weights each family by the mean, over the pairs of genomes carrying it, of 1 / (families shared by the pair), computed on the whole index. Averaging rather than summing keeps the weight from growing with the number of carriers, which would favour the families carried by most genomes regardless of which pairs they inform. Layers are allotted their share of the draw in proportion to the total weight of their families.

Entropy-biased sampling

By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy \mu with a Gaussian curve of width \sigma: a family's probability of being kept scales with \exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right) — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.

See phylo for how to bound and bias the sample, and for session/checkpointing behavior.