Implement advanced sampling and family annex logic
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data. This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
This commit is contained in:
1 parent
0d12494f1a
commit
f29dcbcfb8
25 files changed
+1471
-486
No files matched your search
@@ -13,12 +13,22 @@ A family's informativeness is measured directly rather than assumed uniform. For
|
||||
| Quantity | Definition |
|
||||
|---|---|
|
||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||
| `entropy4` | the same, reduced to the 4 plain bases |
|
||||
| `entropy4` | the same over the 4 plain bases, restricted to the genomes carrying exactly one member of the family (those that stay informative when multi-member genomes are treated as missing data) |
|
||||
|
||||
With `--no-ambiguity` the entropy is `entropy4`, computed over the genomes carrying exactly one member of the family, so that the weighting sees the same genomes as the final alignment; without it, `entropy15` is used. Each is stored in its own annex.
|
||||
|
||||
A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).
|
||||
|
||||
## Quartet restriction
|
||||
|
||||
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts called states only, so absence never forms one of the two pairs.
|
||||
|
||||
## Pair balance
|
||||
|
||||
How well a pair of genomes is represented in a sample depends on how many families the pair shares: pairs of distant genomes share few, so a uniform draw gives them few sites, and the deep branches of a tree — which rest on exactly those pairs — are the least supported. `--pair-balance` weights each family by the mean, over the pairs of genomes carrying it, of 1 / (families shared by the pair), computed on the whole index. Averaging rather than summing keeps the weight from growing with the number of carriers, which would favour the families carried by most genomes regardless of which pairs they inform. Layers are allotted their share of the draw in proportion to the total weight of their families.
|
||||
|
||||
## Entropy-biased sampling
|
||||
|
||||
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---subsample---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
||||
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
||||
Reference in new issue
Block a user