Files
obikmer/UserDocMD/theory/phylogeny/snp_based/sampling_design.md
T
Eric Coissac 5db7b440da
ci.yml / build (pull_request) Successful in 4m33s
Release / create-release (push) Successful in 1m56s
Release / build-linux-x86_64 (push) Successful in 11m18s
Release / build-macos-arm64 (push) Successful in 2m27s
refactor: treat absence as state unless --free-loss and update quartet annexes
Refactor the handling of genome absence to be a distinct state unless `--free-loss` is specified. This change introduces `QuartetConvention` to pair ambiguity policies with the free-loss flag, updating file naming conventions and logic across sampling, masking, and stats modules.

- Update `--exclude-genome` description in `pair_coverage_sampling.md` to clarify absence as a state unless `--free-loss` is specified.
- Move multi-member genome handling convention from `pair_coverage_sampling.md` to file name conventions.
- Clarify persisted annex structure in `pair_coverage_sampling.md` as `PersistentBitVec` per non-monomorphic family with counting convention carried by file name.
- Fix initial version's absence handling in `has_bipartition`, the annex, and `--sample-stats` to correctly count index 0 when absence is a state.
- Replace entropy-only builder with `AnnexRequest` and `ensure_layer_family_annexes`.
- Consolidate sampling parameters into a single `SamplingSpec`.
- Update family constraint description in `sampling_design.md`.
- Modify uniform draws and `--quartet` removal text in `sampling_design.md`.
- Clarify state counting in tests for `--free-loss` option in `sampling_design.md`.
- State that absence forms a pair without `--free-loss` but never with it in `sampling_design.md`.
- Update `--quartet` option to require at least two distinct states, each carried by at least two genomes.
- Define absence as a state unless `--free-loss` is given for family carrier definition.
- Clarify `--pair-balance` favors families covering pairs with fewest shared families.
- Modify logic for families fitting every tree topology equally well in `phylo.md`.
- Change `--quartet` restriction logic to apply after pre-checks involving `--no-ambiguity` and `--exclude-genome`.
- Update `obikmer` dependency version from 1.4.4 to 1.4.5 in `Cargo.lock` and `Cargo.toml`.
- Modify `annex_selection` in `args/mod.rs` to derive `free_loss`, `entropy`, `quartet`, and `overlap` from `SamplingSpec`.
- Update `SamplingArgs::quartet` documentation in `sampling.rs` to describe state determination based on `--free-loss`.
- Add `free_loss` field to `AnnexSelection` struct in `family_annexes.rs`.
- Change `AnnexRequest` to use `Vec<QuartetConvention>` instead of `Vec<AmbiguityPolicy>`.
- Update `request` logic in `family_annexes.rs` to conditionally create `QuartetConvention` based on `self.quartet`.
- Modify `dedup` signature in `family_annexes.rs` to accept iterator of type `T: Ord`.
- Update `ensure_layer_family_annexes` in `family_annexes.rs` to use `missing_files` and `QuartetConvention`.
- Add `missing_files` definition to `family_annexes.rs`.
- Change `has_bipartition` in `masking.rs` to filter counts where count is `>= 2`.
- Update `is_quartet_informative` in `masking.rs` to accept `free_loss` boolean parameter.
- Modify `two_pairs_of_distinct_states_is_informative` and `singleton_variant_is_not_informative` in `masking.rs` to test both `free_loss` policies.
- Add `under_free_loss_absence_is_missing_and_forms_no_pair` and `without_free_loss_absence_is_a_state_that_can_form_a_pair` definitions to `masking.rs`.
- Update `no_ambiguity_drops_multi_member_genomes_from_the_count` in `masking.rs` to check `is_quartet_informative` under `free_loss=true`.
- Add `the_annex_rule_and_the_exact_check_agree_on_the_same_family` definition to `masking.rs`.
- Remove `absent_genomes_are_not_a_state` definition from `masking.rs`.
- Update `restrict_to_quartet` in `quartet.rs` to accept `QuartetConvention` instead of `AmbiquityPolicy`.
- Modify `add_site` in `sample_stats.rs` to handle state 0 as a state participating in bipartition.
- Add `absence_as_a_state_makes_a_site_informative` definition to `sample_stats.rs`.
- Add `quartet_convention` function to `SamplingSpec` in `subsample.rs`.
- Update `sample_index` in `subsample.rs` to use `spec.quartet_convention()` for `AnnexRequest`.
- Update `restrict_to_quartet` call in `subsample.rs` to use `spec.quartet_convention()`.
- Add comment regarding `--no-ambiguity` counting convention in `helpers.rs`.
- Import `QUARTET_ANNEX_FILE_NAMES` and `QuartetConvention` from `quartet_annex` in `mod.rs`.
- Add `QuartetConvention` struct with `policy` and `free_loss` fields to `quartet_annex.rs`.
- Modify `quartet_annex_file_name` in `quartet_annex.rs` to take `QuartetConvention`.
- Add `the_conventions_never_share_a_file` function to `quartet_annex.rs`.
- Remove `the_policies_never_share_a_file` definition from `quartet_annex.rs`.
2026-11-19 07:12:36 +01:00

3.7 KiB

Sampling design

Exhaustive computation over every non-monomorphic family in an index (see Central-position SNP model) is the default, but can be bounded to approximately N families instead. Sampling is designed around two properties: staying representative of the whole index, and letting the user favor families that actually carry signal.

Proportional sampling

Families are drawn in proportion to how many candidate families each part of the index actually holds, rather than uniformly across index partitions — this keeps the sample's composition representative of the whole index regardless of how unevenly candidate families happen to be distributed across partitions.

Per-family informativeness: Shannon entropy

A family's informativeness is measured directly rather than assumed uniform. For a family, entropy is computed over its observed states across genomes:

Quantity Definition
entropy15 Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of {A,C,G,T}), genomes absent from the family excluded from the count
entropy4 the same over the 4 plain bases, restricted to the genomes carrying exactly one member of the family (those that stay informative when multi-member genomes are treated as missing data)

With --no-ambiguity the entropy is entropy4, computed over the genomes carrying exactly one member of the family, so that the weighting sees the same genomes as the final alignment; without it, entropy15 is used. Each is stored in its own annex.

A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).

Quartet restriction

A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. --quartet removes them before the draw. The test counts the states of the alignment: without --free-loss, absence is a state like any other and can form one of the two pairs; with --free-loss it is missing data and never does.

Pair balance

How well a pair of genomes is represented in a sample depends on how many families the pair shares: pairs of distant genomes share few, so a uniform draw gives them few sites, and the deep branches of a tree — which rest on exactly those pairs — are the least supported. --pair-balance weights each family by the mean, over the pairs of genomes carrying it, of 1 / (families shared by the pair), computed on the whole index. Averaging rather than summing keeps the weight from growing with the number of carriers, which would favour the families carried by most genomes regardless of which pairs they inform. Layers are allotted their share of the draw in proportion to the total weight of their families.

Entropy-biased sampling

By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy \mu with a Gaussian curve of width \sigma: a family's probability of being kept scales with \exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right) — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.

See phylo for how to bound and bias the sample, and for session/checkpointing behavior.