diff --git a/DevDocMD/theory/pair_coverage_sampling.md b/DevDocMD/theory/pair_coverage_sampling.md index 2021518b..3ebddffe 100644 --- a/DevDocMD/theory/pair_coverage_sampling.md +++ b/DevDocMD/theory/pair_coverage_sampling.md @@ -1,4 +1,4 @@ -# Pair-coverage-biased family sampling (discussion, 2026-10-07) +&@")--# Pair-coverage-biased family sampling (discussion, 2026-10-07) Idea: bias the family sample used by `--pseudo-alignment`/`--sankoff`/`snp-*` so that the `family_overlap` matrix of the *sample* is as uniform as possible, compensating the scarcity of shared families between distant species. Complements, does not replace, the existing proportional-by-partition and entropy-biased draws (see user doc, *Sampling design*). @@ -278,6 +278,19 @@ The weight flattens the representation (slope toward 0) and, above all, covers m Reading: `--quartet` is the decisive option here — a uniform draw of 100-300 sites contains almost no informative column (0.5-2), the restricted draw contains only informative ones. `--pair-balance` alone barely helps (it favours rare pairs but does not remove the uninformative columns); added to `--quartet` it raises the share of replicates recovering all three species (50→53 % and 33→67 %). The entropy kernel (centre 1 bit, default width) adds nothing at 300 sites and costs at 100. Limits: 11 genomes, three simulated 1x datasets, small budgets, a single criterion (species monophyly, not deep branches), and the differences between the `--quartet` variants (a few runs out of 30) are small relative to the replicate noise — a direction, not a ranking. +## Implemented: no distance matrix unless one is wanted (2026-10-13) + +`--distance` had a default (`jaccard`) and `phylo` always computed and wrote a matrix, even for `--sibling-annex`, `--family-overlap` or `--pseudo-alignment` alone. `--distance` is now optional: a matrix is computed only if `--distance` is given, or `--nj`/`--upgma`/`--shared-kmers` needs one (then `jaccard`, `args::distance_to_compute`, unit-tested); `--csv` alone changes nothing. When nothing is wanted the command stops before the distance step. Checked on the 1x testbed: `--family-overlap` and `--pseudo-alignment` runs write no `_dist` file and run no distance stage; `--nj` writes `_dist.phy` and `_nj.nwk`; `--distance mash --csv` writes `_dist.csv`. Cost saved on a large index: not measured. + +## Implemented: diagnostic outputs (2026-10-13) + +Added so the measurements this file reports no longer need ad hoc scripts, and so a draw can be diagnosed from the files of the run that produced it: + +- `--sample-stats`: a `SampleStatsReducer` (`algorithms/sample_stats.rs`), a `LayerReducer` run beside the alignment over the same surviving families; writes `_sample_genomes.csv`, `_sample_carriers.csv`, `_sample_overlap.csv`. A call is a masked state other than `0`; informative = `has_bipartition` (the quartet rule, now shared). Checked on the 1x testbed: per-genome calls and the overlap matrix equal those recomputed from the alignment. +- `--sample-plan`: `sampled_scan` now returns a `SamplePlan` (one `LayerReport` per layer: eligible, mass, quota, resolved, kept); `_sample_plan.csv` has the layer lines only (quotas sum to the target by cumulative rounding; on the testbed 3000 asked, 2695 kept). +- `--shannon-summary`: `--shannon` and the summary share one traversal (`scan_family_entropies`, which replaces the loop that lived in `shannon_entropy_csv`); the summary keeps counts per (entropy kind, genomes present, genomes without ambiguity, 0.05-bit bin) instead of one line per family: 156 lines for the 46 706 families of the testbed (a few hundred thousand lines at most on any index, by construction: the bins are bounded). Checked: its `entropy15` counts sum to the number of variable families. +- Writers are in `cmd/phylo/stats.rs`; the library computes, the command layer formats and writes. + ## Open questions - Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy. diff --git a/UserDocMD/usage/phylo.md b/UserDocMD/usage/phylo.md index 70c84389..609fa6a7 100644 --- a/UserDocMD/usage/phylo.md +++ b/UserDocMD/usage/phylo.md @@ -16,7 +16,7 @@ obikmer phylo INDEX [OPTIONS] | Option | Default | Description | |---|---|---| -| `--distance` | `jaccard` | See the two tables below for the full list of accepted values | +| `--distance` | none | Compute and write a distance matrix. See the two tables below for the full list of accepted values. No matrix is computed unless this is given, or `--nj`, `--upgma` or `--shared-kmers` is (they need a matrix and then use `jaccard`) | | `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it | | `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index | | `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format | @@ -31,7 +31,7 @@ Every value routes to one of two independent computations: **whole-index distanc ### Output -Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `labelvalue...` row per genome). With `--csv`, the format is instead a header row `genome,,,...` followed by one `