Implement advanced sampling and family annex logic
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data. This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
This commit is contained in:
1 parent
0d12494f1a
commit
f29dcbcfb8
25 files changed
+1471
-486
No files matched your search
@@ -214,6 +214,70 @@ Per-genome share of the 5 003 414 sampled columns with a call: median 5.2 %, max
|
|||||||
|
|
||||||
Consequence for flattening: raw $O_{ij}$ mixes evolutionary distance and per-genome coverage; flattening on it chases pairs involving data-poor genomes, for which no family exists. Normalizing, $O_{ij}/(c_ic_j)$ with $c_i$ the number of families carried by genome $i$ (per-genome counts from `--sibling-stats`), targets distance instead of coverage; the best normalization (smoothing of $c_i$, etc.) is not established.
|
Consequence for flattening: raw $O_{ij}$ mixes evolutionary distance and per-genome coverage; flattening on it chases pairs involving data-poor genomes, for which no family exists. Normalizing, $O_{ij}/(c_ic_j)$ with $c_i$ the number of families carried by genome $i$ (per-genome counts from `--sibling-stats`), targets distance instead of coverage; the best normalization (smoothing of $c_i$, etc.) is not established.
|
||||||
|
|
||||||
|
## Implemented: entropy annex parametrised by `--no-ambiguity` (2026-10-13)
|
||||||
|
|
||||||
|
Design decision (user): the way genomes are counted depends on `--no-ambiguity`, so that flag, not an "alphabet" option, is the annex parameter. Over single-member genomes the 15-state entropy only has the four single-base states, hence `entropy4` (redefined: 4 bases over single-member genomes) equals `entropy15` under `--no-ambiguity`; a unit test checks it.
|
||||||
|
|
||||||
|
- `entropy_annex.rs`: header `flags` (the former padding), bit 0 = `no_ambiguity`; old files have 0 and stay valid. One file per convention (`entropy.pent`, `entropy.noamb.pent`), so switching the flag builds the missing one without discarding the other. `EntropyAnnex::no_ambiguity()`; `iter_full_entropy` rejects a header that disagrees with the requested convention.
|
||||||
|
- `ensure_layer_entropy_annex` takes the list of conventions to build; one scan feeds every missing annex and the overlap accumulator. `sample_index` builds and reads the convention matching its `no_ambiguity` argument, removing the mismatch between the draw (previously always raw masks) and the alignment (masked).
|
||||||
|
- `--sibling-annex` builds both conventions and invalidates both on rebuild. `--shannon` no longer builds any annex as a prerequisite (it recomputes everything from its own scan).
|
||||||
|
- Checked on the 1x benchmark index: a `--no-ambiguity` run creates `entropy.noamb.pent` beside the existing `entropy.pent` in all 512 layers; workspace tests pass.
|
||||||
|
- Still to do, in this order: `--quartet` (per-family bit annex, same parameter), `--pair-balance` (mean of 1/shared-families over the carrier pairs; second overlap matrix for the single-member convention, annex header carrying a fingerprint of the matrix it came from).
|
||||||
|
|
||||||
|
## Implemented: `--quartet` (2026-10-13)
|
||||||
|
|
||||||
|
`--quartet` keeps only the families with at least two called states each carried by at least two genomes (`masking::is_quartet_informative` for the raw masks, `survives_quartet` for the exact check on masked states and `--exclude-genome`). Absence never counts as a state; with `--no-ambiguity`, multi-member genomes are left out of the count. The persisted annex is a plain `PersistentBitVec` (one bit per non-monomorphic family, compact ordinal) — no ad hoc format; the `--no-ambiguity` convention is carried by the file name (`quartet.pbiv`, `quartet.noamb.pbiv`) because that header has no room for a flag. It is a safe pre-filter: excluding genomes can only destroy a bipartition, never create one.
|
||||||
|
|
||||||
|
Plumbing: `AnnexRequest` (which annexes, per convention) and `ensure_layer_family_annexes` (one shared scan for entropy, quartet and overlap) replace the entropy-only builder; the sampling parameters travel as one `SamplingSpec { free_loss, no_ambiguity, quartet, entropy_bias }` instead of a growing positional list.
|
||||||
|
|
||||||
|
First test on the 1x benchmark testbed (3 seeds, exhaustive alignment, `GTR+G4+ASC`, species monophyly out of 3 per seed): without filter 1 / 0 / 0 correct monophylies; carriers >= 4 only: 3 / 1 / 2; `--quartet`: 2 / 2 / 3 (370, 148, 369 columns, all informative by construction). *Klebsiella* stays fragile (wrong in 2 seeds out of 3); two seeds still show a branch at the IQ-TREE ceiling of 10 with so few columns. Small testbed, three seeds: a direction, not a measurement.
|
||||||
|
|
||||||
|
## Implemented: `--subsample` renamed `--n-sites N|all` (2026-10-13)
|
||||||
|
|
||||||
|
`--subsample` read as "draw a subset", while `--pseudo-alignment`/`--sankoff` made it mandatory and accepted no way to say "everything" (the workaround was a number larger than the family count, e.g. `--subsample 100000000`). The option is now `--n-sites <N|all>`: `all` is an explicit value, parsed into its own type (`NSitesArg::{All, N}`, like `GammaShapeArg`) rather than an in-band sentinel (negative number, `0`); `all` gives the exhaustive scan the `snp-*` path already ran when the option was omitted. Still mandatory for `--pseudo-alignment`/`--sankoff`; still optional for `snp-*` `--distance`. No alias is kept for the old name (the software is in active development). The benchmark commands and measurements recorded in the earlier sections of this file were run with the old name.
|
||||||
|
|
||||||
|
## Implemented: `--pair-balance` (2026-10-13)
|
||||||
|
|
||||||
|
Weight of a family = **mean** over the pairs of its carriers of 1/O (O = index-wide shared variable families of the pair, same `--no-ambiguity` convention): the simulation on the *Primulaceae* sample (above) had the mean flatten best (expected-overlap range 23 000× to ~600×, slope 0.37) without chasing high-carrier families, unlike the sum (65 % of the draw at >= 33 carriers).
|
||||||
|
|
||||||
|
- `algorithms/pair_weight.rs`: `inverse_overlap_matrix` (dense n×n of 1/O from the persisted overlap annex), `pair_balance_weight` (mean over carrier pairs; 0 below two carriers). Carriers are defined once, `helpers::is_carrier(mask, no_ambiguity)`, shared with the overlap accumulator and the quartet test.
|
||||||
|
- Storage: no new format. The entropy annex is generalised to `FamilyScalarAnnex` with a `ScalarKind` (`Entropy` magic `PENT`, `PairWeight` magic `PPWT`, one file per kind and convention, header flag `no_ambiguity`); `entropy_annex.rs` became `family_scalar_annex.rs`. Family overlap is now one file per convention too (`family_overlap.povl`, `family_overlap.noamb.povl`, header flag); `--family-overlap` and `--min-shared-family` follow `--no-ambiguity`.
|
||||||
|
- Build order: sweep 1 builds entropy, quartet and the missing overlap annexes together; sweep 2 builds the pair weights, which need the finished matrix (`ensure_layer_pair_weight_annexes`). `AnnexRequest` now lists `entropy`, `quartet`, `overlap`, `pair_weight`; a pair weight implies its overlap. A rebuilt sibling annex removes every annex derived from it (entropy, quartet, pair weight, overlap), so a weight can never outlive the matrix it came from; there is no fingerprint, only that single invalidation point.
|
||||||
|
- Sampler: `LayerPlan` (eligible bits, minorant count, mass) replaces the positional tuple; a layer's quota is `n × mass / Σ mass` (mass = eligible count without the option, Σ weights with it); a family is accepted with probability `quota/mass × weight × entropy kernel`, capped at 1 (the cap, and kernels below 1, undershoot the quota, absorbed by the existing top-up rounds).
|
||||||
|
- Cost note: weights need O(carriers²) per family at build time; not measured on a large index.
|
||||||
|
|
||||||
|
First test, 1x benchmark testbed (20 genomes, 46 706 variable families, `--n-sites 5000 --free-loss --no-ambiguity`, 8 replicates each; sample overlap computed from the alignment, compared with the index `--family-overlap --no-ambiguity`):
|
||||||
|
|
||||||
|
| | slope of log(sample overlap) on log(index overlap) | pairs (index overlap > 0) covered by >= 1 column | columns with >= 10 carriers |
|
||||||
|
|---|---|---|---|
|
||||||
|
| uniform | 1.02 ± 0.14 | 64 % | 0.18 % |
|
||||||
|
| `--pair-balance` | 0.84 ± 0.001 | 97 % | 1.1 % |
|
||||||
|
|
||||||
|
The weight flattens the representation (slope toward 0) and, above all, covers most of the pairs a uniform draw misses. Not measured: effect on the tree (sCF / monophyly), interaction with `--entropy`, behaviour on the *Primulaceae* index. 20 genomes only.
|
||||||
|
|
||||||
|
## Quota rounding fix, and the combined test on the 1x testbed (2026-10-13)
|
||||||
|
|
||||||
|
**Fix.** `sample_index` rounded each layer's quota down on its own, losing up to one site per layer: with 512 layers a budget of 100 sites gave 0 to 3 columns. Quotas now use cumulative rounding (`floor(n × cumulative mass / total)` minus what earlier layers got), so they sum to exactly `n`; a 100-site budget now yields 50-80 usable columns. Not a result of the options below, a defect the small testbed exposed.
|
||||||
|
|
||||||
|
**Test.** 1x testbed, 3 seeds × 10 replicates per variant, 11 genomes (the 9 genomes with fewest calls excluded, so no all-missing sequence), `--free-loss --no-ambiguity`, IQ-TREE `GTR+G4+ASC` on each sampled alignment; score = number of the three species (*E. coli*, *Salmonella*, *Klebsiella*) recovered monophyletic (0-3), mean over the 30 runs; "informative" = columns with two bases each carried by >= 2 genomes.
|
||||||
|
|
||||||
|
| `--n-sites` | variant | trees obtained | columns | informative columns | mean species monophyletic (of 3) | all 3 |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| 100 | uniform | 77 % | 49 | 0.5 | 0.04 | 0 % |
|
||||||
|
| 100 | `--quartet` | 100 % | 82 | 82 | 2.30 | 50 % |
|
||||||
|
| 100 | `--pair-balance` | 100 % | 52 | 2.1 | 0.20 | 0 % |
|
||||||
|
| 100 | `--quartet --pair-balance` | 100 % | 67 | 67 | 2.53 | 53 % |
|
||||||
|
| 100 | `... --entropy 1` | 100 % | 67 | 67 | 2.17 | 17 % |
|
||||||
|
| 300 | uniform | 93 % | 163 | 2.1 | 0.04 | 0 % |
|
||||||
|
| 300 | `--quartet` | 100 % | 239 | 239 | 2.33 | 33 % |
|
||||||
|
| 300 | `--pair-balance` | 100 % | 179 | 6.2 | 0.57 | 0 % |
|
||||||
|
| 300 | `--quartet --pair-balance` | 100 % | 197 | 197 | 2.67 | 67 % |
|
||||||
|
| 300 | `... --entropy 1` | 100 % | 196 | 196 | 2.67 | 67 % |
|
||||||
|
|
||||||
|
(Mean over all runs counting a failed tree as 0 is identical to the "mean species" column wherever trees were obtained; the uniform rows lose runs because IQ-TREE refuses alignments where a genome has no data.)
|
||||||
|
|
||||||
|
Reading: `--quartet` is the decisive option here — a uniform draw of 100-300 sites contains almost no informative column (0.5-2), the restricted draw contains only informative ones. `--pair-balance` alone barely helps (it favours rare pairs but does not remove the uninformative columns); added to `--quartet` it raises the share of replicates recovering all three species (50→53 % and 33→67 %). The entropy kernel (centre 1 bit, default width) adds nothing at 300 sites and costs at 100. Limits: 11 genomes, three simulated 1x datasets, small budgets, a single criterion (species monophyly, not deep branches), and the differences between the `--quartet` variants (a few runs out of 30) are small relative to the replicate noise — a direction, not a ranking.
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
|
- Per-family $S_f$ without materializing the full matrix: a first pass storing only a per-family summary (reach, $|S_f|$), as `--shannon` already does a full unsampled scan for entropy.
|
||||||
|
|||||||
@@ -13,12 +13,22 @@ A family's informativeness is measured directly rather than assumed uniform. For
|
|||||||
| Quantity | Definition |
|
| Quantity | Definition |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||||
| `entropy4` | the same, reduced to the 4 plain bases |
|
| `entropy4` | the same over the 4 plain bases, restricted to the genomes carrying exactly one member of the family (those that stay informative when multi-member genomes are treated as missing data) |
|
||||||
|
|
||||||
|
With `--no-ambiguity` the entropy is `entropy4`, computed over the genomes carrying exactly one member of the family, so that the weighting sees the same genomes as the final alignment; without it, `entropy15` is used. Each is stored in its own annex.
|
||||||
|
|
||||||
A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).
|
A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).
|
||||||
|
|
||||||
|
## Quartet restriction
|
||||||
|
|
||||||
|
A family only constrains the shape of a tree if its carrier genomes show a bipartition: two distinct states, each carried by at least two genomes. With fewer carriers, or with a single variant against a uniform background, every topology explains the family equally well. In a skimming dataset many families have only two or three carriers, so a uniform draw spends a large share of its quota on them. `--quartet` removes them before the draw. The test counts called states only, so absence never forms one of the two pairs.
|
||||||
|
|
||||||
|
## Pair balance
|
||||||
|
|
||||||
|
How well a pair of genomes is represented in a sample depends on how many families the pair shares: pairs of distant genomes share few, so a uniform draw gives them few sites, and the deep branches of a tree — which rest on exactly those pairs — are the least supported. `--pair-balance` weights each family by the mean, over the pairs of genomes carrying it, of 1 / (families shared by the pair), computed on the whole index. Averaging rather than summing keeps the weight from growing with the number of carriers, which would favour the families carried by most genomes regardless of which pairs they inform. Layers are allotted their share of the draw in proportion to the total weight of their families.
|
||||||
|
|
||||||
## Entropy-biased sampling
|
## Entropy-biased sampling
|
||||||
|
|
||||||
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
||||||
|
|
||||||
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---subsample---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
||||||
+30
-13
@@ -27,7 +27,7 @@ obikmer phylo INDEX [OPTIONS]
|
|||||||
|
|
||||||
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
||||||
|
|
||||||
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
|
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||||
|
|
||||||
### Output
|
### Output
|
||||||
|
|
||||||
@@ -53,7 +53,7 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
|||||||
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
||||||
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
||||||
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
||||||
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--subsample N` |
|
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
|
||||||
|
|
||||||
### `--sibling-stats`
|
### `--sibling-stats`
|
||||||
|
|
||||||
@@ -66,7 +66,7 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
|||||||
|
|
||||||
### Family Overlap
|
### Family Overlap
|
||||||
|
|
||||||
`--family-overlap` writes `<prefix>_family_overlap.csv`: header `genome,<label1>,<label2>,...`, one row per genome, cell `[i][j]` = number of variable families (family size ≥ 2) where both genome `i` and genome `j` carry a call. The diagonal is always `0`. Every genome is written, unfiltered by `--exclude-genome`/`--min-shared-family`.
|
`--family-overlap` writes `<prefix>_family_overlap.csv`: header `genome,<label1>,<label2>,...`, one row per genome, cell `[i][j]` = number of variable families (family size ≥ 2) where both genome `i` and genome `j` carry a call. The diagonal is always `0`. Every genome is written, unfiltered by `--exclude-genome`/`--min-shared-family`. With `--no-ambiguity`, a genome carrying several members of a family is not counted as carrying it, as in the alignment; without it, any genome with a call counts. The matrix is stored once per convention, so switching `--no-ambiguity` builds the other one without discarding the first (`--min-shared-family` follows the same convention).
|
||||||
|
|
||||||
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
|
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
|
||||||
|
|
||||||
@@ -76,15 +76,17 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
|||||||
|
|
||||||
| Option | Default | Description |
|
| Option | Default | Description |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `--subsample N` | none (mandatory here) | Target number of families to sample |
|
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
|
||||||
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
||||||
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
||||||
|
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct called states, each carried by at least two genomes. Which genomes count as carriers follows `--no-ambiguity` |
|
||||||
|
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
|
||||||
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
||||||
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
||||||
|
|
||||||
## Sampling at scale: `--subsample`, `--shannon`, `--entropy`
|
## Sampling at scale: `--n-sites`, `--shannon`, `--entropy`
|
||||||
|
|
||||||
`--subsample`, `--free-loss`, `--no-ambiguity`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--subsample` is mandatory for `--pseudo-alignment`/`--sankoff`; for a `snp-*` `--distance` value it is optional (omitted means every non-monomorphic family in the index, not an approximation).
|
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
|
||||||
|
|
||||||
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
|
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
|
||||||
|
|
||||||
@@ -97,13 +99,13 @@ Every invocation, with or without `--session`, draws its own fresh random sample
|
|||||||
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
|
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
|
||||||
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
|
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
|
||||||
|
|
||||||
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--subsample`/`--free-loss`/`--no-ambiguity`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
|
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
|
||||||
|
|
||||||
A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
|
A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
|
||||||
|
|
||||||
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
||||||
|
|
||||||
Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If the index has fewer than `N` candidate families, `--subsample` has no effect.
|
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
|
||||||
|
|
||||||
### `--shannon`: measuring how informative a family is
|
### `--shannon`: measuring how informative a family is
|
||||||
|
|
||||||
@@ -115,23 +117,38 @@ Without `--subsample`, every variable family (family size ≥ 2) is used. With `
|
|||||||
| `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
|
| `layer` | an internal index-layer identifier — stable within one run, not meaningful across indexes |
|
||||||
| `family_idx` | the family's position within that layer |
|
| `family_idx` | the family's position within that layer |
|
||||||
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||||
| `entropy4` | Shannon entropy (bits) reduced to the 4 plain bases, kept alongside `entropy15` for comparison |
|
| `entropy4` | Shannon entropy (bits) over the 4 plain bases, computed only over the genomes that carry exactly one member of the family — the genomes that remain informative under `--no-ambiguity`, since a genome carrying several members of a family (paralogous copies) is treated as missing data there. `0` when `n_genomes_unambiguous` is `0` |
|
||||||
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
|
| `family_size` | number of distinct central bases observed anywhere in the index for this family (2-4) |
|
||||||
| `n_genomes_present` | how many genomes the entropy was computed over |
|
| `n_genomes_present` | how many genomes the family is present in; `entropy15` is computed over these |
|
||||||
|
| `n_genomes_unambiguous` | how many of those genomes carry exactly one member of the family; `entropy4` is computed over these |
|
||||||
|
|
||||||
`--shannon` always scans every non-monomorphic family of the whole index; `--subsample`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
`--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
||||||
|
|
||||||
|
### `--quartet`: keeping only families that can resolve a tree
|
||||||
|
|
||||||
|
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. Absence is never a state here: a genome that does not carry the family does not count, with or without `--free-loss`. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
||||||
|
|
||||||
|
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` convention, in one pass shared with the other annexes, and built for both conventions by `--sibling-annex`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
||||||
|
|
||||||
|
### `--pair-balance`: representing every pair of genomes
|
||||||
|
|
||||||
|
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
|
||||||
|
|
||||||
|
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` convention, after the family-overlap matrix it depends on, and by `--sibling-annex` for both conventions). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
|
||||||
|
|
||||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||||
|
|
||||||
By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||||
|
|
||||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
||||||
|
|
||||||
|
The entropy that weights the draw follows `--no-ambiguity`, so that it describes the same genomes as the alignment. Without the flag, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With the flag, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). The two entropies are saved separately: switching `--no-ambiguity` builds the other one once, without discarding the first, and `--sibling-annex` builds both.
|
||||||
|
|
||||||
## Sankoff calibration and phylogenetic exports
|
## Sankoff calibration and phylogenetic exports
|
||||||
|
|
||||||
| Option | Default | Description |
|
| Option | Default | Description |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment. Requires `--subsample N` |
|
| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment. Requires `--n-sites N\|all` |
|
||||||
| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the base-composition part of the calibration |
|
| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the base-composition part of the calibration |
|
||||||
| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, throughout `--sankoff` and every export built from it |
|
| `--free-loss` | off | Recode a family's non-detection as the `?` missing-data symbol instead of an ordinary, costed state, throughout `--sankoff` and every export built from it |
|
||||||
| `--tnt` | off | Also write a TNT script (implies `--sankoff`) |
|
| `--tnt` | off | Also write a TNT script (implies `--sankoff`) |
|
||||||
|
|||||||
Executable
+57
@@ -0,0 +1,57 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Genome-skimming test bed: re-simulate the benchmark genomes at a low
|
||||||
|
# coverage, index each specimen, and merge them into one presence index.
|
||||||
|
#
|
||||||
|
# Usage: simulate_skim.sh (env: COVERAGE=1 SEED=1 CPUS=...)
|
||||||
|
#
|
||||||
|
# The genome → specimen mapping is read from deps.mk (same source of truth
|
||||||
|
# as the Makefile pipeline); the pipeline's own run/ tree is left untouched.
|
||||||
|
# Output: run/skim_${COVERAGE}x_s${SEED}/
|
||||||
|
# simulated_data/<species>/<strain>/reads_R{1,2}.fastq.gz
|
||||||
|
# specimen_index_presence/<species>--<strain>/
|
||||||
|
# global_index_presence/
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ISS="${SCRIPT_DIR}/../.venv/bin/iss"
|
||||||
|
BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
|
||||||
|
COVERAGE="${COVERAGE:-1}"
|
||||||
|
SEED="${SEED:-1}"
|
||||||
|
READ_LENGTH=150
|
||||||
|
CPUS="${CPUS:-$(sysctl -n hw.logicalcpu 2>/dev/null || nproc 2>/dev/null || echo 2)}"
|
||||||
|
|
||||||
|
OUT="${SCRIPT_DIR}/run/skim_${COVERAGE}x_s${SEED}"
|
||||||
|
mkdir -p "${OUT}/simulated_data" "${OUT}/specimen_index_presence"
|
||||||
|
|
||||||
|
tmp_fasta=$(mktemp "${TMPDIR:-/tmp}/obikmer_skim_XXXXXX.fna")
|
||||||
|
trap 'rm -f "${tmp_fasta}"' EXIT
|
||||||
|
|
||||||
|
# lines of the form run/simulated_data/<species>/<strain>/reads_R1.fastq.gz: run/genomes/<file>
|
||||||
|
sed -nE 's#^run/simulated_data/([^/]+)/([^/]+)/reads_R1\.fastq\.gz: (run/genomes/[^ ]+)$#\1 \2 \3#p' \
|
||||||
|
"${SCRIPT_DIR}/deps.mk" | sort -u | while read -r species strain genome; do
|
||||||
|
specimen="${species}--${strain}"
|
||||||
|
reads_dir="${OUT}/simulated_data/${species}/${strain}"
|
||||||
|
index_dir="${OUT}/specimen_index_presence/${specimen}"
|
||||||
|
mkdir -p "${reads_dir}"
|
||||||
|
|
||||||
|
gzip -dc "${SCRIPT_DIR}/${genome}" > "${tmp_fasta}"
|
||||||
|
genome_size=$(grep -v "^>" "${tmp_fasta}" | tr -d '[:space:]' | wc -c | tr -d ' ')
|
||||||
|
n_reads=$(python3 -c "import math; print(math.ceil(${COVERAGE} * ${genome_size} / (2 * ${READ_LENGTH})))")
|
||||||
|
echo "[${specimen}] genome=${genome_size} bp → ${n_reads} read pairs (${COVERAGE}x, seed ${SEED})"
|
||||||
|
|
||||||
|
"${ISS}" generate \
|
||||||
|
--genomes "${tmp_fasta}" --model HiSeq --n_reads "${n_reads}" \
|
||||||
|
--cpus "${CPUS}" --seed "${SEED}" --compress \
|
||||||
|
--output "${reads_dir}/reads" 2>/dev/null
|
||||||
|
rm -f "${reads_dir}"/reads.iss.tmp.*.vcf
|
||||||
|
|
||||||
|
"${BINARY}" index --output "${index_dir}" --force --theta 0 \
|
||||||
|
--label "${specimen}" --meta "species=${species}" \
|
||||||
|
"${reads_dir}/reads_R1.fastq.gz" "${reads_dir}/reads_R2.fastq.gz" 2>/dev/null
|
||||||
|
done
|
||||||
|
|
||||||
|
mapfile -t sources < <(find "${OUT}/specimen_index_presence" -mindepth 1 -maxdepth 1 -type d | sort)
|
||||||
|
echo "Merging ${#sources[@]} presence indexes → ${OUT}/global_index_presence"
|
||||||
|
"${BINARY}" merge --output "${OUT}/global_index_presence" --force --force-presence \
|
||||||
|
"${sources[@]}" 2>/dev/null
|
||||||
|
echo "Done: ${OUT}/global_index_presence"
|
||||||
Generated
+1
-1
@@ -1532,7 +1532,7 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "obikmer"
|
name = "obikmer"
|
||||||
version = "1.3.5"
|
version = "1.3.6"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"clap",
|
"clap",
|
||||||
"csv",
|
"csv",
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "obikmer"
|
name = "obikmer"
|
||||||
version = "1.3.5"
|
version = "1.3.6"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
|
||||||
[[bin]]
|
[[bin]]
|
||||||
|
|||||||
@@ -32,6 +32,36 @@ fn parse_gamma_shape(s: &str) -> Result<GammaShapeArg, String> {
|
|||||||
.map_err(|_| format!("invalid --gamma-shape value {s:?}: expected a positive number or \"auto\"/\"estimate\""))
|
.map_err(|_| format!("invalid --gamma-shape value {s:?}: expected a positive number or \"auto\"/\"estimate\""))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `--n-sites` CLI value: a number of sites to draw, or `all` for every site
|
||||||
|
/// — the draw then keeps every variable family that passes the other filters,
|
||||||
|
/// with no sampling involved. Its own tiny type (like [`GammaShapeArg`]) so
|
||||||
|
/// the two cases stay distinct up to the call that needs the target, instead
|
||||||
|
/// of an in-band sentinel (a negative number, `0`, ...) that a typo could hit.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum NSitesArg {
|
||||||
|
All,
|
||||||
|
N(usize),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl NSitesArg {
|
||||||
|
/// The target to hand to the sampler: `None` = exhaustive.
|
||||||
|
pub fn target(self) -> Option<usize> {
|
||||||
|
match self {
|
||||||
|
NSitesArg::All => None,
|
||||||
|
NSitesArg::N(n) => Some(n),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_n_sites(s: &str) -> Result<NSitesArg, String> {
|
||||||
|
if s.eq_ignore_ascii_case("all") {
|
||||||
|
return Ok(NSitesArg::All);
|
||||||
|
}
|
||||||
|
s.parse::<usize>()
|
||||||
|
.map(NSitesArg::N)
|
||||||
|
.map_err(|_| format!("invalid --n-sites value {s:?}: expected a non-negative integer or \"all\""))
|
||||||
|
}
|
||||||
|
|
||||||
/// `--distance` value — either one of `obikphylo::DistanceMetric`'s
|
/// `--distance` value — either one of `obikphylo::DistanceMetric`'s
|
||||||
/// whole-index metrics (routed to `IndexCache::distance`) or one of
|
/// whole-index metrics (routed to `IndexCache::distance`) or one of
|
||||||
/// `obikphylo::SnpDistanceKind`'s `snp-*` corrections (routed to
|
/// `obikphylo::SnpDistanceKind`'s `snp-*` corrections (routed to
|
||||||
@@ -108,7 +138,7 @@ impl DistanceArg {
|
|||||||
/// (classic metrics + `snp-*` corrections/NJ/UPGMA), annex construction
|
/// (classic metrics + `snp-*` corrections/NJ/UPGMA), annex construction
|
||||||
/// (`--sibling-annex`), annex diagnostics (`--sibling-stats`,
|
/// (`--sibling-annex`), annex diagnostics (`--sibling-stats`,
|
||||||
/// `--sibling-hist`), entropy reporting (`--shannon`), SNP pseudo-alignment
|
/// `--sibling-hist`), entropy reporting (`--shannon`), SNP pseudo-alignment
|
||||||
/// sampling (`--pseudo-alignment`, `--subsample`, `--free-loss`,
|
/// sampling (`--pseudo-alignment`, `--n-sites`, `--free-loss`,
|
||||||
/// `--no-ambiguity`, `--entropy`/`--entropy-sd`), Sankoff cost-matrix
|
/// `--no-ambiguity`, `--entropy`/`--entropy-sd`), Sankoff cost-matrix
|
||||||
/// calibration (`--sankoff`, `--sankoff-ratio-ceiling`) and its TNT/PhyG/
|
/// calibration (`--sankoff`, `--sankoff-ratio-ceiling`) and its TNT/PhyG/
|
||||||
/// IQ-TREE exports (`--tnt`, `--phyg`, `--iqtree`/`--iqtree-min-freq`,
|
/// IQ-TREE exports (`--tnt`, `--phyg`, `--iqtree`/`--iqtree-min-freq`,
|
||||||
@@ -180,26 +210,25 @@ pub struct PhyloArgs {
|
|||||||
/// Write a per-family Shannon entropy report (CSV) — requires an
|
/// Write a per-family Shannon entropy report (CSV) — requires an
|
||||||
/// already-built sibling annex (`--sibling-annex` first, in this
|
/// already-built sibling annex (`--sibling-annex` first, in this
|
||||||
/// invocation or an earlier one). Always a full, unsampled scan of
|
/// invocation or an earlier one). Always a full, unsampled scan of
|
||||||
/// every family (`--subsample`/`--entropy`/`--entropy-sd` below only
|
/// every family (`--n-sites`/`--entropy`/`--entropy-sd` below only
|
||||||
/// apply to `--pseudo-alignment`, not this).
|
/// apply to `--pseudo-alignment`, not this).
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
pub shannon: bool,
|
pub shannon: bool,
|
||||||
|
|
||||||
/// Write a SNP-only pseudo-alignment (FASTA) — requires
|
/// Write a SNP-only pseudo-alignment (FASTA) — requires
|
||||||
/// `--subsample <N>` and an already-built sibling annex
|
/// `--n-sites <N|all>` and an already-built sibling annex
|
||||||
/// (`--sibling-annex` first, in this invocation or an earlier one).
|
/// (`--sibling-annex` first, in this invocation or an earlier one).
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
pub pseudo_alignment: bool,
|
pub pseudo_alignment: bool,
|
||||||
|
|
||||||
/// Target number of variable sites to sample index-wide for
|
/// Number of variable sites to draw index-wide, or `all` for every site
|
||||||
/// `--pseudo-alignment`/`--sankoff` (mandatory for both) and for a
|
/// that passes the other filters (no sampling then). Mandatory for
|
||||||
/// `snp-*` `--distance` value (optional there: omitted means exhaustive
|
/// `--pseudo-alignment`/`--sankoff`; optional for a `snp-*`
|
||||||
/// — every non-monomorphic minorant of the whole index, not an
|
/// `--distance` value, where omitting it also means every site. A
|
||||||
/// approximation) — a target, not a guarantee when given (proportional
|
/// number is a target, not a guarantee (proportional per-layer
|
||||||
/// per-layer sampling; see `obikphylo::siblings::sampled_scan`'s own
|
/// sampling; see `obikphylo::siblings::sampled_scan`'s own docs).
|
||||||
/// docs).
|
#[arg(long = "n-sites", value_name = "N|all", value_parser = parse_n_sites)]
|
||||||
#[arg(long)]
|
pub n_sites: Option<NSitesArg>,
|
||||||
pub subsample: Option<usize>,
|
|
||||||
|
|
||||||
/// In `--pseudo-alignment`, treat a genome carrying none of a family's
|
/// In `--pseudo-alignment`, treat a genome carrying none of a family's
|
||||||
/// observed members (`∅`) as missing data (`?`) rather than a real
|
/// observed members (`∅`) as missing data (`?`) rather than a real
|
||||||
@@ -213,6 +242,24 @@ pub struct PhyloArgs {
|
|||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
pub no_ambiguity: bool,
|
pub no_ambiguity: bool,
|
||||||
|
|
||||||
|
/// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, keep only
|
||||||
|
/// the families that can separate two pairs of genomes from each other
|
||||||
|
/// (a bipartition): at least two distinct called states, each carried
|
||||||
|
/// by at least two genomes. A singleton variant, or a family carried by
|
||||||
|
/// fewer than four genomes, fits every topology and tells a tree
|
||||||
|
/// nothing. Which genomes count as carriers follows `--no-ambiguity`.
|
||||||
|
#[arg(long)]
|
||||||
|
pub quartet: bool,
|
||||||
|
|
||||||
|
/// In `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance`, favour the
|
||||||
|
/// families that cover the pairs of genomes sharing the fewest families
|
||||||
|
/// across the index: each family is weighted by the mean, over the pairs
|
||||||
|
/// of genomes carrying it, of 1 / (number of families the pair shares).
|
||||||
|
/// Evens out how well each pair of genomes is represented in the sample.
|
||||||
|
/// Which genomes count as carriers follows `--no-ambiguity`.
|
||||||
|
#[arg(long)]
|
||||||
|
pub pair_balance: bool,
|
||||||
|
|
||||||
/// Entropy-biased sampling target (Gaussian kernel mean) for
|
/// Entropy-biased sampling target (Gaussian kernel mean) for
|
||||||
/// `--pseudo-alignment` — activates biasing as soon as this or
|
/// `--pseudo-alignment` — activates biasing as soon as this or
|
||||||
/// `--entropy-sd` is given; the other defaults to 1.0/0.5.
|
/// `--entropy-sd` is given; the other defaults to 1.0/0.5.
|
||||||
@@ -227,7 +274,7 @@ pub struct PhyloArgs {
|
|||||||
/// Persist the `snp-*` `--distance` sample (the pairwise substitution
|
/// Persist the `snp-*` `--distance` sample (the pairwise substitution
|
||||||
/// tally) in `DIR` across separate `obikmer phylo` invocations: a later
|
/// tally) in `DIR` across separate `obikmer phylo` invocations: a later
|
||||||
/// run against the same index with the exact same
|
/// run against the same index with the exact same
|
||||||
/// `--subsample`/`--free-loss`/`--no-ambiguity`/`--exclude-genome`/
|
/// `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/
|
||||||
/// `--min-shared-family`/`--entropy`/`--entropy-sd` restores it instead
|
/// `--min-shared-family`/`--entropy`/`--entropy-sd` restores it instead
|
||||||
/// of resampling. `DIR` is created if it doesn't exist. If it already
|
/// of resampling. `DIR` is created if it doesn't exist. If it already
|
||||||
/// holds a sample built under *different* selection parameters, this
|
/// holds a sample built under *different* selection parameters, this
|
||||||
@@ -247,7 +294,7 @@ pub struct PhyloArgs {
|
|||||||
|
|
||||||
/// Calibrate a 16-state Sankoff cost matrix (and its matching
|
/// Calibrate a 16-state Sankoff cost matrix (and its matching
|
||||||
/// pseudo-alignment) from an already-built sibling annex — requires
|
/// pseudo-alignment) from an already-built sibling annex — requires
|
||||||
/// `--subsample <N>`, and shares `--free-loss`/`--no-ambiguity`/
|
/// `--n-sites <N|all>`, and shares `--free-loss`/`--no-ambiguity`/`--quartet`/
|
||||||
/// `--entropy`/`--entropy-sd` with `--pseudo-alignment` (one draw, same
|
/// `--entropy`/`--entropy-sd` with `--pseudo-alignment` (one draw, same
|
||||||
/// selection feeds both the alignment and every calibration tally).
|
/// selection feeds both the alignment and every calibration tally).
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
|
|||||||
@@ -11,13 +11,14 @@ use std::sync::Arc;
|
|||||||
use obikidxcache::index_cache::IndexCache;
|
use obikidxcache::index_cache::IndexCache;
|
||||||
use obikindex::KmerIndex;
|
use obikindex::KmerIndex;
|
||||||
use obikphylo::siblings::{
|
use obikphylo::siblings::{
|
||||||
EntropyBias, GammaShape, SiblingExt, cardinality_transition_probs,
|
EntropyBias, GammaShape, SamplingSpec, SiblingExt, cardinality_transition_probs,
|
||||||
composition_transition_probs, pairwise_cost_matrix,
|
composition_transition_probs, pairwise_cost_matrix,
|
||||||
};
|
};
|
||||||
use obikphylo::{Metrics, neighbor_joining, upgma};
|
use obikphylo::{Metrics, neighbor_joining, upgma};
|
||||||
use obisys::{Reporter, Stage};
|
use obisys::{Reporter, Stage};
|
||||||
use tracing::info;
|
use tracing::info;
|
||||||
|
|
||||||
|
use args::NSitesArg;
|
||||||
use iqtree::write_iqtree;
|
use iqtree::write_iqtree;
|
||||||
use phyg::write_sankoff_phyg;
|
use phyg::write_sankoff_phyg;
|
||||||
use phylip::write_phylip_relaxed;
|
use phylip::write_phylip_relaxed;
|
||||||
@@ -154,7 +155,7 @@ pub fn run(args: PhyloArgs) {
|
|||||||
// ── Family Overlap matrix (`--family-overlap`) ──────────────────────────────
|
// ── Family Overlap matrix (`--family-overlap`) ──────────────────────────────
|
||||||
if args.family_overlap {
|
if args.family_overlap {
|
||||||
let t = Stage::start("family_overlap");
|
let t = Stage::start("family_overlap");
|
||||||
let overlap = cache.family_overlap().unwrap_or_else(|e| {
|
let overlap = cache.family_overlap(args.no_ambiguity).unwrap_or_else(|e| {
|
||||||
eprintln!("error computing family overlap: {e}");
|
eprintln!("error computing family overlap: {e}");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
});
|
});
|
||||||
@@ -201,7 +202,7 @@ pub fn run(args: PhyloArgs) {
|
|||||||
// `args::PhyloArgs::min_shared_family`'s own docs on why).
|
// `args::PhyloArgs::min_shared_family`'s own docs on why).
|
||||||
let snp_exclude_mask: Vec<bool> = match args.min_shared_family {
|
let snp_exclude_mask: Vec<bool> = match args.min_shared_family {
|
||||||
Some(threshold) => {
|
Some(threshold) => {
|
||||||
let overlap = cache.family_overlap().unwrap_or_else(|e| {
|
let overlap = cache.family_overlap(args.no_ambiguity).unwrap_or_else(|e| {
|
||||||
eprintln!("error computing family overlap: {e}");
|
eprintln!("error computing family overlap: {e}");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
});
|
});
|
||||||
@@ -235,6 +236,13 @@ pub fn run(args: PhyloArgs) {
|
|||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
};
|
};
|
||||||
|
let sampling = SamplingSpec {
|
||||||
|
free_loss: args.free_loss,
|
||||||
|
no_ambiguity: args.no_ambiguity,
|
||||||
|
quartet: args.quartet,
|
||||||
|
pair_balance: args.pair_balance,
|
||||||
|
entropy_bias,
|
||||||
|
};
|
||||||
|
|
||||||
// ── `--session` ──────────────────────────────────────────────────────────
|
// ── `--session` ──────────────────────────────────────────────────────────
|
||||||
// Temporarily unavailable: the per-driver checkpoint/resume plumbing
|
// Temporarily unavailable: the per-driver checkpoint/resume plumbing
|
||||||
@@ -253,7 +261,7 @@ pub fn run(args: PhyloArgs) {
|
|||||||
|
|
||||||
// ── Sampled SNP scan (`--pseudo-alignment`, Sankoff group, `snp-*` `--distance`) ──
|
// ── Sampled SNP scan (`--pseudo-alignment`, Sankoff group, `snp-*` `--distance`) ──
|
||||||
// All three consume exactly the same selection parameters
|
// All three consume exactly the same selection parameters
|
||||||
// (`--subsample`/`--free-loss`/`--no-ambiguity`/`--exclude-genome`/
|
// (`--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/
|
||||||
// `--min-shared-family`/`--entropy`/`--entropy-sd`), and must draw from
|
// `--min-shared-family`/`--entropy`/`--entropy-sd`), and must draw from
|
||||||
// the *same* random sample when combined in one invocation — so
|
// the *same* random sample when combined in one invocation — so
|
||||||
// whichever of them are requested here share exactly one
|
// whichever of them are requested here share exactly one
|
||||||
@@ -265,13 +273,15 @@ pub fn run(args: PhyloArgs) {
|
|||||||
let snp_kind = args.distance.as_snp();
|
let snp_kind = args.distance.as_snp();
|
||||||
let need_alignment = args.pseudo_alignment || sankoff_requested;
|
let need_alignment = args.pseudo_alignment || sankoff_requested;
|
||||||
let need_pairwise = sankoff_requested || snp_kind.is_some();
|
let need_pairwise = sankoff_requested || snp_kind.is_some();
|
||||||
|
// `None` = every site (`--n-sites all`, or `--n-sites` omitted where it is optional)
|
||||||
|
let target = args.n_sites.and_then(NSitesArg::target);
|
||||||
|
|
||||||
if args.pseudo_alignment && args.subsample.is_none() {
|
if args.pseudo_alignment && args.n_sites.is_none() {
|
||||||
eprintln!("error: --pseudo-alignment requires --subsample <N>");
|
eprintln!("error: --pseudo-alignment requires --n-sites <N|all>");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
}
|
}
|
||||||
if sankoff_requested && args.subsample.is_none() {
|
if sankoff_requested && args.n_sites.is_none() {
|
||||||
eprintln!("error: --sankoff requires --subsample <N>");
|
eprintln!("error: --sankoff requires --n-sites <N|all>");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -290,13 +300,13 @@ pub fn run(args: PhyloArgs) {
|
|||||||
|
|
||||||
info!(
|
info!(
|
||||||
"sampling SNP sites{}",
|
"sampling SNP sites{}",
|
||||||
match args.subsample {
|
match target {
|
||||||
Some(n) => format!(" (target {n} site(s))"),
|
Some(n) => format!(" (target {n} site(s))"),
|
||||||
None => " (exhaustive)".into(),
|
None => " (exhaustive)".into(),
|
||||||
}
|
}
|
||||||
);
|
);
|
||||||
let t = Stage::start("sampled_scan");
|
let t = Stage::start("sampled_scan");
|
||||||
obikphylo::siblings::sampled_scan(&cache, args.subsample, args.free_loss, args.no_ambiguity, &snp_exclude_mask, entropy_bias, &mut reducers)
|
obikphylo::siblings::sampled_scan(&cache, target, sampling, &snp_exclude_mask, &mut reducers)
|
||||||
.unwrap_or_else(|e| {
|
.unwrap_or_else(|e| {
|
||||||
eprintln!("error sampling SNP sites: {e}");
|
eprintln!("error sampling SNP sites: {e}");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
|
|||||||
@@ -8,19 +8,6 @@
|
|||||||
//! it is actually observed) is the consistent informativeness proxy for
|
//! it is actually observed) is the consistent informativeness proxy for
|
||||||
//! this project, not a 4-symbol reduction.
|
//! this project, not a 4-symbol reduction.
|
||||||
|
|
||||||
use std::collections::HashSet;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
use obikidxcache::index_cache::IndexCache;
|
|
||||||
use obikindex::{OKIError, OKIResult};
|
|
||||||
|
|
||||||
use super::family_scan::{Selection, scan_layer_families};
|
|
||||||
use super::minorant_selection::{is_non_monomorphic_minorant, non_monomorphic_reindex_layer};
|
|
||||||
use crate::siblings::{
|
|
||||||
ANNEX_FILE_NAME, ENTROPY_ANNEX_FILE_NAME, EntropyAnnex, EntropyAnnexBuilder,
|
|
||||||
FamilyOverlapAccumulator, SiblingAnnex,
|
|
||||||
};
|
|
||||||
|
|
||||||
/// Shannon entropy (bits) of one family's per-genome states, over the 15
|
/// Shannon entropy (bits) of one family's per-genome states, over the 15
|
||||||
/// non-empty subsets of `{A,C,G,T}` actually observed among the genomes
|
/// non-empty subsets of `{A,C,G,T}` actually observed among the genomes
|
||||||
/// carrying it (`genome_mask[g] != 0`) — genomes where the family is absent
|
/// carrying it (`genome_mask[g] != 0`) — genomes where the family is absent
|
||||||
@@ -103,6 +90,26 @@ mod tests {
|
|||||||
assert_eq!(n, 2);
|
assert_eq!(n, 2);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn entropy_15_over_single_member_genomes_equals_entropy_4() {
|
||||||
|
// `--no-ambiguity` leaves only the single-base states, over which the
|
||||||
|
// 15-state entropy and the 4-base one are the same quantity
|
||||||
|
let masks = [0b0001u8, 0b0001, 0b0100, 0b1000, 0b0100, 0b0100, 0b0010];
|
||||||
|
let (h15, n15) = family_entropy(&masks).unwrap();
|
||||||
|
let (h4, n4) = family_entropy_4(&masks).unwrap();
|
||||||
|
assert!((h15 - h4).abs() < 1e-12);
|
||||||
|
assert_eq!(n15, n4);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn entropy_under_selects_the_counting_convention() {
|
||||||
|
let masks = [0b0001u8, 0b0001, 0b0011]; // the last genome carries two members
|
||||||
|
assert_eq!(family_entropy_under(&masks, true).unwrap().1, 2);
|
||||||
|
assert_eq!(family_entropy_under(&masks, false).unwrap().1, 3);
|
||||||
|
assert_eq!(family_entropy_under(&masks, true).unwrap().0, 0.0);
|
||||||
|
assert!(family_entropy_under(&masks, false).unwrap().0 > 0.0);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn entropy_4_is_none_without_unambiguous_genome() {
|
fn entropy_4_is_none_without_unambiguous_genome() {
|
||||||
assert!(family_entropy_4(&[0, 0b0011, 0b0110]).is_none());
|
assert!(family_entropy_4(&[0, 0b0011, 0b0110]).is_none());
|
||||||
@@ -110,104 +117,21 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Builds one layer's entropy annex if it doesn't already exist, and/or
|
/// Shannon entropy (bits) of one family under the counting convention
|
||||||
/// feeds `overlap_acc` (`Some` iff the index-wide family-overlap annex is
|
/// `--no-ambiguity` selects:
|
||||||
/// itself missing — see
|
/// - `false`: [`family_entropy`] — every genome carrying the family counts,
|
||||||
/// [`crate::siblings::FamilyOverlap`]'s own module docs) — a single shared
|
/// a genome with several members being a state of its own;
|
||||||
/// `scan_layer_families` pass covers both, since both need the exact same
|
/// - `true`: [`family_entropy_4`] — only the genomes carrying exactly one
|
||||||
/// per-family `genome_mask` resolution and there is no reason to pay for it
|
/// member count, the others being missing data, exactly as the alignment
|
||||||
/// twice just because only one of the two happens to be missing. A no-op
|
/// itself will see them. Over those genomes the 15-state entropy can only
|
||||||
/// (no scan at all) when entropy already exists for this layer *and*
|
/// ever take the four single-base states, so the two definitions coincide.
|
||||||
/// `overlap_acc` is `None`. Bounded to non-monomorphic minorants only
|
|
||||||
/// ([`non_monomorphic_reindex_layer`], a cheap annex-only pass — no
|
|
||||||
/// cross-partition resolution) whenever a scan does happen: monomorphism is
|
|
||||||
/// already known from the annex bits alone, so there is no reason to pay
|
|
||||||
/// `scan_layer_families`'s expensive per-genome resolution for the ~98% of
|
|
||||||
/// minorants that are monomorphic only to discard the result — true for
|
|
||||||
/// family-overlap's own "variable families only" scope just as much as for
|
|
||||||
/// entropy's.
|
|
||||||
pub(crate) fn ensure_layer_entropy_annex(
|
|
||||||
cache: &IndexCache,
|
|
||||||
partition: usize,
|
|
||||||
layer_idx: usize,
|
|
||||||
n_genomes: usize,
|
|
||||||
fast_mode: bool,
|
|
||||||
mut overlap_acc: Option<&mut FamilyOverlapAccumulator>,
|
|
||||||
) -> OKIResult<()> {
|
|
||||||
let layer = cache
|
|
||||||
.get_layer(partition, layer_idx)
|
|
||||||
.expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)");
|
|
||||||
let path = layer.dir().join(ENTROPY_ANNEX_FILE_NAME);
|
|
||||||
let need_entropy = !path.exists();
|
|
||||||
if !need_entropy && overlap_acc.is_none() {
|
|
||||||
return Ok(());
|
|
||||||
}
|
|
||||||
|
|
||||||
let reindex = non_monomorphic_reindex_layer(layer.dir())?;
|
|
||||||
let eligible: HashSet<usize> = reindex.keys().copied().collect();
|
|
||||||
let mut builder = need_entropy
|
|
||||||
.then(|| EntropyAnnexBuilder::new(reindex.len(), &path))
|
|
||||||
.transpose()
|
|
||||||
.map_err(OKIError::Io)?;
|
|
||||||
|
|
||||||
scan_layer_families(
|
|
||||||
cache,
|
|
||||||
partition,
|
|
||||||
layer_idx,
|
|
||||||
n_genomes,
|
|
||||||
fast_mode,
|
|
||||||
&Selection::Some(&eligible),
|
|
||||||
|family_idx, mask, genome_mask| {
|
|
||||||
debug_assert!(
|
|
||||||
mask.family_size() >= 2,
|
|
||||||
"Selection::Some(eligible) must only visit non-monomorphic minorants"
|
|
||||||
);
|
|
||||||
if let Some(builder) = builder.as_mut() {
|
|
||||||
if let Some((h15, _)) = family_entropy(genome_mask) {
|
|
||||||
builder.set(reindex[&family_idx], h15 as f32);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if let Some(acc) = overlap_acc.as_deref_mut() {
|
|
||||||
acc.add_family(genome_mask);
|
|
||||||
}
|
|
||||||
},
|
|
||||||
)?;
|
|
||||||
if let Some(builder) = builder {
|
|
||||||
builder.close().map_err(OKIError::Io)?;
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Streams one entropy value per minorant of the layer, **in `family_idx`
|
|
||||||
/// order** — monomorphic minorants included, valued `0.0` by convention
|
|
||||||
/// (trivially uninformative, never computed or stored in the compact
|
|
||||||
/// entropy annex). Exists so a caller can `.zip()` this directly against
|
|
||||||
/// any other `family_idx`-ordered stream (e.g.
|
|
||||||
/// `SiblingLayerExt::iter_minorants`) instead of separately consulting
|
|
||||||
/// [`non_monomorphic_reindex_layer`]'s map at each site.
|
|
||||||
///
|
///
|
||||||
/// Requires this layer's entropy annex to already exist
|
/// Second value: the number of genomes the entropy was computed over.
|
||||||
/// ([`ensure_layer_entropy_annex`] first) — reads it positionally
|
pub(crate) fn family_entropy_under(genome_mask: &[u8], no_ambiguity: bool) -> Option<(f64, usize)> {
|
||||||
/// (`compact` advances only on non-monomorphic minorants, mirroring
|
if no_ambiguity {
|
||||||
/// exactly how the annex was written), never recomputes anything.
|
family_entropy_4(genome_mask)
|
||||||
pub(crate) fn iter_full_entropy(layer_dir: &Path) -> OKIResult<impl Iterator<Item = f32> + use<>> {
|
} else {
|
||||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
family_entropy(genome_mask)
|
||||||
let entropy_annex =
|
}
|
||||||
EntropyAnnex::open(&layer_dir.join(ENTROPY_ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
|
||||||
|
|
||||||
let mut compact = 0usize;
|
|
||||||
Ok((0..annex.len()).filter_map(move |slot| {
|
|
||||||
let mask = annex.get(slot)?;
|
|
||||||
if !mask.is_minorant() {
|
|
||||||
return None; // not a minorant at all — no entry in family_idx numbering either
|
|
||||||
}
|
|
||||||
if is_non_monomorphic_minorant(mask) {
|
|
||||||
let value = entropy_annex.get(compact).unwrap_or(0.0);
|
|
||||||
compact += 1;
|
|
||||||
Some(value)
|
|
||||||
} else {
|
|
||||||
Some(0.0) // monomorphic — trivially zero by convention, never stored
|
|
||||||
}
|
|
||||||
}))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -0,0 +1,212 @@
|
|||||||
|
//! Per-layer family annexes built from a `scan_layer_families` sweep: the
|
||||||
|
//! entropy annex and the pair-weight annex (both a
|
||||||
|
//! [`crate::siblings::FamilyScalarAnnex`]), the quartet annex (a
|
||||||
|
//! [`obicompactvec::PersistentBitVec`], see `crate::siblings::quartet_annex`)
|
||||||
|
//! and the index-level family-overlap accumulators. The entropy, quartet and
|
||||||
|
//! overlap builders all need the identical per-family `genome_mask`
|
||||||
|
//! resolution, so one sweep feeds every one that has something missing; the
|
||||||
|
//! pair weights need the *finished* overlap matrix, hence a second sweep
|
||||||
|
//! (`ensure_layer_pair_weight_annexes`).
|
||||||
|
|
||||||
|
use std::collections::HashSet;
|
||||||
|
use std::io;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
use obicompactvec::PersistentBitVecBuilder;
|
||||||
|
use obikidxcache::index_cache::IndexCache;
|
||||||
|
use obikindex::{OKIError, OKIResult};
|
||||||
|
|
||||||
|
use super::entropy::family_entropy_under;
|
||||||
|
use super::family_scan::{Selection, scan_layer_families};
|
||||||
|
use super::masking::is_quartet_informative;
|
||||||
|
use super::minorant_selection::non_monomorphic_reindex_layer;
|
||||||
|
use super::pair_weight::pair_balance_weight;
|
||||||
|
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
||||||
|
use crate::siblings::{FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind};
|
||||||
|
|
||||||
|
/// Which annexes a caller needs, each listed once per `--no-ambiguity`
|
||||||
|
/// counting convention (`false` = genomes carrying several members count as a
|
||||||
|
/// state of their own, `true` = they are left out) — an empty list asks for
|
||||||
|
/// none. A pair weight is computed from the family-overlap annex of its own
|
||||||
|
/// convention, so asking for one asks for the other.
|
||||||
|
#[derive(Clone, Debug, Default)]
|
||||||
|
pub(crate) struct AnnexRequest {
|
||||||
|
pub entropy: Vec<bool>,
|
||||||
|
pub quartet: Vec<bool>,
|
||||||
|
pub overlap: Vec<bool>,
|
||||||
|
pub pair_weight: Vec<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl AnnexRequest {
|
||||||
|
/// Everything under both conventions — what `--sibling-annex` precomputes.
|
||||||
|
pub(crate) fn everything() -> Self {
|
||||||
|
let both = vec![false, true];
|
||||||
|
Self { entropy: both.clone(), quartet: both.clone(), overlap: both.clone(), pair_weight: both }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The overlap conventions to ensure: the requested ones plus every one a
|
||||||
|
/// requested pair weight is computed from, each once.
|
||||||
|
pub(crate) fn overlap_conventions(&self) -> Vec<bool> {
|
||||||
|
dedup(self.overlap.iter().chain(&self.pair_weight).copied())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The pair-weight conventions requested, each once.
|
||||||
|
pub(crate) fn pair_weight_conventions(&self) -> Vec<bool> {
|
||||||
|
dedup(self.pair_weight.iter().copied())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn dedup(variants: impl IntoIterator<Item = bool>) -> Vec<bool> {
|
||||||
|
let mut v: Vec<bool> = variants.into_iter().collect();
|
||||||
|
v.sort_unstable();
|
||||||
|
v.dedup();
|
||||||
|
v
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds the entropy and quartet annexes of one layer that `request` asks
|
||||||
|
/// for and that don't exist yet, and feeds `overlap_accs` (one accumulator
|
||||||
|
/// per family-overlap annex still missing, see
|
||||||
|
/// [`crate::siblings::FamilyOverlap`]'s own module docs), in one
|
||||||
|
/// `scan_layer_families` pass. A no-op (no scan at all) when everything
|
||||||
|
/// requested already exists for this layer *and* `overlap_accs` is empty.
|
||||||
|
/// Bounded to non-monomorphic minorants only
|
||||||
|
/// ([`non_monomorphic_reindex_layer`], a cheap annex-only pass — no
|
||||||
|
/// cross-partition resolution) whenever a scan does happen: monomorphism is
|
||||||
|
/// already known from the annex bits alone, so there is no reason to pay
|
||||||
|
/// `scan_layer_families`'s expensive per-genome resolution for the ~98% of
|
||||||
|
/// minorants that are monomorphic only to discard the result.
|
||||||
|
pub(crate) fn ensure_layer_family_annexes(
|
||||||
|
cache: &IndexCache,
|
||||||
|
partition: usize,
|
||||||
|
layer_idx: usize,
|
||||||
|
n_genomes: usize,
|
||||||
|
fast_mode: bool,
|
||||||
|
request: &AnnexRequest,
|
||||||
|
overlap_accs: &mut [(bool, FamilyOverlapAccumulator)],
|
||||||
|
) -> OKIResult<()> {
|
||||||
|
let layer = cache
|
||||||
|
.get_layer(partition, layer_idx)
|
||||||
|
.expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)");
|
||||||
|
let missing = |name: &dyn Fn(bool) -> &'static str, variants: &[bool]| -> Vec<(bool, PathBuf)> {
|
||||||
|
dedup(variants.iter().copied())
|
||||||
|
.into_iter()
|
||||||
|
.map(|no_ambiguity| (no_ambiguity, layer.dir().join(name(no_ambiguity))))
|
||||||
|
.filter(|(_, path)| !path.exists())
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
let missing_entropy = missing(&|nb| ScalarKind::Entropy.file_name(nb), &request.entropy);
|
||||||
|
let missing_quartet = missing(&quartet_annex_file_name, &request.quartet);
|
||||||
|
if missing_entropy.is_empty() && missing_quartet.is_empty() && overlap_accs.is_empty() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let reindex = non_monomorphic_reindex_layer(layer.dir())?;
|
||||||
|
let eligible: HashSet<usize> = reindex.keys().copied().collect();
|
||||||
|
let mut entropy_builders = missing_entropy
|
||||||
|
.iter()
|
||||||
|
.map(|(no_ambiguity, path)| {
|
||||||
|
FamilyScalarAnnexBuilder::new(reindex.len(), path, ScalarKind::Entropy, *no_ambiguity)
|
||||||
|
.map(|b| (*no_ambiguity, b))
|
||||||
|
})
|
||||||
|
.collect::<io::Result<Vec<_>>>()
|
||||||
|
.map_err(OKIError::Io)?;
|
||||||
|
let mut quartet_builders = missing_quartet
|
||||||
|
.iter()
|
||||||
|
.map(|(no_ambiguity, path)| {
|
||||||
|
PersistentBitVecBuilder::new(reindex.len(), path).map(|b| (*no_ambiguity, b))
|
||||||
|
})
|
||||||
|
.collect::<io::Result<Vec<_>>>()
|
||||||
|
.map_err(OKIError::Io)?;
|
||||||
|
|
||||||
|
scan_layer_families(
|
||||||
|
cache,
|
||||||
|
partition,
|
||||||
|
layer_idx,
|
||||||
|
n_genomes,
|
||||||
|
fast_mode,
|
||||||
|
&Selection::Some(&eligible),
|
||||||
|
|family_idx, mask, genome_mask| {
|
||||||
|
debug_assert!(
|
||||||
|
mask.family_size() >= 2,
|
||||||
|
"Selection::Some(eligible) must only visit non-monomorphic minorants"
|
||||||
|
);
|
||||||
|
let compact = reindex[&family_idx];
|
||||||
|
for (no_ambiguity, builder) in entropy_builders.iter_mut() {
|
||||||
|
if let Some((h, _)) = family_entropy_under(genome_mask, *no_ambiguity) {
|
||||||
|
builder.set(compact, h as f32);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (no_ambiguity, builder) in quartet_builders.iter_mut() {
|
||||||
|
builder.set(compact, is_quartet_informative(genome_mask, *no_ambiguity));
|
||||||
|
}
|
||||||
|
for (_, acc) in overlap_accs.iter_mut() {
|
||||||
|
acc.add_family(genome_mask);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
for (_, builder) in entropy_builders {
|
||||||
|
builder.close().map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
|
for (_, builder) in quartet_builders {
|
||||||
|
builder.close().map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds the pair-weight annexes of one layer that don't exist yet, one per
|
||||||
|
/// `(no_ambiguity, inverse overlap matrix)` in `matrices` (see
|
||||||
|
/// [`super::pair_weight::inverse_overlap_matrix`]), in one scan. A no-op (no
|
||||||
|
/// scan) when they all exist. Needs the family-overlap annex of each
|
||||||
|
/// convention to be complete: run after the sweep that builds it.
|
||||||
|
pub(crate) fn ensure_layer_pair_weight_annexes(
|
||||||
|
cache: &IndexCache,
|
||||||
|
partition: usize,
|
||||||
|
layer_idx: usize,
|
||||||
|
n_genomes: usize,
|
||||||
|
fast_mode: bool,
|
||||||
|
matrices: &[(bool, Vec<f32>)],
|
||||||
|
) -> OKIResult<()> {
|
||||||
|
let layer = cache
|
||||||
|
.get_layer(partition, layer_idx)
|
||||||
|
.expect("cache-consistent: caller already bounds layer_idx by n_layer(partition)");
|
||||||
|
let pending: Vec<&(bool, Vec<f32>)> = matrices
|
||||||
|
.iter()
|
||||||
|
.filter(|(no_ambiguity, _)| !layer.dir().join(ScalarKind::PairWeight.file_name(*no_ambiguity)).exists())
|
||||||
|
.collect();
|
||||||
|
if pending.is_empty() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let reindex = non_monomorphic_reindex_layer(layer.dir())?;
|
||||||
|
let eligible: HashSet<usize> = reindex.keys().copied().collect();
|
||||||
|
let mut builders = pending
|
||||||
|
.iter()
|
||||||
|
.map(|(no_ambiguity, inverse)| {
|
||||||
|
let path = layer.dir().join(ScalarKind::PairWeight.file_name(*no_ambiguity));
|
||||||
|
FamilyScalarAnnexBuilder::new(reindex.len(), &path, ScalarKind::PairWeight, *no_ambiguity)
|
||||||
|
.map(|b| (*no_ambiguity, inverse, b))
|
||||||
|
})
|
||||||
|
.collect::<io::Result<Vec<_>>>()
|
||||||
|
.map_err(OKIError::Io)?;
|
||||||
|
|
||||||
|
scan_layer_families(
|
||||||
|
cache,
|
||||||
|
partition,
|
||||||
|
layer_idx,
|
||||||
|
n_genomes,
|
||||||
|
fast_mode,
|
||||||
|
&Selection::Some(&eligible),
|
||||||
|
|family_idx, _mask, genome_mask| {
|
||||||
|
let compact = reindex[&family_idx];
|
||||||
|
for (no_ambiguity, inverse, builder) in builders.iter_mut() {
|
||||||
|
builder.set(compact, pair_balance_weight(genome_mask, *no_ambiguity, inverse));
|
||||||
|
}
|
||||||
|
},
|
||||||
|
)?;
|
||||||
|
for (_, _, builder) in builders {
|
||||||
|
builder.close().map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -17,6 +17,8 @@
|
|||||||
//! waste sample quota on sites that die anyway once the excluded genome's
|
//! waste sample quota on sites that die anyway once the excluded genome's
|
||||||
//! data is dropped.
|
//! data is dropped.
|
||||||
|
|
||||||
|
use crate::siblings::helpers::is_carrier;
|
||||||
|
|
||||||
/// This genome's masked character state, or `None` for "?" (missing). `m`
|
/// This genome's masked character state, or `None` for "?" (missing). `m`
|
||||||
/// is one `genome_mask` entry (bit `b` set iff this genome carries the
|
/// is one `genome_mask` entry (bit `b` set iff this genome carries the
|
||||||
/// family member whose own canonical central base is `b`; `0` = no member
|
/// family member whose own canonical central base is `b`; `0` = no member
|
||||||
@@ -64,6 +66,97 @@ pub(crate) fn survives_masking(masked: &[Option<u8>], excluded: &[bool]) -> bool
|
|||||||
seen.count_ones() >= 2
|
seen.count_ones() >= 2
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether `counts` — the number of genomes per *called* state, indexed by
|
||||||
|
/// the `0..=15` state value, index `0` (absence) never counted — shows a
|
||||||
|
/// bipartition: at least two distinct states each carried by at least two
|
||||||
|
/// genomes. That is the pattern that can separate two pairs of genomes from
|
||||||
|
/// each other (`AB|CD`); a singleton variant, or a family carried by fewer
|
||||||
|
/// than four genomes, fits every topology and informs none.
|
||||||
|
fn has_bipartition(counts: &[u32; 16]) -> bool {
|
||||||
|
counts[1..].iter().filter(|&&c| c >= 2).count() >= 2
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Quartet-informativeness of a family from its raw per-genome masks alone,
|
||||||
|
/// under the `--no-ambiguity` counting convention: absent genomes (`0`)
|
||||||
|
/// never count, and with `no_ambiguity` a genome carrying several members
|
||||||
|
/// doesn't either — otherwise its combination of bases is a state of its
|
||||||
|
/// own. Ignores `--exclude-genome` and `--free-loss` on purpose: absence is
|
||||||
|
/// never a called state here, and removing genomes can only destroy a
|
||||||
|
/// bipartition, never create one — so `false` is final, and `true` is only
|
||||||
|
/// a necessary condition, re-checked exactly by [`survives_quartet`] once
|
||||||
|
/// the family is resolved. This is what the persisted quartet annex stores.
|
||||||
|
pub(crate) fn is_quartet_informative(genome_mask: &[u8], no_ambiguity: bool) -> bool {
|
||||||
|
let mut counts = [0u32; 16];
|
||||||
|
for &m in genome_mask {
|
||||||
|
if !is_carrier(m, no_ambiguity) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
counts[(m & 0b1111) as usize] += 1;
|
||||||
|
}
|
||||||
|
has_bipartition(&counts)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Exact quartet check on an already-[`masked_state`]d family, over the
|
||||||
|
/// *included* genomes (`excluded` as in [`survives_masking`]). Only called
|
||||||
|
/// states count: a genome whose masked state is `0` (absence kept as a real
|
||||||
|
/// state, i.e. no `--free-loss`) is not a carrier of anything.
|
||||||
|
pub(crate) fn survives_quartet(masked: &[Option<u8>], excluded: &[bool]) -> bool {
|
||||||
|
let mut counts = [0u32; 16];
|
||||||
|
for (g, &state) in masked.iter().enumerate() {
|
||||||
|
if excluded.get(g).copied().unwrap_or(false) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Some(state) = state {
|
||||||
|
counts[(state & 0b1111) as usize] += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
has_bipartition(&counts)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod quartet_tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn two_pairs_of_distinct_states_is_informative() {
|
||||||
|
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], false));
|
||||||
|
assert!(is_quartet_informative(&[0b0001, 0b0001, 0b0100, 0b0100], true));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn singleton_variant_is_not_informative() {
|
||||||
|
// three genomes in one state plus one lone variant: fits any topology
|
||||||
|
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0001, 0b0100], false));
|
||||||
|
// fewer than four carriers can never be informative
|
||||||
|
assert!(!is_quartet_informative(&[0b0001, 0b0100, 0, 0], false));
|
||||||
|
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0b0100], false));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn absent_genomes_are_not_a_state() {
|
||||||
|
// two carriers of one base, two absences: absence must not form the 2nd pair
|
||||||
|
assert!(!is_quartet_informative(&[0b0001, 0b0001, 0, 0], false));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_ambiguity_drops_multi_member_genomes_from_the_count() {
|
||||||
|
// the two-member genomes form a pair of their own state without the flag
|
||||||
|
let masks = [0b0001u8, 0b0001, 0b0011, 0b0011];
|
||||||
|
assert!(is_quartet_informative(&masks, false));
|
||||||
|
assert!(!is_quartet_informative(&masks, true));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn exact_check_honours_exclusions_and_missing() {
|
||||||
|
let masked = [Some(1u8), Some(1), Some(4), Some(4), None, Some(0)];
|
||||||
|
assert!(survives_quartet(&masked, &[]));
|
||||||
|
// excluding one genome of the second pair breaks the bipartition
|
||||||
|
assert!(!survives_quartet(&masked, &[false, false, true]));
|
||||||
|
// an absence kept as state 0 (no --free-loss) is not a called state
|
||||||
|
assert!(!survives_quartet(&[Some(1), Some(1), Some(0), Some(0)], &[]));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// IUPAC ambiguity code for a [`masked_state`] value (`0..=15`): single bit
|
/// IUPAC ambiguity code for a [`masked_state`] value (`0..=15`): single bit
|
||||||
/// -> the plain base; 2 or 3 bits -> the matching IUPAC ambiguity code
|
/// -> the plain base; 2 or 3 bits -> the matching IUPAC ambiguity code
|
||||||
/// (preserves partial information instead of collapsing to `N`, the same
|
/// (preserves partial information instead of collapsing to `N`, the same
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ use std::path::Path;
|
|||||||
use obicompactvec::TempBitVecBuilder;
|
use obicompactvec::TempBitVecBuilder;
|
||||||
use obikindex::{OKIError, OKIResult};
|
use obikindex::{OKIError, OKIResult};
|
||||||
|
|
||||||
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, SiblingAnnex};
|
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
|
||||||
|
|
||||||
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
|
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
|
||||||
mask.is_minorant() && mask.siblings() >= 1
|
mask.is_minorant() && mask.siblings() >= 1
|
||||||
@@ -82,3 +82,49 @@ pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBui
|
|||||||
}
|
}
|
||||||
Ok(eligible)
|
Ok(eligible)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Streams one stored `f32` per minorant of the layer, **in `family_idx`
|
||||||
|
/// order** — monomorphic minorants included, valued `0.0` by convention
|
||||||
|
/// (trivially uninformative, never computed or stored in the compact
|
||||||
|
/// annex). Exists so a caller can `.zip()` this directly against any other
|
||||||
|
/// `family_idx`-ordered stream (e.g. `SiblingLayerExt::iter_minorants`)
|
||||||
|
/// instead of separately consulting [`non_monomorphic_reindex_layer`]'s map
|
||||||
|
/// at each site.
|
||||||
|
///
|
||||||
|
/// Reads the `kind` annex (entropy, pair weight) built under the
|
||||||
|
/// `no_ambiguity` counting convention; it must already exist
|
||||||
|
/// (`ensure_layer_family_annexes` / `ensure_layer_pair_weight_annexes`
|
||||||
|
/// first) — read positionally (`compact` advances only on non-monomorphic
|
||||||
|
/// minorants, mirroring exactly how the annex was written), never
|
||||||
|
/// recomputed. A file whose header records the other convention is rejected
|
||||||
|
/// rather than silently read as this one.
|
||||||
|
pub(crate) fn iter_full_scalar(
|
||||||
|
layer_dir: &Path,
|
||||||
|
kind: ScalarKind,
|
||||||
|
no_ambiguity: bool,
|
||||||
|
) -> OKIResult<impl Iterator<Item = f32> + use<>> {
|
||||||
|
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
||||||
|
let scalars =
|
||||||
|
FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?;
|
||||||
|
if scalars.no_ambiguity() != no_ambiguity {
|
||||||
|
return Err(OKIError::Io(std::io::Error::new(
|
||||||
|
std::io::ErrorKind::InvalidData,
|
||||||
|
format!("{kind:?} annex was built under the other --no-ambiguity convention"),
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut compact = 0usize;
|
||||||
|
Ok((0..annex.len()).filter_map(move |slot| {
|
||||||
|
let mask = annex.get(slot)?;
|
||||||
|
if !mask.is_minorant() {
|
||||||
|
return None; // not a minorant at all — no entry in family_idx numbering either
|
||||||
|
}
|
||||||
|
if is_non_monomorphic_minorant(mask) {
|
||||||
|
let value = scalars.get(compact).unwrap_or(0.0);
|
||||||
|
compact += 1;
|
||||||
|
Some(value)
|
||||||
|
} else {
|
||||||
|
Some(0.0) // monomorphic — trivially zero by convention, never stored
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
}
|
||||||
@@ -17,10 +17,13 @@ mod alignment;
|
|||||||
mod annex;
|
mod annex;
|
||||||
mod cardcomp;
|
mod cardcomp;
|
||||||
mod entropy;
|
mod entropy;
|
||||||
|
mod family_annexes;
|
||||||
mod family_scan;
|
mod family_scan;
|
||||||
mod masking;
|
mod masking;
|
||||||
mod minorant_selection;
|
mod minorant_selection;
|
||||||
|
mod pair_weight;
|
||||||
mod pairwise;
|
mod pairwise;
|
||||||
|
mod quartet;
|
||||||
mod sankoff;
|
mod sankoff;
|
||||||
mod scan;
|
mod scan;
|
||||||
mod snp_distance;
|
mod snp_distance;
|
||||||
@@ -38,10 +41,15 @@ pub use sankoff::SankoffBundle;
|
|||||||
pub use scan::{LayerReducer, sampled_scan};
|
pub use scan::{LayerReducer, sampled_scan};
|
||||||
pub use snp_distance::{GammaShape, SnpDistanceKind, distance_matrix};
|
pub use snp_distance::{GammaShape, SnpDistanceKind, distance_matrix};
|
||||||
pub use stats::SiblingAnnexStats;
|
pub use stats::SiblingAnnexStats;
|
||||||
pub use subsample::{EntropyBias, SurvivingFamily};
|
pub use subsample::{EntropyBias, SamplingSpec, SurvivingFamily};
|
||||||
|
|
||||||
pub(crate) use annex::build_layer_sibling_annex;
|
pub(crate) use annex::build_layer_sibling_annex;
|
||||||
pub(crate) use entropy::{ensure_layer_entropy_annex, family_entropy, family_entropy_4};
|
pub(crate) use entropy::{family_entropy, family_entropy_4};
|
||||||
|
pub(crate) use family_annexes::{
|
||||||
|
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes,
|
||||||
|
};
|
||||||
|
pub(crate) use pair_weight::inverse_overlap_matrix;
|
||||||
|
pub(crate) use quartet::restrict_to_quartet;
|
||||||
pub(crate) use family_scan::{Selection, scan_layer_families};
|
pub(crate) use family_scan::{Selection, scan_layer_families};
|
||||||
pub(crate) use minorant_selection::non_monomorphic_reindex_layer;
|
pub(crate) use minorant_selection::non_monomorphic_reindex_layer;
|
||||||
pub(crate) use stats::{sibling_annex_stats, sibling_family_size_histogram};
|
pub(crate) use stats::{sibling_annex_stats, sibling_family_size_histogram};
|
||||||
|
|||||||
@@ -0,0 +1,121 @@
|
|||||||
|
//! `--pair-balance`: a per-family sampling weight that favours the families
|
||||||
|
//! covering the pairs of genomes the index shares the fewest families between.
|
||||||
|
//!
|
||||||
|
//! The weight of a family is the **mean**, over the pairs of genomes carrying
|
||||||
|
//! it, of `1 / O(pair)`, where `O` is the index-wide number of variable
|
||||||
|
//! families the pair shares (the family-overlap annex, under the same
|
||||||
|
//! `--no-ambiguity` counting convention). It is the mean, not the sum: a sum
|
||||||
|
//! grows with the square of the number of carriers, so it would chase the
|
||||||
|
//! families carried by most genomes — in a skimming dataset typically
|
||||||
|
//! high-copy regions, sampled deeply for reasons unrelated to phylogeny —
|
||||||
|
//! instead of the ones covering rare pairs.
|
||||||
|
//!
|
||||||
|
//! Values are only meaningful relative to each other; the sampler rescales
|
||||||
|
//! them per layer (`SamplingSpec::pair_balance`).
|
||||||
|
|
||||||
|
use crate::siblings::FamilyOverlap;
|
||||||
|
use crate::siblings::helpers::is_carrier;
|
||||||
|
|
||||||
|
/// `1 / O` for every pair of genomes, as a dense symmetric `n × n` matrix
|
||||||
|
/// (diagonal and pairs sharing nothing hold `0.0`) — what
|
||||||
|
/// [`pair_balance_weight`] reads, built once per run from the persisted
|
||||||
|
/// family-overlap annex.
|
||||||
|
pub(crate) fn inverse_overlap_matrix(overlap: &FamilyOverlap) -> Vec<f32> {
|
||||||
|
let n = overlap.n_genomes();
|
||||||
|
let mut inv = vec![0.0f32; n * n];
|
||||||
|
for i in 0..n {
|
||||||
|
for j in (i + 1)..n {
|
||||||
|
let shared = overlap.get(i, j);
|
||||||
|
if shared > 0 {
|
||||||
|
let r = 1.0 / shared as f32;
|
||||||
|
inv[i * n + j] = r;
|
||||||
|
inv[j * n + i] = r;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
inv
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The pair-balance weight of one family from its raw per-genome masks:
|
||||||
|
/// mean of `inv_overlap` over the pairs of carriers (carriers per
|
||||||
|
/// [`is_carrier`], under `no_ambiguity`). `0.0` for a family with fewer than
|
||||||
|
/// two carriers — it covers no pair, so it can balance none.
|
||||||
|
pub(crate) fn pair_balance_weight(
|
||||||
|
genome_mask: &[u8],
|
||||||
|
no_ambiguity: bool,
|
||||||
|
inv_overlap: &[f32],
|
||||||
|
) -> f32 {
|
||||||
|
let n = genome_mask.len();
|
||||||
|
let carriers: Vec<usize> =
|
||||||
|
(0..n).filter(|&g| is_carrier(genome_mask[g], no_ambiguity)).collect();
|
||||||
|
let k = carriers.len();
|
||||||
|
if k < 2 {
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
let mut sum = 0.0f64;
|
||||||
|
for (a, &i) in carriers.iter().enumerate() {
|
||||||
|
for &j in &carriers[a + 1..] {
|
||||||
|
sum += f64::from(inv_overlap[i * n + j]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
(sum / (k * (k - 1) / 2) as f64) as f32
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
/// 4 genomes; pair (0,1) shares 2 families, (0,2) shares 4, the others 1.
|
||||||
|
fn matrix() -> Vec<f32> {
|
||||||
|
let n = 4;
|
||||||
|
let mut m = vec![0.0f32; n * n];
|
||||||
|
let mut set = |i: usize, j: usize, shared: f32| {
|
||||||
|
m[i * n + j] = 1.0 / shared;
|
||||||
|
m[j * n + i] = 1.0 / shared;
|
||||||
|
};
|
||||||
|
set(0, 1, 2.0);
|
||||||
|
set(0, 2, 4.0);
|
||||||
|
for (i, j) in [(0, 3), (1, 2), (1, 3), (2, 3)] {
|
||||||
|
set(i, j, 1.0);
|
||||||
|
}
|
||||||
|
m
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn single_pair_weight_is_the_inverse_overlap() {
|
||||||
|
let w = pair_balance_weight(&[1, 1, 0, 0], false, &matrix());
|
||||||
|
assert!((w - 0.5).abs() < 1e-6);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn weight_is_the_mean_not_the_sum_over_pairs() {
|
||||||
|
// carriers 0, 1, 2: pairs (0,1)=1/2, (0,2)=1/4, (1,2)=1 -> mean 7/12
|
||||||
|
let w = pair_balance_weight(&[1, 1, 1, 0], false, &matrix());
|
||||||
|
assert!((w - 7.0 / 12.0).abs() < 1e-6);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn fewer_than_two_carriers_weigh_nothing() {
|
||||||
|
assert_eq!(pair_balance_weight(&[1, 0, 0, 0], false, &matrix()), 0.0);
|
||||||
|
assert_eq!(pair_balance_weight(&[0, 0, 0, 0], false, &matrix()), 0.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_ambiguity_leaves_multi_member_genomes_out_of_the_pairs() {
|
||||||
|
let masks = [1u8, 1, 0b0011, 0];
|
||||||
|
// without the flag genome 2 carries the family: pairs (0,1), (0,2), (1,2)
|
||||||
|
let with = pair_balance_weight(&masks, false, &matrix());
|
||||||
|
// with it, only the pair (0,1) remains
|
||||||
|
let without = pair_balance_weight(&masks, true, &matrix());
|
||||||
|
assert!((with - 7.0 / 12.0).abs() < 1e-6);
|
||||||
|
assert!((without - 0.5).abs() < 1e-6);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_rare_pair_outweighs_a_common_one() {
|
||||||
|
// genomes (0,2) share many families, (1,3) few
|
||||||
|
let common = pair_balance_weight(&[1, 0, 1, 0], false, &matrix());
|
||||||
|
let rare = pair_balance_weight(&[0, 1, 0, 1], false, &matrix());
|
||||||
|
assert!(rare > common);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
//! `--quartet`: restricting a layer's eligible families to the
|
||||||
|
//! quartet-informative ones, from the persisted quartet annex.
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use obicompactvec::{PersistentBitVec, TempBitVecBuilder};
|
||||||
|
use obikindex::{OKIError, OKIResult};
|
||||||
|
|
||||||
|
use super::minorant_selection::is_non_monomorphic_minorant;
|
||||||
|
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
||||||
|
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
|
||||||
|
|
||||||
|
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
|
||||||
|
/// included), every family the quartet annex flags as not quartet-informative
|
||||||
|
/// under the `no_ambiguity` counting convention. The annex is indexed by the
|
||||||
|
/// compact non-monomorphic ordinal, so the sibling annex is walked in the same
|
||||||
|
/// order to translate one numbering into the other — annex-only, no
|
||||||
|
/// cross-partition resolution. The annex must already exist
|
||||||
|
/// ([`super::family_annexes::ensure_layer_family_annexes`] first).
|
||||||
|
pub(crate) fn restrict_to_quartet(
|
||||||
|
eligible: &mut TempBitVecBuilder,
|
||||||
|
layer_dir: &Path,
|
||||||
|
no_ambiguity: bool,
|
||||||
|
) -> OKIResult<()> {
|
||||||
|
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
||||||
|
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity)))
|
||||||
|
.map_err(OKIError::Io)?;
|
||||||
|
|
||||||
|
let mut family_idx = 0usize;
|
||||||
|
let mut compact = 0usize;
|
||||||
|
for slot in 0..annex.len() {
|
||||||
|
let Some(mask) = annex.get(slot) else { continue };
|
||||||
|
if !mask.is_minorant() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if is_non_monomorphic_minorant(mask) {
|
||||||
|
if !flags.get(compact) {
|
||||||
|
eligible.set(family_idx, false);
|
||||||
|
}
|
||||||
|
compact += 1;
|
||||||
|
}
|
||||||
|
family_idx += 1;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
@@ -14,7 +14,7 @@ use obikidxcache::index_cache::IndexCache;
|
|||||||
use obikindex::OKIResult;
|
use obikindex::OKIResult;
|
||||||
|
|
||||||
use super::stats::sibling_family_size_histogram;
|
use super::stats::sibling_family_size_histogram;
|
||||||
use super::subsample::{EntropyBias, SurvivingFamily, sample_index};
|
use super::subsample::{SamplingSpec, SurvivingFamily, sample_index};
|
||||||
|
|
||||||
/// One accumulator fed every layer's [`SurvivingFamily`] batch during a
|
/// One accumulator fed every layer's [`SurvivingFamily`] batch during a
|
||||||
/// [`sampled_scan`] pass — self-contained (no sampling/masking logic of
|
/// [`sampled_scan`] pass — self-contained (no sampling/masking logic of
|
||||||
@@ -42,26 +42,24 @@ fn resolve_target(cache: &IndexCache, n: Option<usize>) -> OKIResult<usize> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Run one shared, possibly-subsampled/entropy-biased selection over
|
/// Run one shared, possibly-subsampled/entropy-biased/quartet-restricted
|
||||||
/// `cache` (`n`/`free_loss`/`no_ambiguity`/`excluded`/`entropy_bias` — see
|
/// selection over `cache` (`n`/`spec`/`excluded` — see [`sample_index`]'s
|
||||||
/// [`sample_index`]'s own docs for what each controls), feeding every
|
/// own docs and [`SamplingSpec`] for what each controls), feeding every
|
||||||
/// surviving family of every visited layer to every reducer in
|
/// surviving family of every visited layer to every reducer in
|
||||||
/// `reducers`, in the same order, before moving to the next layer. A
|
/// `reducers`, in the same order, before moving to the next layer. A
|
||||||
/// no-op (not an error) if the resolved target is `0`.
|
/// no-op (not an error) if the resolved target is `0`.
|
||||||
pub fn sampled_scan(
|
pub fn sampled_scan(
|
||||||
cache: &IndexCache,
|
cache: &IndexCache,
|
||||||
n: Option<usize>,
|
n: Option<usize>,
|
||||||
free_loss: bool,
|
spec: SamplingSpec,
|
||||||
no_ambiguity: bool,
|
|
||||||
excluded: &[bool],
|
excluded: &[bool],
|
||||||
entropy_bias: Option<EntropyBias>,
|
|
||||||
reducers: &mut [&mut dyn LayerReducer],
|
reducers: &mut [&mut dyn LayerReducer],
|
||||||
) -> OKIResult<()> {
|
) -> OKIResult<()> {
|
||||||
let target = resolve_target(cache, n)?;
|
let target = resolve_target(cache, n)?;
|
||||||
if target == 0 {
|
if target == 0 {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
sample_index(cache, target, free_loss, no_ambiguity, excluded, entropy_bias, 0, |partition, layer, _raw_index, survivors| {
|
sample_index(cache, target, spec, excluded, 0, |partition, layer, _raw_index, survivors| {
|
||||||
for reducer in reducers.iter_mut() {
|
for reducer in reducers.iter_mut() {
|
||||||
reducer.reduce(partition, layer, &survivors);
|
reducer.reduce(partition, layer, &survivors);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -51,12 +51,11 @@ use obikidxcache::index_cache::IndexCache;
|
|||||||
use obikindex::OKIResult;
|
use obikindex::OKIResult;
|
||||||
use rand::RngExt;
|
use rand::RngExt;
|
||||||
|
|
||||||
use super::entropy::iter_full_entropy;
|
|
||||||
use super::family_scan::{Selection, scan_layer_families};
|
use super::family_scan::{Selection, scan_layer_families};
|
||||||
use super::is_fast_mode;
|
use super::masking::{masked_state, survives_masking, survives_quartet};
|
||||||
use super::masking::{masked_state, survives_masking};
|
use super::minorant_selection::{eligible_bitset_layer, iter_full_scalar};
|
||||||
use super::minorant_selection::eligible_bitset_layer;
|
use super::{AnnexRequest, is_fast_mode, restrict_to_quartet};
|
||||||
use crate::siblings::FamilyMask;
|
use crate::siblings::{FamilyMask, ScalarKind};
|
||||||
use crate::siblings::extensions::SiblingBuilder;
|
use crate::siblings::extensions::SiblingBuilder;
|
||||||
|
|
||||||
/// Resolution rounds a single layer's sampling may take before giving up on
|
/// Resolution rounds a single layer's sampling may take before giving up on
|
||||||
@@ -96,21 +95,61 @@ pub struct EntropyBias {
|
|||||||
pub sigma: f64,
|
pub sigma: f64,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// [`iter_full_entropy`], rotated to start at `start` and wrap once back to
|
/// How a draw is conditioned: the character masking (`--free-loss`,
|
||||||
/// it — the same `family_idx` order [`sample_layer`]'s own circular index
|
/// `--no-ambiguity`), the quartet restriction (`--quartet`: only families
|
||||||
/// sequence walks, so the two can be `.zip()`-ed directly. Two more streamed
|
/// that can separate two pairs of genomes) and the optional entropy bias.
|
||||||
/// passes over the layer's (small) annex/entropy files, not a materialised
|
/// `--no-ambiguity` also selects the counting convention of the persisted
|
||||||
/// `Vec`/`HashMap` — `iter_full_entropy` itself never allocates more than
|
/// entropy and quartet annexes the draw reads, so the draw and the alignment
|
||||||
/// its own iterator state.
|
/// it feeds always describe the same genomes. `pub`, not `pub(crate)`: part
|
||||||
fn circular_entropy_values(
|
/// of the public signature of [`super::scan::sampled_scan`].
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
pub struct SamplingSpec {
|
||||||
|
pub free_loss: bool,
|
||||||
|
pub no_ambiguity: bool,
|
||||||
|
pub quartet: bool,
|
||||||
|
/// `--pair-balance`: weight each family by the mean, over the pairs of
|
||||||
|
/// genomes carrying it, of 1 / (number of families the pair shares), so
|
||||||
|
/// the draw favours the families covering the rarest pairs (see
|
||||||
|
/// `algorithms::pair_weight`).
|
||||||
|
pub pair_balance: bool,
|
||||||
|
pub entropy_bias: Option<EntropyBias>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// [`iter_full_scalar`] of `kind` (entropy, pair weight), rotated to start at
|
||||||
|
/// `start` and wrap once back to it — the same `family_idx` order
|
||||||
|
/// [`sample_layer`]'s own circular index sequence walks, so the streams can be
|
||||||
|
/// `.zip()`-ed directly. Two more streamed passes over the layer's (small)
|
||||||
|
/// annex files, not a materialised `Vec`/`HashMap` — `iter_full_scalar`
|
||||||
|
/// itself never allocates more than its own iterator state. `no_ambiguity`
|
||||||
|
/// selects the annex built under the same counting convention as the masking
|
||||||
|
/// the survivors go through afterwards.
|
||||||
|
fn circular_scalar_values(
|
||||||
layer_dir: &Path,
|
layer_dir: &Path,
|
||||||
|
kind: ScalarKind,
|
||||||
start: usize,
|
start: usize,
|
||||||
|
no_ambiguity: bool,
|
||||||
) -> OKIResult<impl Iterator<Item = f32> + use<>> {
|
) -> OKIResult<impl Iterator<Item = f32> + use<>> {
|
||||||
let head = iter_full_entropy(layer_dir)?.skip(start);
|
let head = iter_full_scalar(layer_dir, kind, no_ambiguity)?.skip(start);
|
||||||
let tail = iter_full_entropy(layer_dir)?.take(start);
|
let tail = iter_full_scalar(layer_dir, kind, no_ambiguity)?.take(start);
|
||||||
Ok(head.chain(tail))
|
Ok(head.chain(tail))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// One layer's place in a draw, planned before any layer is sampled: which of
|
||||||
|
/// its families may be drawn, and how much of the index-wide quota it
|
||||||
|
/// deserves.
|
||||||
|
struct LayerPlan {
|
||||||
|
partition: usize,
|
||||||
|
layer: usize,
|
||||||
|
/// One bit per minorant (`family_idx` order): still eligible to be drawn.
|
||||||
|
eligible: TempBitVecBuilder,
|
||||||
|
n_minorants: usize,
|
||||||
|
/// Total sampling weight of the eligible families: their number, or, with
|
||||||
|
/// `--pair-balance`, the sum of their pair weights. A layer's quota is its
|
||||||
|
/// share of this mass over every layer, so a layer rich in families that
|
||||||
|
/// cover rare pairs gets more of the draw.
|
||||||
|
mass: f64,
|
||||||
|
}
|
||||||
|
|
||||||
/// Samples up to `n` non-monomorphic families index-wide, proportionally
|
/// Samples up to `n` non-monomorphic families index-wide, proportionally
|
||||||
/// per layer, calling `on_layer(partition, layer, survivors)` once per
|
/// per layer, calling `on_layer(partition, layer, survivors)` once per
|
||||||
/// cached layer that contributed anything, with every family from that
|
/// cached layer that contributed anything, with every family from that
|
||||||
@@ -159,10 +198,8 @@ fn circular_entropy_values(
|
|||||||
pub(crate) fn sample_index(
|
pub(crate) fn sample_index(
|
||||||
cache: &IndexCache,
|
cache: &IndexCache,
|
||||||
n: usize,
|
n: usize,
|
||||||
free_loss: bool,
|
spec: SamplingSpec,
|
||||||
no_ambiguity: bool,
|
|
||||||
excluded: &[bool],
|
excluded: &[bool],
|
||||||
entropy_bias: Option<EntropyBias>,
|
|
||||||
skip_layers: usize,
|
skip_layers: usize,
|
||||||
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
|
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
|
||||||
) -> OKIResult<usize> {
|
) -> OKIResult<usize> {
|
||||||
@@ -170,64 +207,79 @@ pub(crate) fn sample_index(
|
|||||||
let fast_mode = is_fast_mode(cache);
|
let fast_mode = is_fast_mode(cache);
|
||||||
|
|
||||||
// Entropy-biased mode reads each candidate's entropy from the entropy
|
// Entropy-biased mode reads each candidate's entropy from the entropy
|
||||||
// annex (`EntropyWeights::open`, per layer, below) — build any missing
|
// annex, `--quartet` restricts the candidates through the quartet annex,
|
||||||
// one now, once, rather than mid-walk.
|
// and `--pair-balance` weights them by the pair-weight annex (all per
|
||||||
if entropy_bias.is_some() {
|
// layer, below, each under this run's `--no-ambiguity` counting
|
||||||
cache.ensure_entropy_and_overlap_annexes()?;
|
// convention) — build any missing one now, once, rather than mid-walk.
|
||||||
|
let request = AnnexRequest {
|
||||||
|
entropy: spec.entropy_bias.map(|_| vec![spec.no_ambiguity]).unwrap_or_default(),
|
||||||
|
quartet: if spec.quartet { vec![spec.no_ambiguity] } else { Vec::new() },
|
||||||
|
overlap: Vec::new(),
|
||||||
|
pair_weight: if spec.pair_balance { vec![spec.no_ambiguity] } else { Vec::new() },
|
||||||
|
};
|
||||||
|
if !request.entropy.is_empty() || !request.quartet.is_empty() || !request.pair_weight.is_empty() {
|
||||||
|
cache.ensure_family_annexes(&request)?;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Every cached layer's eligibility bitset, built up front — annex-only,
|
// Every cached layer's plan, built up front — annex-only, no
|
||||||
// no cross-partition resolution, so cheap even summed over the whole
|
// cross-partition resolution, so cheap even summed over the whole index
|
||||||
// index — needed before any layer's quota can be computed (each
|
// — needed before any layer's quota can be computed (each layer's share
|
||||||
// layer's share depends on every other layer's own count).
|
// depends on every other layer's own mass).
|
||||||
let mut layers: Vec<(usize, usize, TempBitVecBuilder, usize)> = Vec::new();
|
let mut plans: Vec<LayerPlan> = Vec::new();
|
||||||
for part in cache.partitions() {
|
for partition in cache.partitions() {
|
||||||
let n_layer = cache.n_layer(part).unwrap_or(0);
|
let n_layer = cache.n_layer(partition).unwrap_or(0);
|
||||||
for l in 0..n_layer {
|
for layer_idx in 0..n_layer {
|
||||||
let layer = cache
|
let layer = cache
|
||||||
.get_layer(part, l)
|
.get_layer(partition, layer_idx)
|
||||||
.expect("cache-consistent: n_layer(part) already bounds l");
|
.expect("cache-consistent: n_layer(partition) already bounds layer_idx");
|
||||||
let eligible = eligible_bitset_layer(layer.dir())?;
|
let mut eligible = eligible_bitset_layer(layer.dir())?;
|
||||||
|
if spec.quartet {
|
||||||
|
restrict_to_quartet(&mut eligible, layer.dir(), spec.no_ambiguity)?;
|
||||||
|
}
|
||||||
let n_minorants = eligible.view().len();
|
let n_minorants = eligible.view().len();
|
||||||
layers.push((part, l, eligible, n_minorants));
|
let mass = if spec.pair_balance {
|
||||||
|
iter_full_scalar(layer.dir(), ScalarKind::PairWeight, spec.no_ambiguity)?
|
||||||
|
.enumerate()
|
||||||
|
.filter(|&(family_idx, _)| eligible.view().get(family_idx))
|
||||||
|
.map(|(_, weight)| f64::from(weight))
|
||||||
|
.sum()
|
||||||
|
} else {
|
||||||
|
eligible.view().count_ones() as f64
|
||||||
|
};
|
||||||
|
plans.push(LayerPlan { partition, layer: layer_idx, eligible, n_minorants, mass });
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let total_eligible: u64 = layers
|
let total_mass: f64 = plans.iter().map(|plan| plan.mass).sum();
|
||||||
.iter()
|
if total_mass <= 0.0 {
|
||||||
.map(|(_, _, e, _)| e.view().count_ones())
|
|
||||||
.sum();
|
|
||||||
if total_eligible == 0 {
|
|
||||||
return Ok(0);
|
return Ok(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Quotas by cumulative rounding: layer `i` is allotted
|
||||||
|
// `floor(n × cumulative mass up to i / total) − what the previous layers
|
||||||
|
// were allotted`, so the quotas sum to exactly `n`. Rounding each layer's
|
||||||
|
// share down on its own would lose up to one site per layer — all of
|
||||||
|
// them, when `n` is smaller than the number of layers.
|
||||||
let mut total_kept = 0usize;
|
let mut total_kept = 0usize;
|
||||||
for (raw_index, (part, l, mut eligible, n_minorants)) in layers.into_iter().enumerate() {
|
let mut cumulative_mass = 0.0f64;
|
||||||
if raw_index < skip_layers {
|
let mut allotted = 0usize;
|
||||||
continue;
|
for (raw_index, mut plan) in plans.into_iter().enumerate() {
|
||||||
}
|
cumulative_mass += plan.mass;
|
||||||
let count_layer = eligible.view().count_ones();
|
let cumulative_quota = (n as f64 * cumulative_mass / total_mass) as usize;
|
||||||
if count_layer == 0 {
|
let quota = cumulative_quota.saturating_sub(allotted);
|
||||||
continue;
|
allotted = allotted.max(cumulative_quota);
|
||||||
}
|
if raw_index < skip_layers || plan.mass <= 0.0 || quota == 0 {
|
||||||
let quota = ((n as u128 * count_layer as u128) / total_eligible as u128) as usize;
|
|
||||||
if quota == 0 {
|
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
total_kept += sample_layer(
|
total_kept += sample_layer(
|
||||||
cache,
|
cache,
|
||||||
part,
|
|
||||||
l,
|
|
||||||
raw_index,
|
raw_index,
|
||||||
&mut eligible,
|
&mut plan,
|
||||||
n_minorants,
|
|
||||||
quota,
|
quota,
|
||||||
n_genomes,
|
n_genomes,
|
||||||
fast_mode,
|
fast_mode,
|
||||||
free_loss,
|
spec,
|
||||||
no_ambiguity,
|
|
||||||
excluded,
|
excluded,
|
||||||
entropy_bias,
|
|
||||||
&mut on_layer,
|
&mut on_layer,
|
||||||
)?;
|
)?;
|
||||||
}
|
}
|
||||||
@@ -236,32 +288,28 @@ pub(crate) fn sample_index(
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// One layer's share of [`sample_index`] — see the module docs for the
|
/// One layer's share of [`sample_index`] — see the module docs for the
|
||||||
/// per-family accept/reject/mark rules and the round-batching scheme.
|
/// per-family accept/reject/mark rules and the round-batching scheme. Each
|
||||||
#[allow(clippy::too_many_arguments)]
|
/// eligible family is accepted with probability
|
||||||
|
/// `quota / mass × pair weight × entropy kernel` (capped at 1): the pair
|
||||||
|
/// weight is `1` and `mass` the number of eligible families without
|
||||||
|
/// `--pair-balance`, and the kernel `1` without an entropy bias.
|
||||||
fn sample_layer(
|
fn sample_layer(
|
||||||
cache: &IndexCache,
|
cache: &IndexCache,
|
||||||
partition: usize,
|
|
||||||
layer_idx: usize,
|
|
||||||
raw_index: usize,
|
raw_index: usize,
|
||||||
eligible: &mut TempBitVecBuilder,
|
plan: &mut LayerPlan,
|
||||||
n_minorants: usize,
|
|
||||||
quota: usize,
|
quota: usize,
|
||||||
n_genomes: usize,
|
n_genomes: usize,
|
||||||
fast_mode: bool,
|
fast_mode: bool,
|
||||||
free_loss: bool,
|
spec: SamplingSpec,
|
||||||
no_ambiguity: bool,
|
|
||||||
excluded: &[bool],
|
excluded: &[bool],
|
||||||
entropy_bias: Option<EntropyBias>,
|
|
||||||
on_layer: &mut impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
|
on_layer: &mut impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
|
||||||
) -> OKIResult<usize> {
|
) -> OKIResult<usize> {
|
||||||
if n_minorants == 0 {
|
let LayerPlan { partition, layer: layer_idx, n_minorants, mass, .. } = *plan;
|
||||||
|
let eligible = &mut plan.eligible;
|
||||||
|
if n_minorants == 0 || mass <= 0.0 {
|
||||||
return Ok(0);
|
return Ok(0);
|
||||||
}
|
}
|
||||||
let count_layer = eligible.view().count_ones() as usize;
|
let p0 = quota as f64 / mass;
|
||||||
if count_layer == 0 {
|
|
||||||
return Ok(0);
|
|
||||||
}
|
|
||||||
let p0 = (quota as f64 / count_layer as f64).min(1.0);
|
|
||||||
let layer_dir = cache
|
let layer_dir = cache
|
||||||
.get_layer(partition, layer_idx)
|
.get_layer(partition, layer_idx)
|
||||||
.expect("cache-consistent")
|
.expect("cache-consistent")
|
||||||
@@ -301,34 +349,50 @@ fn sample_layer(
|
|||||||
|
|
||||||
// This round's own circular walk, resuming exactly where the
|
// This round's own circular walk, resuming exactly where the
|
||||||
// previous round left off (`start + visited`, wrapped) — zipped
|
// previous round left off (`start + visited`, wrapped) — zipped
|
||||||
// against that same rotation of the entropy stream when biased, so
|
// against that same rotation of the entropy and pair-weight streams
|
||||||
// each candidate's own kernel weight is read in lockstep, never via
|
// when they are used, so each candidate's own kernel and weight are
|
||||||
// a separate lookup structure. Unbiased draws skip the entropy
|
// read in lockstep, never via a separate lookup structure. A draw
|
||||||
// stream entirely (a flat `1.0` kernel), never touching the entropy
|
// without the corresponding option skips its stream entirely (flat
|
||||||
// annex.
|
// `1.0`), never touching the annex.
|
||||||
let round_start = (start + visited) % n_minorants;
|
let round_start = (start + visited) % n_minorants;
|
||||||
let entropies: Box<dyn Iterator<Item = f32>> = match entropy_bias {
|
let entropies: Box<dyn Iterator<Item = f32>> = match spec.entropy_bias {
|
||||||
Some(_) => Box::new(circular_entropy_values(&layer_dir, round_start)?),
|
Some(_) => Box::new(circular_scalar_values(
|
||||||
|
&layer_dir,
|
||||||
|
ScalarKind::Entropy,
|
||||||
|
round_start,
|
||||||
|
spec.no_ambiguity,
|
||||||
|
)?),
|
||||||
None => Box::new(std::iter::repeat(0.0f32)), // never read: kernel forced to 1.0 below
|
None => Box::new(std::iter::repeat(0.0f32)), // never read: kernel forced to 1.0 below
|
||||||
};
|
};
|
||||||
|
let weights: Box<dyn Iterator<Item = f32>> = if spec.pair_balance {
|
||||||
|
Box::new(circular_scalar_values(
|
||||||
|
&layer_dir,
|
||||||
|
ScalarKind::PairWeight,
|
||||||
|
round_start,
|
||||||
|
spec.no_ambiguity,
|
||||||
|
)?)
|
||||||
|
} else {
|
||||||
|
Box::new(std::iter::repeat(1.0f32))
|
||||||
|
};
|
||||||
let candidates = (round_start..n_minorants)
|
let candidates = (round_start..n_minorants)
|
||||||
.chain(0..round_start)
|
.chain(0..round_start)
|
||||||
.zip(entropies);
|
.zip(entropies)
|
||||||
|
.zip(weights);
|
||||||
|
|
||||||
let mut accepted: HashSet<usize> = HashSet::new();
|
let mut accepted: HashSet<usize> = HashSet::new();
|
||||||
for (family_idx, entropy) in candidates {
|
for ((family_idx, entropy), weight) in candidates {
|
||||||
visited += 1;
|
visited += 1;
|
||||||
if !eligible.view().get(family_idx) {
|
if !eligible.view().get(family_idx) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let kernel = match entropy_bias {
|
let kernel = match spec.entropy_bias {
|
||||||
Some(bias) => {
|
Some(bias) => {
|
||||||
let d = (entropy as f64 - bias.mu) / bias.sigma;
|
let d = (entropy as f64 - bias.mu) / bias.sigma;
|
||||||
(-0.5 * d * d).exp()
|
(-0.5 * d * d).exp()
|
||||||
}
|
}
|
||||||
None => 1.0,
|
None => 1.0,
|
||||||
};
|
};
|
||||||
if rng.random::<f64>() < (p0 * kernel).min(1.0) {
|
if rng.random::<f64>() < (p0 * f64::from(weight) * kernel).min(1.0) {
|
||||||
eligible.set(family_idx, false);
|
eligible.set(family_idx, false);
|
||||||
accepted.insert(family_idx);
|
accepted.insert(family_idx);
|
||||||
if accepted.len() >= target_this_round {
|
if accepted.len() >= target_this_round {
|
||||||
@@ -353,9 +417,13 @@ fn sample_layer(
|
|||||||
mask.family_size() >= 2,
|
mask.family_size() >= 2,
|
||||||
"eligible bitset must only accept non-monomorphic minorants"
|
"eligible bitset must only accept non-monomorphic minorants"
|
||||||
);
|
);
|
||||||
let masked: Vec<Option<u8>> =
|
let masked: Vec<Option<u8>> = genome_mask
|
||||||
genome_mask.iter().map(|&m| masked_state(m, free_loss, no_ambiguity)).collect();
|
.iter()
|
||||||
if survives_masking(&masked, excluded) {
|
.map(|&m| masked_state(m, spec.free_loss, spec.no_ambiguity))
|
||||||
|
.collect();
|
||||||
|
if survives_masking(&masked, excluded)
|
||||||
|
&& (!spec.quartet || survives_quartet(&masked, excluded))
|
||||||
|
{
|
||||||
survivors.push(SurvivingFamily { family_idx, mask, masked });
|
survivors.push(SurvivingFamily { family_idx, mask, masked });
|
||||||
kept += 1;
|
kept += 1;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,162 +0,0 @@
|
|||||||
//! Persisted, per-layer Shannon entropy (`entropy15`, see
|
|
||||||
//! `algorithms::entropy`), one value per **non-monomorphic minorant
|
|
||||||
//! family** — compact: indexed by the dedicated ordinal
|
|
||||||
//! `algorithms::non_monomorphic_reindex_layer` computes (skips monomorphic
|
|
||||||
//! minorants entirely, ~98% of minorants on real data), never by
|
|
||||||
//! `scan_layer_families`'s own `family_idx` (which counts every minorant,
|
|
||||||
//! monomorphic included).
|
|
||||||
//!
|
|
||||||
//! Exists so repeated reads of the same layer's entropy values (e.g.
|
|
||||||
//! entropy-biased family selection, once reconnected) are a plain
|
|
||||||
//! positional mmap lookup instead of re-paying `scan_layer_families`'s
|
|
||||||
//! cross-partition resolution — building this file costs exactly that same
|
|
||||||
//! full scan the first time, it does not make any single computation of
|
|
||||||
//! entropy itself faster.
|
|
||||||
//!
|
|
||||||
//! Mirrors `SiblingAnnex`'s on-disk convention (magic + header + flat
|
|
||||||
//! per-entry data, read-only after build), just with `f32` payloads and a
|
|
||||||
//! different (compact) numbering — see the module docs above.
|
|
||||||
|
|
||||||
use std::fs::{File, OpenOptions};
|
|
||||||
use std::io;
|
|
||||||
use std::path::Path;
|
|
||||||
|
|
||||||
use memmap2::{Mmap, MmapMut};
|
|
||||||
|
|
||||||
const MAGIC: [u8; 4] = *b"PENT";
|
|
||||||
|
|
||||||
// Header: magic(4) + _pad(4) + n(8) = 16 bytes. Data (4 bytes/entry) follows.
|
|
||||||
const HEADER_SIZE: usize = 16;
|
|
||||||
|
|
||||||
/// Sentinel: not yet computed. Entropy is always `>= 0.0`, so any negative
|
|
||||||
/// value is unambiguous — chosen over `NaN` to avoid `NaN`-comparison
|
|
||||||
/// footguns at every read site.
|
|
||||||
const SENTINEL: f32 = -1.0;
|
|
||||||
|
|
||||||
pub(crate) const ENTROPY_ANNEX_FILE_NAME: &str = "entropy.pent";
|
|
||||||
|
|
||||||
pub struct EntropyAnnex {
|
|
||||||
mmap: Mmap,
|
|
||||||
n: usize,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl EntropyAnnex {
|
|
||||||
pub fn open(path: &Path) -> io::Result<Self> {
|
|
||||||
let mmap = unsafe { Mmap::map(&File::open(path)?)? };
|
|
||||||
if mmap.len() < HEADER_SIZE {
|
|
||||||
return Err(io::Error::new(
|
|
||||||
io::ErrorKind::InvalidData,
|
|
||||||
"PENT file too short",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
if mmap[0..4] != MAGIC {
|
|
||||||
return Err(io::Error::new(io::ErrorKind::InvalidData, "bad PENT magic"));
|
|
||||||
}
|
|
||||||
let n = u64::from_le_bytes(mmap[8..16].try_into().unwrap()) as usize;
|
|
||||||
if mmap.len() < HEADER_SIZE + n * 4 {
|
|
||||||
return Err(io::Error::new(
|
|
||||||
io::ErrorKind::InvalidData,
|
|
||||||
"PENT file truncated",
|
|
||||||
));
|
|
||||||
}
|
|
||||||
Ok(Self { mmap, n })
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn len(&self) -> usize {
|
|
||||||
self.n
|
|
||||||
}
|
|
||||||
|
|
||||||
/// `None` for the sentinel (not yet computed — shouldn't happen against
|
|
||||||
/// a fully-built annex).
|
|
||||||
pub fn get(&self, idx: usize) -> Option<f32> {
|
|
||||||
let off = HEADER_SIZE + idx * 4;
|
|
||||||
let v = f32::from_le_bytes(self.mmap[off..off + 4].try_into().unwrap());
|
|
||||||
(v >= 0.0).then_some(v)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
pub struct EntropyAnnexBuilder {
|
|
||||||
mmap: MmapMut,
|
|
||||||
}
|
|
||||||
|
|
||||||
impl EntropyAnnexBuilder {
|
|
||||||
/// Create a new annex of `n` entries (a layer's non-monomorphic
|
|
||||||
/// minorant count) at `path`, pre-initialised to the sentinel.
|
|
||||||
pub(crate) fn new(n: usize, path: &Path) -> io::Result<Self> {
|
|
||||||
let file_size = HEADER_SIZE + n * 4;
|
|
||||||
let file = OpenOptions::new()
|
|
||||||
.read(true)
|
|
||||||
.write(true)
|
|
||||||
.create(true)
|
|
||||||
.truncate(true)
|
|
||||||
.open(path)?;
|
|
||||||
file.set_len(file_size as u64)?;
|
|
||||||
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
|
|
||||||
mmap[0..4].copy_from_slice(&MAGIC);
|
|
||||||
mmap[4..8].copy_from_slice(&[0u8; 4]);
|
|
||||||
mmap[8..16].copy_from_slice(&(n as u64).to_le_bytes());
|
|
||||||
for i in 0..n {
|
|
||||||
let off = HEADER_SIZE + i * 4;
|
|
||||||
mmap[off..off + 4].copy_from_slice(&SENTINEL.to_le_bytes());
|
|
||||||
}
|
|
||||||
Ok(Self { mmap })
|
|
||||||
}
|
|
||||||
|
|
||||||
pub(crate) fn set(&mut self, idx: usize, value: f32) {
|
|
||||||
debug_assert!(value >= 0.0, "entropy must be non-negative, got {value}");
|
|
||||||
let off = HEADER_SIZE + idx * 4;
|
|
||||||
self.mmap[off..off + 4].copy_from_slice(&value.to_le_bytes());
|
|
||||||
}
|
|
||||||
|
|
||||||
pub(crate) fn close(self) -> io::Result<()> {
|
|
||||||
self.mmap.flush()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[cfg(test)]
|
|
||||||
mod tests {
|
|
||||||
use super::*;
|
|
||||||
use tempfile::tempdir;
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn sentinel_is_negative_and_unset_entries_read_as_none() {
|
|
||||||
let dir = tempdir().unwrap();
|
|
||||||
let path = dir.path().join("test.pent");
|
|
||||||
let builder = EntropyAnnexBuilder::new(4, &path).unwrap();
|
|
||||||
builder.close().unwrap();
|
|
||||||
let annex = EntropyAnnex::open(&path).unwrap();
|
|
||||||
for i in 0..4 {
|
|
||||||
assert_eq!(annex.get(i), None);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn roundtrip_including_zero_entropy() {
|
|
||||||
let dir = tempdir().unwrap();
|
|
||||||
let path = dir.path().join("test.pent");
|
|
||||||
let mut builder = EntropyAnnexBuilder::new(4, &path).unwrap();
|
|
||||||
let values = [0.0f32, 1.5, 3.9, 0.0001];
|
|
||||||
for (i, &v) in values.iter().enumerate() {
|
|
||||||
builder.set(i, v);
|
|
||||||
}
|
|
||||||
builder.close().unwrap();
|
|
||||||
let annex = EntropyAnnex::open(&path).unwrap();
|
|
||||||
for (i, &v) in values.iter().enumerate() {
|
|
||||||
assert_eq!(annex.get(i), Some(v), "entry {i}");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn mixed_sentinel_and_real_values() {
|
|
||||||
let dir = tempdir().unwrap();
|
|
||||||
let path = dir.path().join("test.pent");
|
|
||||||
let mut builder = EntropyAnnexBuilder::new(3, &path).unwrap();
|
|
||||||
builder.set(1, 2.0);
|
|
||||||
builder.close().unwrap();
|
|
||||||
let annex = EntropyAnnex::open(&path).unwrap();
|
|
||||||
assert_eq!(annex.get(0), None);
|
|
||||||
assert_eq!(annex.get(1), Some(2.0));
|
|
||||||
assert_eq!(annex.get(2), None);
|
|
||||||
assert_eq!(annex.len(), 3);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,7 +1,8 @@
|
|||||||
//! [`SiblingBuilder`] — construction triggered *internally*, as a side
|
//! [`SiblingBuilder`] — construction triggered *internally*, as a side
|
||||||
//! effect of a [`crate::siblings::extensions::SiblingExt`] method (e.g.
|
//! effect of a [`crate::siblings::extensions::SiblingExt`] method (e.g.
|
||||||
//! `shannon_entropy_csv`/`family_overlap` ensuring their annexes exist
|
//! `family_overlap` ensuring its annex exists before reading it, or the
|
||||||
//! before reading them), never called directly from outside this crate.
|
//! sampler ensuring the entropy/quartet/pair-weight annexes it draws from),
|
||||||
|
//! never called directly from outside this crate.
|
||||||
//! Unlike `SiblingExt` (explicit, CLI-facing actions), nothing here is
|
//! Unlike `SiblingExt` (explicit, CLI-facing actions), nothing here is
|
||||||
//! meant to be its own CLI flag — same `pub`/`pub(crate)` split
|
//! meant to be its own CLI flag — same `pub`/`pub(crate)` split
|
||||||
//! `obikindexer` uses for its own `IndexBuilder`/`PrivateBuilder`.
|
//! `obikindexer` uses for its own `IndexBuilder`/`PrivateBuilder`.
|
||||||
@@ -10,44 +11,75 @@ use obikidxcache::index_cache::IndexCache;
|
|||||||
use obikindex::{OKIError, OKIResult};
|
use obikindex::{OKIError, OKIResult};
|
||||||
use obisys::progress_bar;
|
use obisys::progress_bar;
|
||||||
|
|
||||||
use crate::siblings::algorithms::{ensure_layer_entropy_annex, is_fast_mode};
|
use crate::siblings::algorithms::{
|
||||||
use crate::siblings::{FAMILY_OVERLAP_FILE_NAME, FamilyOverlapAccumulator};
|
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes,
|
||||||
|
inverse_overlap_matrix, is_fast_mode,
|
||||||
|
};
|
||||||
|
use crate::siblings::{FamilyOverlap, FamilyOverlapAccumulator, family_overlap_file_name};
|
||||||
|
|
||||||
pub(crate) trait SiblingBuilder {
|
pub(crate) trait SiblingBuilder {
|
||||||
/// Build every cached layer's entropy annex, and the whole index's
|
/// Build, for every cached layer, the annexes `request` asks for that
|
||||||
/// family-overlap annex, that don't already exist — one call covers
|
/// don't already exist (entropy, quartet and pair weight, each per
|
||||||
/// both, whichever of the two a caller actually needs
|
/// `--no-ambiguity` counting convention), and the index-level
|
||||||
/// (`shannon_entropy_csv`/entropy-biased sampling on one side,
|
/// family-overlap annexes it asks for or that a pair weight is computed
|
||||||
/// `family_overlap`/`--min-shared-family` on the other): a missing
|
/// from. One call covers all of them, whichever a caller actually needs:
|
||||||
/// annex always triggers a scan that feeds both, per
|
/// a missing entropy, quartet or overlap annex triggers one scan that
|
||||||
/// `ensure_layer_entropy_annex`'s own docs. A no-op, cheap
|
/// feeds every missing one, per `ensure_layer_family_annexes`'s own docs,
|
||||||
/// existence-check pass, once both already exist everywhere.
|
/// then a second scan builds the missing pair weights, which need the
|
||||||
fn ensure_entropy_and_overlap_annexes(&self) -> OKIResult<()>;
|
/// finished overlap matrix. A no-op, cheap existence-check pass, once
|
||||||
|
/// everything requested already exists.
|
||||||
|
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
|
||||||
}
|
}
|
||||||
|
|
||||||
impl SiblingBuilder for IndexCache {
|
impl SiblingBuilder for IndexCache {
|
||||||
fn ensure_entropy_and_overlap_annexes(&self) -> OKIResult<()> {
|
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
|
||||||
let n_genomes = self.meta().genomes().len();
|
let n_genomes = self.meta().genomes().len();
|
||||||
let fast_mode = is_fast_mode(self);
|
let fast_mode = is_fast_mode(self);
|
||||||
|
let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity));
|
||||||
|
|
||||||
let overlap_path = self.index().dir().join(FAMILY_OVERLAP_FILE_NAME);
|
// Sweep 1: entropy, quartet, and every family-overlap annex still missing.
|
||||||
let mut overlap_acc =
|
let mut overlap_accs: Vec<(bool, FamilyOverlapAccumulator)> = request
|
||||||
(!overlap_path.exists()).then(|| FamilyOverlapAccumulator::new(n_genomes));
|
.overlap_conventions()
|
||||||
|
.into_iter()
|
||||||
let pb = progress_bar("entropy_overlap_annex", self.n_partition() as u64, "partitions");
|
.filter(|&no_ambiguity| !overlap_path(no_ambiguity).exists())
|
||||||
|
.map(|no_ambiguity| (no_ambiguity, FamilyOverlapAccumulator::new(n_genomes, no_ambiguity)))
|
||||||
|
.collect();
|
||||||
|
let pb = progress_bar("family_annexes", self.n_partition() as u64, "partitions");
|
||||||
for part in self.partitions() {
|
for part in self.partitions() {
|
||||||
let n_layer = self.n_layer(part).unwrap_or(0);
|
let n_layer = self.n_layer(part).unwrap_or(0);
|
||||||
for l in 0..n_layer {
|
for l in 0..n_layer {
|
||||||
ensure_layer_entropy_annex(self, part, l, n_genomes, fast_mode, overlap_acc.as_mut())?;
|
ensure_layer_family_annexes(self, part, l, n_genomes, fast_mode, request, &mut overlap_accs)?;
|
||||||
|
}
|
||||||
|
pb.inc(1);
|
||||||
|
}
|
||||||
|
pb.finish_and_clear();
|
||||||
|
for (no_ambiguity, acc) in overlap_accs {
|
||||||
|
acc.write(&overlap_path(no_ambiguity)).map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sweep 2: pair weights, from the now-complete overlap annexes.
|
||||||
|
if request.pair_weight.is_empty() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
let matrices = request
|
||||||
|
.pair_weight_conventions()
|
||||||
|
.into_iter()
|
||||||
|
.map(|no_ambiguity| {
|
||||||
|
FamilyOverlap::open(&overlap_path(no_ambiguity))
|
||||||
|
.map(|overlap| (no_ambiguity, inverse_overlap_matrix(&overlap)))
|
||||||
|
.map_err(OKIError::Io)
|
||||||
|
})
|
||||||
|
.collect::<OKIResult<Vec<_>>>()?;
|
||||||
|
let pb = progress_bar("pair_weight_annex", self.n_partition() as u64, "partitions");
|
||||||
|
for part in self.partitions() {
|
||||||
|
let n_layer = self.n_layer(part).unwrap_or(0);
|
||||||
|
for l in 0..n_layer {
|
||||||
|
ensure_layer_pair_weight_annexes(self, part, l, n_genomes, fast_mode, &matrices)?;
|
||||||
}
|
}
|
||||||
pb.inc(1);
|
pb.inc(1);
|
||||||
}
|
}
|
||||||
pb.finish_and_clear();
|
pb.finish_and_clear();
|
||||||
|
|
||||||
if let Some(acc) = overlap_acc {
|
|
||||||
acc.write(&overlap_path).map_err(OKIError::Io)?;
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -14,12 +14,15 @@ use obikindex::{OKIError, OKIResult};
|
|||||||
use obisys::progress_bar;
|
use obisys::progress_bar;
|
||||||
|
|
||||||
use crate::siblings::algorithms::{
|
use crate::siblings::algorithms::{
|
||||||
Selection, SiblingAnnexStats, build_layer_sibling_annex, family_entropy, family_entropy_4,
|
AnnexRequest, Selection, SiblingAnnexStats, build_layer_sibling_annex, family_entropy,
|
||||||
is_fast_mode, non_monomorphic_reindex_layer, scan_layer_families, sibling_annex_stats,
|
family_entropy_4, is_fast_mode, non_monomorphic_reindex_layer, scan_layer_families,
|
||||||
sibling_family_size_histogram,
|
sibling_annex_stats, sibling_family_size_histogram,
|
||||||
};
|
};
|
||||||
use crate::siblings::extensions::SiblingBuilder;
|
use crate::siblings::extensions::SiblingBuilder;
|
||||||
use crate::siblings::{ENTROPY_ANNEX_FILE_NAME, FAMILY_OVERLAP_FILE_NAME, FamilyOverlap};
|
use crate::siblings::{
|
||||||
|
FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, QUARTET_ANNEX_FILE_NAMES, ScalarKind,
|
||||||
|
family_overlap_file_name,
|
||||||
|
};
|
||||||
|
|
||||||
pub trait SiblingExt {
|
pub trait SiblingExt {
|
||||||
/// Build the sibling-count/minorant annex for every layer of every
|
/// Build the sibling-count/minorant annex for every layer of every
|
||||||
@@ -81,8 +84,10 @@ pub trait SiblingExt {
|
|||||||
fn sibling_annex_stats(&self) -> OKIResult<SiblingAnnexStats>;
|
fn sibling_annex_stats(&self) -> OKIResult<SiblingAnnexStats>;
|
||||||
|
|
||||||
/// The Family Overlap annex — number of shared *variable* families per
|
/// The Family Overlap annex — number of shared *variable* families per
|
||||||
/// genome pair, index-wide (see [`FamilyOverlap`]'s own module docs).
|
/// genome pair, index-wide (see [`FamilyOverlap`]'s own module docs),
|
||||||
/// Built (alongside the entropy annex) if missing, same as every other
|
/// under the `--no-ambiguity` counting convention `no_ambiguity` selects
|
||||||
|
/// (genomes carrying several members of a family do not count as
|
||||||
|
/// carrying it when `true`). Built if missing, same as every other
|
||||||
/// annex-backed method here. Feeds `--family-overlap`'s own matrix
|
/// annex-backed method here. Feeds `--family-overlap`'s own matrix
|
||||||
/// output directly, and `--min-shared-family`'s auto-exclusion
|
/// output directly, and `--min-shared-family`'s auto-exclusion
|
||||||
/// (`FamilyOverlap::mean_row(g) < threshold` — same statistic
|
/// (`FamilyOverlap::mean_row(g) < threshold` — same statistic
|
||||||
@@ -90,7 +95,7 @@ pub trait SiblingExt {
|
|||||||
/// triggers adding `g` to the excluded-genome set consumed by
|
/// triggers adding `g` to the excluded-genome set consumed by
|
||||||
/// `algorithms::scan::sampled_scan`'s callers; this needs no
|
/// `algorithms::scan::sampled_scan`'s callers; this needs no
|
||||||
/// computation of its own beyond that check).
|
/// computation of its own beyond that check).
|
||||||
fn family_overlap(&self) -> OKIResult<FamilyOverlap>;
|
fn family_overlap(&self, no_ambiguity: bool) -> OKIResult<FamilyOverlap>;
|
||||||
}
|
}
|
||||||
|
|
||||||
impl SiblingExt for IndexCache {
|
impl SiblingExt for IndexCache {
|
||||||
@@ -115,16 +120,24 @@ impl SiblingExt for IndexCache {
|
|||||||
.expect("cache-consistent: n_layer(part) already bounds l");
|
.expect("cache-consistent: n_layer(part) already bounds l");
|
||||||
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
||||||
|
|
||||||
// Invalidate this layer's entropy annex, if any — it was
|
// Invalidate this layer's entropy and quartet annexes, if
|
||||||
// built against the sibling annex's *previous*
|
// any — they were built against the sibling annex's
|
||||||
// `family_idx` numbering (`non_monomorphic_reindex_layer`
|
// *previous* `family_idx` numbering
|
||||||
// reads the sibling annex), now stale.
|
// (`non_monomorphic_reindex_layer` reads the sibling
|
||||||
// `ensure_entropy_and_overlap_annexes` below only builds
|
// annex), now stale. `ensure_family_annexes` below only
|
||||||
// what's *missing*, so the old file must be removed here
|
// builds what's *missing*, so the old files must be removed
|
||||||
// for it to be rebuilt rather than silently left stale.
|
// here for them to be rebuilt rather than silently left
|
||||||
let entropy_path = layer.dir().join(ENTROPY_ANNEX_FILE_NAME);
|
// stale.
|
||||||
if entropy_path.exists() {
|
for name in ScalarKind::Entropy
|
||||||
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
|
.file_names()
|
||||||
|
.into_iter()
|
||||||
|
.chain(ScalarKind::PairWeight.file_names())
|
||||||
|
.chain(QUARTET_ANNEX_FILE_NAMES)
|
||||||
|
{
|
||||||
|
let entropy_path = layer.dir().join(name);
|
||||||
|
if entropy_path.exists() {
|
||||||
|
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
total_slots += part_slots;
|
total_slots += part_slots;
|
||||||
@@ -136,13 +149,16 @@ impl SiblingExt for IndexCache {
|
|||||||
pb.finish_and_clear();
|
pb.finish_and_clear();
|
||||||
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
|
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
|
||||||
|
|
||||||
// The family-overlap annex is index-level, not per-layer (see its
|
// The family-overlap annexes are index-level, not per-layer (see
|
||||||
// own module docs) — any layer's sibling annex changing makes the
|
// their own module docs) — any layer's sibling annex changing makes
|
||||||
// whole thing stale, so it's invalidated once here rather than
|
// them all stale (and, with them, every pair weight computed from
|
||||||
// per-layer above.
|
// them, removed per layer above), so they are invalidated once here
|
||||||
let overlap_path = self.index().dir().join(FAMILY_OVERLAP_FILE_NAME);
|
// rather than per-layer above.
|
||||||
if overlap_path.exists() {
|
for name in FAMILY_OVERLAP_FILE_NAMES {
|
||||||
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
|
let overlap_path = self.index().dir().join(name);
|
||||||
|
if overlap_path.exists() {
|
||||||
|
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Precompute the entropy and family-overlap annexes too, now that
|
// Precompute the entropy and family-overlap annexes too, now that
|
||||||
@@ -152,19 +168,17 @@ impl SiblingExt for IndexCache {
|
|||||||
// rather than leaving them to be built lazily on first
|
// rather than leaving them to be built lazily on first
|
||||||
// `--shannon`/`--entropy-bias`/`--min-shared-family`/
|
// `--shannon`/`--entropy-bias`/`--min-shared-family`/
|
||||||
// `--family-overlap` use.
|
// `--family-overlap` use.
|
||||||
tracing::info!("building entropy and family-overlap annexes");
|
tracing::info!("building entropy and quartet (both --no-ambiguity conventions) and family-overlap annexes");
|
||||||
self.ensure_entropy_and_overlap_annexes()?;
|
self.ensure_family_annexes(&AnnexRequest::everything())?;
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> {
|
fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> {
|
||||||
// Entropy/family-overlap annexes aren't this method's own output,
|
// Needs only the sibling annex (`non_monomorphic_reindex_layer`
|
||||||
// but ensuring they exist is a cheap prerequisite check, not a
|
// below): this method runs its own, differently-scoped scan and
|
||||||
// reason to reuse their scan here — see below, this method
|
// recomputes every value it writes, so it neither reads nor
|
||||||
// deliberately runs its own, differently-scoped one.
|
// requires the entropy/family-overlap annexes.
|
||||||
self.ensure_entropy_and_overlap_annexes()?;
|
|
||||||
|
|
||||||
let n_genomes = self.meta().genomes().len();
|
let n_genomes = self.meta().genomes().len();
|
||||||
let fast_mode = is_fast_mode(self);
|
let fast_mode = is_fast_mode(self);
|
||||||
|
|
||||||
@@ -237,9 +251,9 @@ impl SiblingExt for IndexCache {
|
|||||||
sibling_annex_stats(self)
|
sibling_annex_stats(self)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn family_overlap(&self) -> OKIResult<FamilyOverlap> {
|
fn family_overlap(&self, no_ambiguity: bool) -> OKIResult<FamilyOverlap> {
|
||||||
self.ensure_entropy_and_overlap_annexes()?;
|
self.ensure_family_annexes(&AnnexRequest { overlap: vec![no_ambiguity], ..Default::default() })?;
|
||||||
let path = self.index().dir().join(FAMILY_OVERLAP_FILE_NAME);
|
let path = self.index().dir().join(family_overlap_file_name(no_ambiguity));
|
||||||
FamilyOverlap::open(&path).map_err(OKIError::Io)
|
FamilyOverlap::open(&path).map_err(OKIError::Io)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -24,20 +24,43 @@ use std::path::Path;
|
|||||||
|
|
||||||
use memmap2::Mmap;
|
use memmap2::Mmap;
|
||||||
|
|
||||||
use crate::siblings::helpers::triangle_index;
|
use crate::siblings::helpers::{is_carrier, triangle_index};
|
||||||
|
|
||||||
const MAGIC: [u8; 4] = *b"POVL";
|
const MAGIC: [u8; 4] = *b"POVL";
|
||||||
|
|
||||||
// Header: magic(4) + _pad(4) + n_genomes(8) = 16 bytes. Data (8 bytes/pair,
|
// Header: magic(4) + flags(4) + n_genomes(8) = 16 bytes. Data (8 bytes/pair,
|
||||||
// upper triangle) follows.
|
// upper triangle) follows. `flags` bit 0 = `no_ambiguity` (see
|
||||||
|
// `FamilyOverlapAccumulator::new`); older files have `flags == 0`, i.e.
|
||||||
|
// `no_ambiguity == false`, which is what they were computed under.
|
||||||
const HEADER_SIZE: usize = 16;
|
const HEADER_SIZE: usize = 16;
|
||||||
|
const FLAG_NO_AMBIGUITY: u32 = 1;
|
||||||
|
|
||||||
pub(crate) const FAMILY_OVERLAP_FILE_NAME: &str = "family_overlap.povl";
|
pub(crate) const FAMILY_OVERLAP_FILE_NAME: &str = "family_overlap.povl";
|
||||||
|
|
||||||
|
/// Same annex, built with `no_ambiguity == true`.
|
||||||
|
pub(crate) const FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME: &str = "family_overlap.noamb.povl";
|
||||||
|
|
||||||
|
/// Every family-overlap file name, whatever its counting convention — for
|
||||||
|
/// invalidating them all together when the sibling annex they derive from is
|
||||||
|
/// rebuilt.
|
||||||
|
pub(crate) const FAMILY_OVERLAP_FILE_NAMES: [&str; 2] =
|
||||||
|
[FAMILY_OVERLAP_FILE_NAME, FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME];
|
||||||
|
|
||||||
|
/// File name of the family-overlap annex built under the given
|
||||||
|
/// `--no-ambiguity` counting convention. The two never share a file.
|
||||||
|
pub(crate) fn family_overlap_file_name(no_ambiguity: bool) -> &'static str {
|
||||||
|
if no_ambiguity {
|
||||||
|
FAMILY_OVERLAP_NO_AMBIGUITY_FILE_NAME
|
||||||
|
} else {
|
||||||
|
FAMILY_OVERLAP_FILE_NAME
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Read-only, mmap-backed view of an already-built Family Overlap annex.
|
/// Read-only, mmap-backed view of an already-built Family Overlap annex.
|
||||||
pub struct FamilyOverlap {
|
pub struct FamilyOverlap {
|
||||||
mmap: Mmap,
|
mmap: Mmap,
|
||||||
n_genomes: usize,
|
n_genomes: usize,
|
||||||
|
no_ambiguity: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl FamilyOverlap {
|
impl FamilyOverlap {
|
||||||
@@ -54,13 +77,22 @@ impl FamilyOverlap {
|
|||||||
if mmap.len() < HEADER_SIZE + n_pairs * 8 {
|
if mmap.len() < HEADER_SIZE + n_pairs * 8 {
|
||||||
return Err(io::Error::new(io::ErrorKind::InvalidData, "POVL file truncated"));
|
return Err(io::Error::new(io::ErrorKind::InvalidData, "POVL file truncated"));
|
||||||
}
|
}
|
||||||
Ok(Self { mmap, n_genomes })
|
let flags = u32::from_le_bytes(mmap[4..8].try_into().unwrap());
|
||||||
|
Ok(Self { mmap, n_genomes, no_ambiguity: flags & FLAG_NO_AMBIGUITY != 0 })
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn n_genomes(&self) -> usize {
|
pub fn n_genomes(&self) -> usize {
|
||||||
self.n_genomes
|
self.n_genomes
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The counting convention these counts were computed under: `true`
|
||||||
|
/// when a genome carrying several members of a family was *not* counted
|
||||||
|
/// as carrying it (`--no-ambiguity`), `false` when any non-empty
|
||||||
|
/// state counts as carrying it.
|
||||||
|
pub fn no_ambiguity(&self) -> bool {
|
||||||
|
self.no_ambiguity
|
||||||
|
}
|
||||||
|
|
||||||
/// Number of variable families shared by genomes `i` and `j` — `0` for
|
/// Number of variable families shared by genomes `i` and `j` — `0` for
|
||||||
/// `i == j` (not stored; a genome trivially "shares" every one of its
|
/// `i == j` (not stored; a genome trivially "shares" every one of its
|
||||||
/// own families with itself, not a meaningful statistic here).
|
/// own families with itself, not a meaningful statistic here).
|
||||||
@@ -95,24 +127,30 @@ impl FamilyOverlap {
|
|||||||
/// `EntropyAnnexBuilder` do for their own, much larger, per-k-mer data.
|
/// `EntropyAnnexBuilder` do for their own, much larger, per-k-mer data.
|
||||||
pub(crate) struct FamilyOverlapAccumulator {
|
pub(crate) struct FamilyOverlapAccumulator {
|
||||||
n_genomes: usize,
|
n_genomes: usize,
|
||||||
|
no_ambiguity: bool,
|
||||||
counts: Vec<u64>,
|
counts: Vec<u64>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl FamilyOverlapAccumulator {
|
impl FamilyOverlapAccumulator {
|
||||||
pub(crate) fn new(n_genomes: usize) -> Self {
|
/// `no_ambiguity` is the counting convention: `false` counts every
|
||||||
|
/// genome with a non-empty state as carrying the family, `true` only the
|
||||||
|
/// genomes carrying exactly one member of it — the others are missing
|
||||||
|
/// data in the alignment (`--no-ambiguity`), so they share nothing there.
|
||||||
|
pub(crate) fn new(n_genomes: usize, no_ambiguity: bool) -> Self {
|
||||||
let n_pairs = n_genomes.saturating_sub(1) * n_genomes / 2;
|
let n_pairs = n_genomes.saturating_sub(1) * n_genomes / 2;
|
||||||
Self { n_genomes, counts: vec![0u64; n_pairs] }
|
Self { n_genomes, no_ambiguity, counts: vec![0u64; n_pairs] }
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One variable family's contribution: every pair of genomes both
|
/// One variable family's contribution: every pair of genomes both
|
||||||
/// carrying it (`genome_mask[g] != 0`) gets its count incremented once.
|
/// carrying it (per the counting convention) gets its count incremented
|
||||||
|
/// once.
|
||||||
pub(crate) fn add_family(&mut self, genome_mask: &[u8]) {
|
pub(crate) fn add_family(&mut self, genome_mask: &[u8]) {
|
||||||
for i in 0..self.n_genomes {
|
for i in 0..self.n_genomes {
|
||||||
if genome_mask[i] == 0 {
|
if !is_carrier(genome_mask[i], self.no_ambiguity) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
for j in (i + 1)..self.n_genomes {
|
for j in (i + 1)..self.n_genomes {
|
||||||
if genome_mask[j] != 0 {
|
if is_carrier(genome_mask[j], self.no_ambiguity) {
|
||||||
self.counts[triangle_index(self.n_genomes, i, j)] += 1;
|
self.counts[triangle_index(self.n_genomes, i, j)] += 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -122,7 +160,8 @@ impl FamilyOverlapAccumulator {
|
|||||||
pub(crate) fn write(&self, path: &Path) -> io::Result<()> {
|
pub(crate) fn write(&self, path: &Path) -> io::Result<()> {
|
||||||
let mut buf = Vec::with_capacity(HEADER_SIZE + self.counts.len() * 8);
|
let mut buf = Vec::with_capacity(HEADER_SIZE + self.counts.len() * 8);
|
||||||
buf.extend_from_slice(&MAGIC);
|
buf.extend_from_slice(&MAGIC);
|
||||||
buf.extend_from_slice(&[0u8; 4]);
|
let flags = if self.no_ambiguity { FLAG_NO_AMBIGUITY } else { 0 };
|
||||||
|
buf.extend_from_slice(&flags.to_le_bytes());
|
||||||
buf.extend_from_slice(&(self.n_genomes as u64).to_le_bytes());
|
buf.extend_from_slice(&(self.n_genomes as u64).to_le_bytes());
|
||||||
for &c in &self.counts {
|
for &c in &self.counts {
|
||||||
buf.extend_from_slice(&c.to_le_bytes());
|
buf.extend_from_slice(&c.to_le_bytes());
|
||||||
@@ -140,7 +179,7 @@ mod tests {
|
|||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
let path = dir.path().join("test.povl");
|
let path = dir.path().join("test.povl");
|
||||||
|
|
||||||
let mut acc = FamilyOverlapAccumulator::new(3);
|
let mut acc = FamilyOverlapAccumulator::new(3, false);
|
||||||
acc.add_family(&[1, 1, 0]); // genomes 0,1 share this family
|
acc.add_family(&[1, 1, 0]); // genomes 0,1 share this family
|
||||||
acc.add_family(&[1, 1, 1]); // all three share this one
|
acc.add_family(&[1, 1, 1]); // all three share this one
|
||||||
acc.add_family(&[0, 1, 1]); // genomes 1,2 share this family
|
acc.add_family(&[0, 1, 1]); // genomes 1,2 share this family
|
||||||
@@ -160,8 +199,27 @@ mod tests {
|
|||||||
fn single_genome_mean_row_is_zero() {
|
fn single_genome_mean_row_is_zero() {
|
||||||
let dir = tempfile::tempdir().unwrap();
|
let dir = tempfile::tempdir().unwrap();
|
||||||
let path = dir.path().join("test.povl");
|
let path = dir.path().join("test.povl");
|
||||||
FamilyOverlapAccumulator::new(1).write(&path).unwrap();
|
FamilyOverlapAccumulator::new(1, false).write(&path).unwrap();
|
||||||
let overlap = FamilyOverlap::open(&path).unwrap();
|
let overlap = FamilyOverlap::open(&path).unwrap();
|
||||||
assert_eq!(overlap.mean_row(0), 0.0);
|
assert_eq!(overlap.mean_row(0), 0.0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_ambiguity_does_not_count_multi_member_genomes_as_carriers() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let masks: [&[u8]; 2] = [&[1, 1, 0b0011], &[1, 0b0100, 0b0100]];
|
||||||
|
for no_ambiguity in [false, true] {
|
||||||
|
let path = dir.path().join(family_overlap_file_name(no_ambiguity));
|
||||||
|
let mut acc = FamilyOverlapAccumulator::new(3, no_ambiguity);
|
||||||
|
for m in masks {
|
||||||
|
acc.add_family(m);
|
||||||
|
}
|
||||||
|
acc.write(&path).unwrap();
|
||||||
|
let overlap = FamilyOverlap::open(&path).unwrap();
|
||||||
|
assert_eq!(overlap.no_ambiguity(), no_ambiguity);
|
||||||
|
// genome 2 carries two members in the first family: only counted without the flag
|
||||||
|
assert_eq!(overlap.get(0, 2), if no_ambiguity { 1 } else { 2 });
|
||||||
|
assert_eq!(overlap.get(0, 1), 1 + 1);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,247 @@
|
|||||||
|
//! Persisted, per-layer `f32` value per **non-monomorphic minorant family**:
|
||||||
|
//! the Shannon entropy of the family ([`ScalarKind::Entropy`], see
|
||||||
|
//! `algorithms::entropy`) or its pair-balance weight
|
||||||
|
//! ([`ScalarKind::PairWeight`], see `algorithms::pair_weight`). Compact:
|
||||||
|
//! indexed by the dedicated ordinal `algorithms::non_monomorphic_reindex_layer`
|
||||||
|
//! computes (skips monomorphic minorants entirely, ~98% of minorants on real
|
||||||
|
//! data), never by `scan_layer_families`'s own `family_idx` (which counts
|
||||||
|
//! every minorant, monomorphic included).
|
||||||
|
//!
|
||||||
|
//! Exists so repeated reads of the same layer's values (entropy-biased or
|
||||||
|
//! pair-balanced family selection) are a plain positional mmap lookup
|
||||||
|
//! instead of re-paying `scan_layer_families`'s cross-partition resolution —
|
||||||
|
//! building a file costs exactly that same full scan the first time, it does
|
||||||
|
//! not make any single computation of its value faster.
|
||||||
|
//!
|
||||||
|
//! Mirrors `SiblingAnnex`'s on-disk convention (magic + header + flat
|
||||||
|
//! per-entry data, read-only after build), just with `f32` payloads and a
|
||||||
|
//! different (compact) numbering. The two kinds share the format and differ
|
||||||
|
//! by magic and file name only, so one kind can never be read as the other.
|
||||||
|
|
||||||
|
use std::fs::{File, OpenOptions};
|
||||||
|
use std::io;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use memmap2::{Mmap, MmapMut};
|
||||||
|
|
||||||
|
// Header: magic(4) + flags(4) + n(8) = 16 bytes. Data (4 bytes/entry) follows.
|
||||||
|
// `flags` bit 0 = `no_ambiguity` (the counting convention the values were
|
||||||
|
// computed under, see `FamilyScalarAnnexBuilder::new`); every file written
|
||||||
|
// before the flag existed has `flags == 0`, i.e. `no_ambiguity == false`,
|
||||||
|
// which is what it was computed under.
|
||||||
|
const FLAG_NO_AMBIGUITY: u32 = 1;
|
||||||
|
const HEADER_SIZE: usize = 16;
|
||||||
|
|
||||||
|
/// Sentinel: not yet computed. Every stored value is `>= 0.0`, so any
|
||||||
|
/// negative value is unambiguous — chosen over `NaN` to avoid
|
||||||
|
/// `NaN`-comparison footguns at every read site.
|
||||||
|
const SENTINEL: f32 = -1.0;
|
||||||
|
|
||||||
|
/// What the stored `f32` means.
|
||||||
|
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||||
|
pub(crate) enum ScalarKind {
|
||||||
|
/// Shannon entropy of the family (bits).
|
||||||
|
Entropy,
|
||||||
|
/// Mean, over the pairs of genomes carrying the family, of 1 / (number of
|
||||||
|
/// families the pair shares) — see `algorithms::pair_weight`.
|
||||||
|
PairWeight,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ScalarKind {
|
||||||
|
fn magic(self) -> [u8; 4] {
|
||||||
|
match self {
|
||||||
|
ScalarKind::Entropy => *b"PENT",
|
||||||
|
ScalarKind::PairWeight => *b"PPWT",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// File name of this kind's annex built under the given `--no-ambiguity`
|
||||||
|
/// counting convention. The two conventions never share a file:
|
||||||
|
/// switching between them builds the missing annex without discarding the
|
||||||
|
/// other.
|
||||||
|
pub(crate) fn file_name(self, no_ambiguity: bool) -> &'static str {
|
||||||
|
match (self, no_ambiguity) {
|
||||||
|
(ScalarKind::Entropy, false) => "entropy.pent",
|
||||||
|
(ScalarKind::Entropy, true) => "entropy.noamb.pent",
|
||||||
|
(ScalarKind::PairWeight, false) => "pairweight.ppwt",
|
||||||
|
(ScalarKind::PairWeight, true) => "pairweight.noamb.ppwt",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every file name of this kind, whatever its counting convention — for
|
||||||
|
/// invalidating them all together (e.g. when the sibling annex they were
|
||||||
|
/// numbered against is rebuilt).
|
||||||
|
pub(crate) fn file_names(self) -> [&'static str; 2] {
|
||||||
|
[self.file_name(false), self.file_name(true)]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct FamilyScalarAnnex {
|
||||||
|
mmap: Mmap,
|
||||||
|
n: usize,
|
||||||
|
no_ambiguity: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FamilyScalarAnnex {
|
||||||
|
pub(crate) fn open(path: &Path, kind: ScalarKind) -> io::Result<Self> {
|
||||||
|
let mmap = unsafe { Mmap::map(&File::open(path)?)? };
|
||||||
|
if mmap.len() < HEADER_SIZE {
|
||||||
|
return Err(io::Error::new(io::ErrorKind::InvalidData, "scalar annex file too short"));
|
||||||
|
}
|
||||||
|
if mmap[0..4] != kind.magic() {
|
||||||
|
return Err(io::Error::new(
|
||||||
|
io::ErrorKind::InvalidData,
|
||||||
|
format!("bad magic: not a {kind:?} annex"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let n = u64::from_le_bytes(mmap[8..16].try_into().unwrap()) as usize;
|
||||||
|
if mmap.len() < HEADER_SIZE + n * 4 {
|
||||||
|
return Err(io::Error::new(io::ErrorKind::InvalidData, "scalar annex file truncated"));
|
||||||
|
}
|
||||||
|
let flags = u32::from_le_bytes(mmap[4..8].try_into().unwrap());
|
||||||
|
Ok(Self { mmap, n, no_ambiguity: flags & FLAG_NO_AMBIGUITY != 0 })
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn len(&self) -> usize {
|
||||||
|
self.n
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The counting convention these values were computed under: `true`
|
||||||
|
/// when genomes carrying several members of a family were left out of
|
||||||
|
/// the count (`--no-ambiguity`), `false` when they were counted as a
|
||||||
|
/// state of their own.
|
||||||
|
pub fn no_ambiguity(&self) -> bool {
|
||||||
|
self.no_ambiguity
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `None` for the sentinel (not yet computed — shouldn't happen against
|
||||||
|
/// a fully-built annex).
|
||||||
|
pub fn get(&self, idx: usize) -> Option<f32> {
|
||||||
|
let off = HEADER_SIZE + idx * 4;
|
||||||
|
let v = f32::from_le_bytes(self.mmap[off..off + 4].try_into().unwrap());
|
||||||
|
(v >= 0.0).then_some(v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct FamilyScalarAnnexBuilder {
|
||||||
|
mmap: MmapMut,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FamilyScalarAnnexBuilder {
|
||||||
|
/// Create a new annex of `n` entries (a layer's non-monomorphic
|
||||||
|
/// minorant count) at `path`, pre-initialised to the sentinel.
|
||||||
|
/// `no_ambiguity` records, in the header, the counting convention the
|
||||||
|
/// values will be computed under (see [`FamilyScalarAnnex::no_ambiguity`]).
|
||||||
|
pub(crate) fn new(n: usize, path: &Path, kind: ScalarKind, no_ambiguity: bool) -> io::Result<Self> {
|
||||||
|
let file_size = HEADER_SIZE + n * 4;
|
||||||
|
let file = OpenOptions::new()
|
||||||
|
.read(true)
|
||||||
|
.write(true)
|
||||||
|
.create(true)
|
||||||
|
.truncate(true)
|
||||||
|
.open(path)?;
|
||||||
|
file.set_len(file_size as u64)?;
|
||||||
|
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
|
||||||
|
mmap[0..4].copy_from_slice(&kind.magic());
|
||||||
|
let flags = if no_ambiguity { FLAG_NO_AMBIGUITY } else { 0 };
|
||||||
|
mmap[4..8].copy_from_slice(&flags.to_le_bytes());
|
||||||
|
mmap[8..16].copy_from_slice(&(n as u64).to_le_bytes());
|
||||||
|
for i in 0..n {
|
||||||
|
let off = HEADER_SIZE + i * 4;
|
||||||
|
mmap[off..off + 4].copy_from_slice(&SENTINEL.to_le_bytes());
|
||||||
|
}
|
||||||
|
Ok(Self { mmap })
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn set(&mut self, idx: usize, value: f32) {
|
||||||
|
debug_assert!(value >= 0.0, "stored values must be non-negative, got {value}");
|
||||||
|
let off = HEADER_SIZE + idx * 4;
|
||||||
|
self.mmap[off..off + 4].copy_from_slice(&value.to_le_bytes());
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn close(self) -> io::Result<()> {
|
||||||
|
self.mmap.flush()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use tempfile::tempdir;
|
||||||
|
|
||||||
|
const E: ScalarKind = ScalarKind::Entropy;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn sentinel_is_negative_and_unset_entries_read_as_none() {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let path = dir.path().join("test.pent");
|
||||||
|
FamilyScalarAnnexBuilder::new(4, &path, E, false).unwrap().close().unwrap();
|
||||||
|
let annex = FamilyScalarAnnex::open(&path, E).unwrap();
|
||||||
|
for i in 0..4 {
|
||||||
|
assert_eq!(annex.get(i), None);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn roundtrip_including_zero_value() {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let path = dir.path().join("test.pent");
|
||||||
|
let mut builder = FamilyScalarAnnexBuilder::new(4, &path, E, false).unwrap();
|
||||||
|
let values = [0.0f32, 1.5, 3.9, 0.0001];
|
||||||
|
for (i, &v) in values.iter().enumerate() {
|
||||||
|
builder.set(i, v);
|
||||||
|
}
|
||||||
|
builder.close().unwrap();
|
||||||
|
let annex = FamilyScalarAnnex::open(&path, E).unwrap();
|
||||||
|
for (i, &v) in values.iter().enumerate() {
|
||||||
|
assert_eq!(annex.get(i), Some(v), "entry {i}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn mixed_sentinel_and_real_values() {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let path = dir.path().join("test.pent");
|
||||||
|
let mut builder = FamilyScalarAnnexBuilder::new(3, &path, E, false).unwrap();
|
||||||
|
builder.set(1, 2.0);
|
||||||
|
builder.close().unwrap();
|
||||||
|
let annex = FamilyScalarAnnex::open(&path, E).unwrap();
|
||||||
|
assert_eq!(annex.get(0), None);
|
||||||
|
assert_eq!(annex.get(1), Some(2.0));
|
||||||
|
assert_eq!(annex.get(2), None);
|
||||||
|
assert_eq!(annex.len(), 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn counting_convention_roundtrips_through_the_header() {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
for kind in [ScalarKind::Entropy, ScalarKind::PairWeight] {
|
||||||
|
for no_ambiguity in [false, true] {
|
||||||
|
let path = dir.path().join(kind.file_name(no_ambiguity));
|
||||||
|
FamilyScalarAnnexBuilder::new(2, &path, kind, no_ambiguity).unwrap().close().unwrap();
|
||||||
|
assert_eq!(FamilyScalarAnnex::open(&path, kind).unwrap().no_ambiguity(), no_ambiguity);
|
||||||
|
}
|
||||||
|
assert_ne!(kind.file_name(false), kind.file_name(true));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn one_kind_cannot_be_read_as_the_other() {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let path = dir.path().join("w.ppwt");
|
||||||
|
FamilyScalarAnnexBuilder::new(1, &path, ScalarKind::PairWeight, false).unwrap().close().unwrap();
|
||||||
|
assert!(FamilyScalarAnnex::open(&path, ScalarKind::Entropy).is_err());
|
||||||
|
assert!(FamilyScalarAnnex::open(&path, ScalarKind::PairWeight).is_ok());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn file_with_zero_flags_reads_as_ambiguity_counted() {
|
||||||
|
// files written before the flag existed carried four zero bytes there
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let path = dir.path().join("old.pent");
|
||||||
|
FamilyScalarAnnexBuilder::new(1, &path, E, false).unwrap().close().unwrap();
|
||||||
|
let bytes = std::fs::read(&path).unwrap();
|
||||||
|
assert_eq!(&bytes[4..8], &[0u8; 4]);
|
||||||
|
assert!(!FamilyScalarAnnex::open(&path, E).unwrap().no_ambiguity());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -34,3 +34,15 @@ pub(super) fn triangle_index(n: usize, i: usize, j: usize) -> usize {
|
|||||||
debug_assert!(j < n && i != j);
|
debug_assert!(j < n && i != j);
|
||||||
(i * (2 * n - i - 1)) / 2 + (j - i - 1)
|
(i * (2 * n - i - 1)) / 2 + (j - i - 1)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether a genome with this per-family `mask` (bit `b` set iff it carries
|
||||||
|
/// the member whose central base is `b`) counts as carrying the family, under
|
||||||
|
/// the `--no-ambiguity` counting convention: any non-empty state does, except
|
||||||
|
/// — with `no_ambiguity` — a genome carrying several members, which is
|
||||||
|
/// missing data in the alignment and so shares nothing there. One definition
|
||||||
|
/// for every annex that counts carriers (family overlap, pair weight,
|
||||||
|
/// quartet), so they can never disagree.
|
||||||
|
#[inline]
|
||||||
|
pub(crate) fn is_carrier(mask: u8, no_ambiguity: bool) -> bool {
|
||||||
|
mask != 0 && !(no_ambiguity && mask.count_ones() > 1)
|
||||||
|
}
|
||||||
@@ -20,17 +20,21 @@
|
|||||||
//! reverse.
|
//! reverse.
|
||||||
|
|
||||||
pub mod algorithms;
|
pub mod algorithms;
|
||||||
mod entropy_annex;
|
|
||||||
pub mod extensions;
|
pub mod extensions;
|
||||||
mod family_overlap;
|
mod family_overlap;
|
||||||
|
mod family_scalar_annex;
|
||||||
mod helpers;
|
mod helpers;
|
||||||
pub mod iter;
|
pub mod iter;
|
||||||
|
mod quartet_annex;
|
||||||
mod siblingannex;
|
mod siblingannex;
|
||||||
|
|
||||||
pub(crate) use entropy_annex::ENTROPY_ANNEX_FILE_NAME;
|
pub(crate) use family_overlap::{
|
||||||
pub use entropy_annex::{EntropyAnnex, EntropyAnnexBuilder};
|
FAMILY_OVERLAP_FILE_NAMES, FamilyOverlapAccumulator, family_overlap_file_name,
|
||||||
pub(crate) use family_overlap::{FAMILY_OVERLAP_FILE_NAME, FamilyOverlapAccumulator};
|
};
|
||||||
|
pub(crate) use family_scalar_annex::ScalarKind;
|
||||||
|
pub use family_scalar_annex::{FamilyScalarAnnex, FamilyScalarAnnexBuilder};
|
||||||
pub use family_overlap::FamilyOverlap;
|
pub use family_overlap::FamilyOverlap;
|
||||||
|
pub(crate) use quartet_annex::QUARTET_ANNEX_FILE_NAMES;
|
||||||
pub use iter::{
|
pub use iter::{
|
||||||
MinorantBatchIter, MinorantIter, SiblingBatchIter, SiblingEntry, SiblingIter, SiblingLayerExt,
|
MinorantBatchIter, MinorantIter, SiblingBatchIter, SiblingEntry, SiblingIter, SiblingLayerExt,
|
||||||
};
|
};
|
||||||
@@ -38,7 +42,8 @@ pub use siblingannex::{FamilyMask, SiblingAnnex, SiblingAnnexBuilder};
|
|||||||
|
|
||||||
pub use algorithms::{
|
pub use algorithms::{
|
||||||
AlignmentReducer, BasePairTally, CardinalityTally, EntropyBias, GammaShape, LayerReducer,
|
AlignmentReducer, BasePairTally, CardinalityTally, EntropyBias, GammaShape, LayerReducer,
|
||||||
PairwiseReducer, RawSnpDistanceOutput, SankoffBundle, SiblingAnnexStats, SnpAlignment,
|
PairwiseReducer, RawSnpDistanceOutput, SamplingSpec, SankoffBundle, SiblingAnnexStats,
|
||||||
|
SnpAlignment,
|
||||||
SnpDistanceKind, cardinality_transition_probs, composition_transition_probs, distance_matrix,
|
SnpDistanceKind, cardinality_transition_probs, composition_transition_probs, distance_matrix,
|
||||||
pairwise_cost_matrix, sampled_scan,
|
pairwise_cost_matrix, sampled_scan,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
//! Persisted, per-layer quartet-informativeness flag: one bit per
|
||||||
|
//! non-monomorphic minorant family, stored as a plain
|
||||||
|
//! [`obicompactvec::PersistentBitVec`] (no format of its own) and numbered
|
||||||
|
//! by the same compact ordinal as [`crate::siblings::EntropyAnnex`]
|
||||||
|
//! (`algorithms::non_monomorphic_reindex_layer`), never by
|
||||||
|
//! `scan_layer_families`'s own `family_idx`.
|
||||||
|
//!
|
||||||
|
//! A family is quartet-informative when its carrier genomes show at least
|
||||||
|
//! two distinct called states, each carried by at least two genomes: the
|
||||||
|
//! pattern that can separate two pairs of genomes from each other
|
||||||
|
//! (`AB|CD`). Anything weaker — a singleton variant, or a family carried by
|
||||||
|
//! fewer than four genomes — fits every topology and tells a tree nothing.
|
||||||
|
//! See `algorithms::masking::is_quartet_informative` for the exact rule.
|
||||||
|
//!
|
||||||
|
//! Which genomes count as carriers depends on `--no-ambiguity`, so each
|
||||||
|
//! convention has its own file, and **the file name is the parameter**: a
|
||||||
|
//! `PersistentBitVec` header has no room for a flag, and changing it would
|
||||||
|
//! change a shared structure for the sake of one annex.
|
||||||
|
//!
|
||||||
|
//! The flag is computed over every genome of the index. A later
|
||||||
|
//! `--exclude-genome` can only remove carriers, never create a bipartition,
|
||||||
|
//! so a family flagged `false` here is also uninformative once genomes are
|
||||||
|
//! excluded: the annex is a safe pre-filter, and the exact check
|
||||||
|
//! (`algorithms::masking::survives_quartet`) still runs on the resolved
|
||||||
|
//! family.
|
||||||
|
|
||||||
|
pub(crate) const QUARTET_ANNEX_FILE_NAME: &str = "quartet.pbiv";
|
||||||
|
|
||||||
|
/// Same annex, built with `no_ambiguity == true`.
|
||||||
|
pub(crate) const QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME: &str = "quartet.noamb.pbiv";
|
||||||
|
|
||||||
|
/// Every quartet-annex file name, whatever its counting convention — for
|
||||||
|
/// invalidating them all together when the sibling annex they were numbered
|
||||||
|
/// against is rebuilt.
|
||||||
|
pub(crate) const QUARTET_ANNEX_FILE_NAMES: [&str; 2] =
|
||||||
|
[QUARTET_ANNEX_FILE_NAME, QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME];
|
||||||
|
|
||||||
|
/// File name of the quartet annex built under the given `--no-ambiguity`
|
||||||
|
/// convention.
|
||||||
|
pub(crate) fn quartet_annex_file_name(no_ambiguity: bool) -> &'static str {
|
||||||
|
if no_ambiguity {
|
||||||
|
QUARTET_ANNEX_NO_AMBIGUITY_FILE_NAME
|
||||||
|
} else {
|
||||||
|
QUARTET_ANNEX_FILE_NAME
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_two_conventions_never_share_a_file() {
|
||||||
|
assert_ne!(quartet_annex_file_name(false), quartet_annex_file_name(true));
|
||||||
|
for no_ambiguity in [false, true] {
|
||||||
|
assert!(QUARTET_ANNEX_FILE_NAMES.contains(&quartet_annex_file_name(no_ambiguity)));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user