feat: add Mash distance metric and rolling entropy support
Implement the Mash distance metric across the CLI, index, and compact vector traits. This includes adding a `Mash` variant to the `DistanceMetric` enum and `MetricArg` CLI argument, implementing the conversion from Jaccard distances using the standard mutation-rate estimator formula, and updating documentation with supported metrics and algorithmic references. Additionally, add an `entropy` method to rolling statistics for computing order-specific entropy.
This commit is contained in:
@@ -347,11 +347,24 @@ Provided finalisations:
|
||||
| `relfreq_euclidean_dist_matrix()` | `√partial_relfreq_euclidean[i,j]` |
|
||||
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
|
||||
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
|
||||
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
|
||||
|
||||
### BitPartials
|
||||
|
||||
Required: `partial_jaccard() -> (Array2<u64>, Array2<u64>)`, `partial_hamming() -> Array2<u64>`. Both additive across layers and partitions.
|
||||
|
||||
Provided finalisations also include `jaccard_dist_matrix()`, `hamming_dist_matrix()`, and `mash_dist_matrix(k)`.
|
||||
|
||||
### Mash distance
|
||||
|
||||
`mash_dist_matrix`/`threshold_mash_dist_matrix` add no new additive primitive: both are a pointwise transform of the existing Jaccard distance matrix, per the Mash mutation-rate estimator [@Mash-distances-doc; @Fan2015-mash-formula]:
|
||||
|
||||
```
|
||||
D = -1/k · ln(2J / (1+J)), J = 1 - d_jaccard
|
||||
```
|
||||
|
||||
`J ≤ 0` (i.e. `d_jaccard ≥ 1`, no shared k-mers) maps to `D = 1` (maximal distance) rather than the `ln` singularity at `J = 0`.
|
||||
|
||||
---
|
||||
|
||||
## Temp-file-backed types
|
||||
|
||||
+1
-1
@@ -13,7 +13,7 @@
|
||||
| `query` | Query an index with sequences and annotate matches |
|
||||
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
|
||||
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
|
||||
| `distance` | Compute pairwise distance matrix between genomes; optionally build NJ/UPGMA trees; `--presence-threshold N` sets the minimum count to consider a k-mer present when computing Jaccard on count indexes (default 1) |
|
||||
| `distance` | Compute pairwise distance matrix between genomes (`--metric jaccard\|mash\|hamming\|bray-curtis\|relfreq-bray-curtis\|euclidean\|relfreq-euclidean\|hellinger\|hellinger-euclidean`); optionally build NJ/UPGMA trees; `--presence-threshold N` sets the minimum count to consider a k-mer present when computing Jaccard/Mash on count indexes (default 1) |
|
||||
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
|
||||
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
|
||||
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
|
||||
|
||||
@@ -241,3 +241,21 @@
|
||||
volume = 33,
|
||||
year = 2017,
|
||||
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btw832}}
|
||||
|
||||
@misc{Mash-distances-doc,
|
||||
author = {{Marbl Lab}},
|
||||
howpublished = {Mash documentation},
|
||||
title = {Mash Distance},
|
||||
url = {https://mash.readthedocs.io/en/latest/distances.html},
|
||||
urldate = {2026-07-09},
|
||||
year = 2026}
|
||||
|
||||
@article{Fan2015-mash-formula,
|
||||
author = {Fan, Huan and Ives, Anthony R and Surget-Groba, Yann and Cannon, Charles H},
|
||||
doi = {10.1186/s12864-015-1647-5},
|
||||
journal = {BMC Genomics},
|
||||
number = 1,
|
||||
title = {An assembly and alignment-free method of phylogeny reconstruction from next-generation sequencing data},
|
||||
url = {https://doi.org/10.1186/s12864-015-1647-5},
|
||||
volume = 16,
|
||||
year = 2015}
|
||||
|
||||
Reference in New Issue
Block a user