feat: add threshold-based Hamming distance and refine phylo documentation
Add `threshold_hamming_dist_matrix` to calculate k-mers present in exactly one genome using union minus intersection. Update `DistanceArgs` and `distance.rs` to support this metric on count indexes via `--presence-threshold`, removing the previous restriction that hamming was only for presence/absence indexes. - Add `threshold_hamming_dist_matrix` method to `CountPartials` in `src/obikmer/src/traits.rs` - Implement new test `threshold_hamming_counts_kmers_present_in_one_genome_only` in `src/obicompactvec/src/tests/intmatrix.rs` - Update `DistanceArgs` in `src/obikmer/src/cmd/phylo/args/distance.rs` to clarify threshold effects on Jaccard/Mash/Hamming - Modify `distance` function in `src/obikphylo/src/distance.rs` to support Hamming on count indexes and use pre-computed bit structures - Update `obicompactvec.md` documentation with new function details and partial result explanations - Refine `index.md` descriptions for `phylo`, `unitig`, and `select` options
This commit is contained in:
1 parent
c4eacb3f72
commit
c93616b432
6 files changed
+33
-13
No files matched your search
@@ -509,6 +509,7 @@ Provided finalisations:
|
|||||||
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
|
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
|
||||||
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
|
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
|
||||||
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
|
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
|
||||||
|
| `threshold_hamming_dist_matrix(t)` | `union[i,j] − inter[i,j]` (k-mers present in exactly one genome) — no separate partial |
|
||||||
|
|
||||||
### BitPartials
|
### BitPartials
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -13,7 +13,7 @@
|
|||||||
| `query` | Query an index with sequences and annotate matches |
|
| `query` | Query an index with sequences and annotate matches |
|
||||||
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
|
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
|
||||||
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
|
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
|
||||||
| `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash on count indexes; `hamming` is currently refused on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
|
| `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash/Hamming on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
|
||||||
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
|
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
|
||||||
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
|
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
|
||||||
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
|
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
|
||||||
|
|||||||
@@ -249,6 +249,20 @@ fn partial_threshold_jaccard_consistent() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn threshold_hamming_counts_kmers_present_in_one_genome_only() {
|
||||||
|
// At threshold 2 the genomes (columns) hold the k-mers (rows):
|
||||||
|
// c0 = {r0, r2}, c1 = {r1}, c2 = {r1}.
|
||||||
|
let (_d, m) = make_matrix(&[&[3, 0, 2], &[1, 2, 0], &[0, 3, 1]]);
|
||||||
|
let h = m.threshold_hamming_dist_matrix(2);
|
||||||
|
let expected = [[0, 3, 3], [3, 0, 0], [3, 0, 0]];
|
||||||
|
for i in 0..3 {
|
||||||
|
for j in 0..3 {
|
||||||
|
assert_eq!(h[[i, j]], expected[i][j], "[{i},{j}]");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn single_col_distance_matrix_is_zero() {
|
fn single_col_distance_matrix_is_zero() {
|
||||||
let (_d, m) = make_matrix(&[&[1, 2, 3]]);
|
let (_d, m) = make_matrix(&[&[1, 2, 3]]);
|
||||||
|
|||||||
@@ -129,6 +129,16 @@ pub trait CountPartials: ColumnWeights {
|
|||||||
jaccard_to_mash(&self.threshold_jaccard_dist_matrix(threshold), k)
|
jaccard_to_mash(&self.threshold_jaccard_dist_matrix(threshold), k)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Hamming distance on presence/absence: the number of k-mers present
|
||||||
|
/// (count ≥ `threshold`) in exactly one of the two genomes, i.e. the union
|
||||||
|
/// minus the intersection that [`partial_threshold_jaccard`] already gives.
|
||||||
|
///
|
||||||
|
/// [`partial_threshold_jaccard`]: Self::partial_threshold_jaccard
|
||||||
|
fn threshold_hamming_dist_matrix(&self, threshold: u32) -> Array2<u64> {
|
||||||
|
let (inter, union) = self.partial_threshold_jaccard(threshold);
|
||||||
|
union - inter
|
||||||
|
}
|
||||||
|
|
||||||
fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
|
fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
|
||||||
let global = self.col_weights();
|
let global = self.col_weights();
|
||||||
let mut m = self.partial_relfreq_bray(&global).mapv(|v| 1.0 - v);
|
let mut m = self.partial_relfreq_bray(&global).mapv(|v| 1.0 - v);
|
||||||
|
|||||||
@@ -21,8 +21,7 @@ pub struct DistanceArgs {
|
|||||||
/// mash Mash distance, a divergence estimate derived
|
/// mash Mash distance, a divergence estimate derived
|
||||||
/// from the Jaccard index
|
/// from the Jaccard index
|
||||||
/// hamming number of k-mers present in one genome and
|
/// hamming number of k-mers present in one genome and
|
||||||
/// absent from the other (only for indexes
|
/// absent from the other
|
||||||
/// without counts)
|
|
||||||
/// The others compare k-mer counts, and need an index that stores them:
|
/// The others compare k-mer counts, and need an index that stores them:
|
||||||
/// bray-curtis Bray-Curtis dissimilarity on k-mer counts
|
/// bray-curtis Bray-Curtis dissimilarity on k-mer counts
|
||||||
/// relfreq-bray-curtis Bray-Curtis dissimilarity on counts divided
|
/// relfreq-bray-curtis Bray-Curtis dissimilarity on counts divided
|
||||||
@@ -56,9 +55,9 @@ pub struct DistanceArgs {
|
|||||||
|
|
||||||
/// Minimum k-mer count to count as present
|
/// Minimum k-mer count to count as present
|
||||||
///
|
///
|
||||||
/// Converts counts into presence/absence for the Jaccard and Mash
|
/// Converts counts into presence/absence for the Jaccard, Mash and
|
||||||
/// distances and for the shared k-mer counts, when the index stores
|
/// Hamming distances and for the shared k-mer counts, when the index
|
||||||
/// counts: a k-mer is present in a genome only when its count reaches
|
/// stores counts: a k-mer is present in a genome only when its count reaches
|
||||||
/// this threshold. It has no effect on indexes without counts, nor on
|
/// this threshold. It has no effect on indexes without counts, nor on
|
||||||
/// the distances that use the counts themselves.
|
/// the distances that use the counts themselves.
|
||||||
#[arg(long, default_value = "1")]
|
#[arg(long, default_value = "1")]
|
||||||
|
|||||||
@@ -214,8 +214,8 @@ impl Metrics for IndexCache {
|
|||||||
let bit = AggregatedBit { cache: self, n_genomes };
|
let bit = AggregatedBit { cache: self, n_genomes };
|
||||||
|
|
||||||
// One construction decides both what the index can serve and how:
|
// One construction decides both what the index can serve and how:
|
||||||
// the six count-based distances need a count index, Hamming needs a
|
// the six count-based distances need a count index; Jaccard, Mash and
|
||||||
// presence/absence one, Jaccard and Mash work on either.
|
// Hamming work on either (on counts, through the presence threshold).
|
||||||
let matrix = match (use_counts, metric) {
|
let matrix = match (use_counts, metric) {
|
||||||
(true, BrayCurtis) => count.bray_dist_matrix(),
|
(true, BrayCurtis) => count.bray_dist_matrix(),
|
||||||
(true, RelfreqBrayCurtis) => count.relfreq_bray_dist_matrix(),
|
(true, RelfreqBrayCurtis) => count.relfreq_bray_dist_matrix(),
|
||||||
@@ -225,11 +225,7 @@ impl Metrics for IndexCache {
|
|||||||
(true, HellingerEuclidean) => count.hellinger_euclidean_dist_matrix(),
|
(true, HellingerEuclidean) => count.hellinger_euclidean_dist_matrix(),
|
||||||
(true, Jaccard) => count.threshold_jaccard_dist_matrix(presence_threshold),
|
(true, Jaccard) => count.threshold_jaccard_dist_matrix(presence_threshold),
|
||||||
(true, Mash) => count.threshold_mash_dist_matrix(kmer_size, presence_threshold),
|
(true, Mash) => count.threshold_mash_dist_matrix(kmer_size, presence_threshold),
|
||||||
(true, Hamming) => {
|
(true, Hamming) => count.threshold_hamming_dist_matrix(presence_threshold).mapv(|v| v as f64),
|
||||||
return Err(OKIError::InvalidInput(
|
|
||||||
"Hamming is only available for presence/absence indexes".into(),
|
|
||||||
));
|
|
||||||
}
|
|
||||||
(false, Jaccard) => bit.jaccard_dist_matrix(),
|
(false, Jaccard) => bit.jaccard_dist_matrix(),
|
||||||
(false, Mash) => bit.mash_dist_matrix(kmer_size),
|
(false, Mash) => bit.mash_dist_matrix(kmer_size),
|
||||||
(false, Hamming) => bit.hamming_dist_matrix().mapv(|v| v as f64),
|
(false, Hamming) => bit.hamming_dist_matrix().mapv(|v| v as f64),
|
||||||
|
|||||||
Reference in new issue
Block a user