feat: add threshold-based Hamming distance and refine phylo documentation

Add `threshold_hamming_dist_matrix` to calculate k-mers present in exactly one genome using union minus intersection. Update `DistanceArgs` and `distance.rs` to support this metric on count indexes via `--presence-threshold`, removing the previous restriction that hamming was only for presence/absence indexes.

- Add `threshold_hamming_dist_matrix` method to `CountPartials` in `src/obikmer/src/traits.rs`
- Implement new test `threshold_hamming_counts_kmers_present_in_one_genome_only` in `src/obicompactvec/src/tests/intmatrix.rs`
- Update `DistanceArgs` in `src/obikmer/src/cmd/phylo/args/distance.rs` to clarify threshold effects on Jaccard/Mash/Hamming
- Modify `distance` function in `src/obikphylo/src/distance.rs` to support Hamming on count indexes and use pre-computed bit structures
- Update `obicompactvec.md` documentation with new function details and partial result explanations
- Refine `index.md` descriptions for `phylo`, `unitig`, and `select` options
This commit is contained in:
Eric Coissac committed 2026-11-19 07:11:57 +01:00
1 parent c4eacb3f72
commit c93616b432
6 files changed
+33 -13

No files matched your search

+1
View File
@@ -509,6 +509,7 @@ Provided finalisations:
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
| `threshold_hamming_dist_matrix(t)` | `union[i,j] − inter[i,j]` (k-mers present in exactly one genome) — no separate partial |
### BitPartials
+1 -1
View File
@@ -13,7 +13,7 @@
| `query` | Query an index with sequences and annotate matches |
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
| `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash on count indexes; `hamming` is currently refused on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
| `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash/Hamming on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
+14
View File
@@ -249,6 +249,20 @@ fn partial_threshold_jaccard_consistent() {
}
}
#[test]
fn threshold_hamming_counts_kmers_present_in_one_genome_only() {
// At threshold 2 the genomes (columns) hold the k-mers (rows):
// c0 = {r0, r2}, c1 = {r1}, c2 = {r1}.
let (_d, m) = make_matrix(&[&[3, 0, 2], &[1, 2, 0], &[0, 3, 1]]);
let h = m.threshold_hamming_dist_matrix(2);
let expected = [[0, 3, 3], [3, 0, 0], [3, 0, 0]];
for i in 0..3 {
for j in 0..3 {
assert_eq!(h[[i, j]], expected[i][j], "[{i},{j}]");
}
}
}
#[test]
fn single_col_distance_matrix_is_zero() {
let (_d, m) = make_matrix(&[&[1, 2, 3]]);
+10
View File
@@ -129,6 +129,16 @@ pub trait CountPartials: ColumnWeights {
jaccard_to_mash(&self.threshold_jaccard_dist_matrix(threshold), k)
}
/// Hamming distance on presence/absence: the number of k-mers present
/// (count ≥ `threshold`) in exactly one of the two genomes, i.e. the union
/// minus the intersection that [`partial_threshold_jaccard`] already gives.
///
/// [`partial_threshold_jaccard`]: Self::partial_threshold_jaccard
fn threshold_hamming_dist_matrix(&self, threshold: u32) -> Array2<u64> {
let (inter, union) = self.partial_threshold_jaccard(threshold);
union - inter
}
fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
let global = self.col_weights();
let mut m = self.partial_relfreq_bray(&global).mapv(|v| 1.0 - v);
+4 -5
View File
@@ -21,8 +21,7 @@ pub struct DistanceArgs {
/// mash Mash distance, a divergence estimate derived
/// from the Jaccard index
/// hamming number of k-mers present in one genome and
/// absent from the other (only for indexes
/// without counts)
/// absent from the other
/// The others compare k-mer counts, and need an index that stores them:
/// bray-curtis Bray-Curtis dissimilarity on k-mer counts
/// relfreq-bray-curtis Bray-Curtis dissimilarity on counts divided
@@ -56,9 +55,9 @@ pub struct DistanceArgs {
/// Minimum k-mer count to count as present
///
/// Converts counts into presence/absence for the Jaccard and Mash
/// distances and for the shared k-mer counts, when the index stores
/// counts: a k-mer is present in a genome only when its count reaches
/// Converts counts into presence/absence for the Jaccard, Mash and
/// Hamming distances and for the shared k-mer counts, when the index
/// stores counts: a k-mer is present in a genome only when its count reaches
/// this threshold. It has no effect on indexes without counts, nor on
/// the distances that use the counts themselves.
#[arg(long, default_value = "1")]
+3 -7
View File
@@ -214,8 +214,8 @@ impl Metrics for IndexCache {
let bit = AggregatedBit { cache: self, n_genomes };
// One construction decides both what the index can serve and how:
// the six count-based distances need a count index, Hamming needs a
// presence/absence one, Jaccard and Mash work on either.
// the six count-based distances need a count index; Jaccard, Mash and
// Hamming work on either (on counts, through the presence threshold).
let matrix = match (use_counts, metric) {
(true, BrayCurtis) => count.bray_dist_matrix(),
(true, RelfreqBrayCurtis) => count.relfreq_bray_dist_matrix(),
@@ -225,11 +225,7 @@ impl Metrics for IndexCache {
(true, HellingerEuclidean) => count.hellinger_euclidean_dist_matrix(),
(true, Jaccard) => count.threshold_jaccard_dist_matrix(presence_threshold),
(true, Mash) => count.threshold_mash_dist_matrix(kmer_size, presence_threshold),
(true, Hamming) => {
return Err(OKIError::InvalidInput(
"Hamming is only available for presence/absence indexes".into(),
));
}
(true, Hamming) => count.threshold_hamming_dist_matrix(presence_threshold).mapv(|v| v as f64),
(false, Jaccard) => bit.jaccard_dist_matrix(),
(false, Mash) => bit.mash_dist_matrix(kmer_size),
(false, Hamming) => bit.hamming_dist_matrix().mapv(|v| v as f64),