Add `threshold_hamming_dist_matrix` to calculate k-mers present in exactly one genome using union minus intersection. Update `DistanceArgs` and `distance.rs` to support this metric on count indexes via `--presence-threshold`, removing the previous restriction that hamming was only for presence/absence indexes. - Add `threshold_hamming_dist_matrix` method to `CountPartials` in `src/obikmer/src/traits.rs` - Implement new test `threshold_hamming_counts_kmers_present_in_one_genome_only` in `src/obicompactvec/src/tests/intmatrix.rs` - Update `DistanceArgs` in `src/obikmer/src/cmd/phylo/args/distance.rs` to clarify threshold effects on Jaccard/Mash/Hamming - Modify `distance` function in `src/obikphylo/src/distance.rs` to support Hamming on count indexes and use pre-computed bit structures - Update `obicompactvec.md` documentation with new function details and partial result explanations - Refine `index.md` descriptions for `phylo`, `unitig`, and `select` options
5.1 KiB
5.1 KiB
obikmer
obikmer is a Rust tool for manipulation, counting, indexing, and set operations on DNA sequences represented as kmer sets.
Subcommands
| Subcommand | Purpose |
|---|---|
superkmer |
Extract super-kmers from a sequence file and write to stdout |
index |
Build a complete genome index (scatter → dereplicate → count → layered MPHF) |
merge |
Merge multiple built indexes into one |
filter |
Apply row-level selection (σ) to an index: retain only k-mers matching the ingroup/outgroup predicates. Output is a new single-layer index — compaction is a consequence, not the goal. Supports the shared kmer filtering system |
query |
Query an index with sequences and annotate matches |
dump |
Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared kmer filtering system; --head N limits output to the first N k-mers |
annotate |
Add or update genome metadata from a CSV file; or dump metadata as CSV |
phylo |
Genome-vs-genome distances, trees and SNP-based pseudo-alignments. --distance selects either a whole-genome distance computed from k-mer content (jaccard, mash, hamming, bray-curtis, relfreq-bray-curtis, euclidean, relfreq-euclidean, hellinger, hellinger-euclidean; --presence-threshold N converts counts to presence for Jaccard/Mash/Hamming on count indexes) or a snp-* distance (snp-raw, snp-jc, snp-k2p, snp-k81, snp-f81, snp-t92, snp-tn93, snp-tv; optional --gamma-shape) computed from central-position SNPs of k-mer families. --nj/--upgma build trees (Jaccard if no --distance); --shared-kmers writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (--sibling-annex forces it), with site selection/sampling options (--n-sites, --quartet, --pair-balance, --entropy, --free-loss, --no-ambiguity/--only-diploid, --session), a pseudo-alignment export (--pseudo-alignment, --fasta, --vcf) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See evolutionary distances |
unitig |
Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared kmer filtering system |
select |
Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of filter (see select) |
estimate |
Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
convert |
Convert an index's evidence representation (exact / approximate / hybrid), in place |
utils |
Miscellaneous index utilities: --new-label NEW=OLD renames a genome label; --upgrade-index adds missing layer_meta.json to old indexes |
pack |
Pack per-column matrix files into single-file format to reduce query I/O |
Constraints
- Target scale: individual genome datasets, tens of Gbases
- Maximum efficiency in computation, memory, and disk usage
- k odd, k ∈ [11, 31], fixed at runtime; kmer fits in a u64 (2 bits/base)
- Canonical form:
min(kmer, revcomp(kmer))reduces strand-symmetric space by half - Input formats for
index/superkmer: FASTA (.fa,.fasta), FASTQ (.fq,.fastq), GenBank flat file (.gb,.gbk,.gbff), all optionally gzip-compressed; directories expanded recursively; streaming stdin via- - Input formats for
query: FASTA, FASTQ, optionally gzip-compressed; streaming stdin via-
Parameter constraints (enforced at CLI)
All constraints below are checked by CommonArgs::validate() at the start of superkmer and index. Invalid values exit immediately with an error.
| Parameter | Constraint | Reason |
|---|---|---|
k (--kmer-size) |
odd | even k allows palindromic k-mers: kmer == revcomp(kmer), breaking the canonical form invariant |
k (--kmer-size) |
k ∈ [11, 31] | k > 31 overflows u64 at 2 bits/base; k < 11 gives insufficient specificity |
m (--minimizer-size) |
odd | same palindrome argument as k |
m (--minimizer-size) |
3 ≤ m ≤ k−1 | minimizer must be strictly shorter than the kmer |
z (-z, Findere, index --approx only) |
z ≤ k−1 | effective indexed kmer size is k−z+1; z ≥ k would make it ≤ 0 |
Genome label constraints
Genome labels are arbitrary Unicode strings with the following restrictions:
| Character | Forbidden | Reason |
|---|---|---|
/ |
yes | filesystem path separator |
= |
yes | --new-label parser separator |
\0 |
yes | null byte |
\n \r \t |
yes | break CSV output |
| spaces | allowed | use shell quoting: --new-label 'new label=old label' |
Empty labels are also rejected. Labels derived automatically from the index directory name (when --label is omitted) are not validated since they come from the filesystem and are already safe.
Priority operations
- Kmer counting (frequencies)
- Fast search / query
- Set operations: union, intersection, difference