Files
obikmer/DevDocMD/index.md
T
Eric Coissac c4eacb3f72 feat(phylo): expand genome distance metrics and refactor CLI arguments
Refactor the `phylo` command to support explicit distance metrics (Jaccard, Hamming) and SNP-based pseudo-alignments. Replace implicit metric selection with explicit requirements, such as requiring `--n-sites` for SNP analysis. Add a new `shared-kmers` output option and update documentation for performance characteristics of sparse vs dense formats.

- Update `phylo` CLI to support genome-vs-genome distances using k-mer content or SNPs, including options for NJ/UPGMA tree building and data export
- Replace `reindex` command with `convert` for in-place index evidence conversion (exact/approx/hybrid) and add `--upgrade-index` to `utils` for missing layer metadata
- Refactor `distance_to_compute` logic to remove implicit Jaccard implications for `--nj`/`--upgma` unless `--distance` is specified, and enforce `--n-sites` for `--pseudo-alignment`/`--sankoff`
- Add `shared_kmers` method to `Metrics` trait in `src/obikphylo/src/distance.rs` and new `write_shared_kmers` function in `src/obikmer/src/cmd/phylo/shared.rs` for CSV export
- Update `PackArgs::dense` documentation to clarify that dense format is faster for column-oriented distance matrices while sparse is better for single-row access
- Remove `requires_counts` definition and update assertions in `values.rs` to reflect new distance computation signatures and metric requirements
2026-11-19 07:11:44 +01:00

5.1 KiB

obikmer

obikmer is a Rust tool for manipulation, counting, indexing, and set operations on DNA sequences represented as kmer sets.

Subcommands

Subcommand Purpose
superkmer Extract super-kmers from a sequence file and write to stdout
index Build a complete genome index (scatter → dereplicate → count → layered MPHF)
merge Merge multiple built indexes into one
filter Apply row-level selection (σ) to an index: retain only k-mers matching the ingroup/outgroup predicates. Output is a new single-layer index — compaction is a consequence, not the goal. Supports the shared kmer filtering system
query Query an index with sequences and annotate matches
dump Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared kmer filtering system; --head N limits output to the first N k-mers
annotate Add or update genome metadata from a CSV file; or dump metadata as CSV
phylo Genome-vs-genome distances, trees and SNP-based pseudo-alignments. --distance selects either a whole-genome distance computed from k-mer content (jaccard, mash, hamming, bray-curtis, relfreq-bray-curtis, euclidean, relfreq-euclidean, hellinger, hellinger-euclidean; --presence-threshold N converts counts to presence for Jaccard/Mash on count indexes; hamming is currently refused on count indexes) or a snp-* distance (snp-raw, snp-jc, snp-k2p, snp-k81, snp-f81, snp-t92, snp-tn93, snp-tv; optional --gamma-shape) computed from central-position SNPs of k-mer families. --nj/--upgma build trees (Jaccard if no --distance); --shared-kmers writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (--sibling-annex forces it), with site selection/sampling options (--n-sites, --quartet, --pair-balance, --entropy, --free-loss, --no-ambiguity/--only-diploid, --session), a pseudo-alignment export (--pseudo-alignment, --fasta, --vcf) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See evolutionary distances
unitig Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared kmer filtering system
select Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of filter (see select)
estimate Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing
convert Convert an index's evidence representation (exact / approximate / hybrid), in place
utils Miscellaneous index utilities: --new-label NEW=OLD renames a genome label; --upgrade-index adds missing layer_meta.json to old indexes
pack Pack per-column matrix files into single-file format to reduce query I/O

Constraints

  • Target scale: individual genome datasets, tens of Gbases
  • Maximum efficiency in computation, memory, and disk usage
  • k odd, k ∈ [11, 31], fixed at runtime; kmer fits in a u64 (2 bits/base)
  • Canonical form: min(kmer, revcomp(kmer)) reduces strand-symmetric space by half
  • Input formats for index/superkmer: FASTA (.fa, .fasta), FASTQ (.fq, .fastq), GenBank flat file (.gb, .gbk, .gbff), all optionally gzip-compressed; directories expanded recursively; streaming stdin via -
  • Input formats for query: FASTA, FASTQ, optionally gzip-compressed; streaming stdin via -

Parameter constraints (enforced at CLI)

All constraints below are checked by CommonArgs::validate() at the start of superkmer and index. Invalid values exit immediately with an error.

Parameter Constraint Reason
k (--kmer-size) odd even k allows palindromic k-mers: kmer == revcomp(kmer), breaking the canonical form invariant
k (--kmer-size) k ∈ [11, 31] k > 31 overflows u64 at 2 bits/base; k < 11 gives insufficient specificity
m (--minimizer-size) odd same palindrome argument as k
m (--minimizer-size) 3 ≤ m ≤ k−1 minimizer must be strictly shorter than the kmer
z (-z, Findere, index --approx only) z ≤ k−1 effective indexed kmer size is k−z+1; z ≥ k would make it ≤ 0

Genome label constraints

Genome labels are arbitrary Unicode strings with the following restrictions:

Character Forbidden Reason
/ yes filesystem path separator
= yes --new-label parser separator
\0 yes null byte
\n \r \t yes break CSV output
spaces allowed use shell quoting: --new-label 'new label=old label'

Empty labels are also rejected. Labels derived automatically from the index directory name (when --label is omitted) are not validated since they come from the filesystem and are already safe.

Priority operations

  • Kmer counting (frequencies)
  • Fast search / query
  • Set operations: union, intersection, difference