Refactor the `phylo` command to support explicit distance metrics (Jaccard, Hamming) and SNP-based pseudo-alignments. Replace implicit metric selection with explicit requirements, such as requiring `--n-sites` for SNP analysis. Add a new `shared-kmers` output option and update documentation for performance characteristics of sparse vs dense formats. - Update `phylo` CLI to support genome-vs-genome distances using k-mer content or SNPs, including options for NJ/UPGMA tree building and data export - Replace `reindex` command with `convert` for in-place index evidence conversion (exact/approx/hybrid) and add `--upgrade-index` to `utils` for missing layer metadata - Refactor `distance_to_compute` logic to remove implicit Jaccard implications for `--nj`/`--upgma` unless `--distance` is specified, and enforce `--n-sites` for `--pseudo-alignment`/`--sankoff` - Add `shared_kmers` method to `Metrics` trait in `src/obikphylo/src/distance.rs` and new `write_shared_kmers` function in `src/obikmer/src/cmd/phylo/shared.rs` for CSV export - Update `PackArgs::dense` documentation to clarify that dense format is faster for column-oriented distance matrices while sparse is better for single-row access - Remove `requires_counts` definition and update assertions in `values.rs` to reflect new distance computation signatures and metric requirements
64 lines
5.1 KiB
Markdown
64 lines
5.1 KiB
Markdown
# obikmer
|
|
|
|
`obikmer` is a Rust tool for manipulation, counting, indexing, and set operations on DNA sequences represented as kmer sets.
|
|
|
|
## Subcommands
|
|
|
|
| Subcommand | Purpose |
|
|
|-------------|---------|
|
|
| `superkmer` | Extract super-kmers from a sequence file and write to stdout |
|
|
| `index` | Build a complete genome index (scatter → dereplicate → count → layered MPHF) |
|
|
| `merge` | Merge multiple built indexes into one |
|
|
| `filter` | Apply row-level selection (σ) to an index: retain only k-mers matching the ingroup/outgroup predicates. Output is a new single-layer index — compaction is a consequence, not the goal. Supports the shared [kmer filtering](implementation/filtering.md) system |
|
|
| `query` | Query an index with sequences and annotate matches |
|
|
| `dump` | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
|
|
| `annotate` | Add or update genome metadata from a CSV file; or dump metadata as CSV |
|
|
| `phylo` | Genome-vs-genome distances, trees and SNP-based pseudo-alignments. `--distance` selects either a whole-genome distance computed from k-mer content (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`; `--presence-threshold N` converts counts to presence for Jaccard/Mash on count indexes; `hamming` is currently refused on count indexes) or a `snp-*` distance (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv`; optional `--gamma-shape`) computed from central-position SNPs of k-mer families. `--nj`/`--upgma` build trees (Jaccard if no `--distance`); `--shared-kmers` writes the shared-k-mer count matrix on its own. SNP analyses read the sibling annex, built on demand (`--sibling-annex` forces it), with site selection/sampling options (`--n-sites`, `--quartet`, `--pair-balance`, `--entropy`, `--free-loss`, `--no-ambiguity`/`--only-diploid`, `--session`), a pseudo-alignment export (`--pseudo-alignment`, `--fasta`, `--vcf`) and Sankoff cost calibration with TNT/PhyG/IQ-TREE exports. See [evolutionary distances](theory/evolutionary_distances.md) |
|
|
| `unitig` | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
|
|
| `select` | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
|
|
| `estimate` | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
|
|
| `convert` | Convert an index's evidence representation (exact / approximate / hybrid), in place |
|
|
| `utils` | Miscellaneous index utilities: `--new-label NEW=OLD` renames a genome label; `--upgrade-index` adds missing `layer_meta.json` to old indexes |
|
|
| `pack` | Pack per-column matrix files into single-file format to reduce query I/O |
|
|
|
|
## Constraints
|
|
|
|
- Target scale: individual genome datasets, tens of Gbases
|
|
- Maximum efficiency in computation, memory, and disk usage
|
|
- k odd, k ∈ [11, 31], fixed at runtime; kmer fits in a u64 (2 bits/base)
|
|
- Canonical form: `min(kmer, revcomp(kmer))` reduces strand-symmetric space by half
|
|
- Input formats for `index`/`superkmer`: FASTA (`.fa`, `.fasta`), FASTQ (`.fq`, `.fastq`), GenBank flat file (`.gb`, `.gbk`, `.gbff`), all optionally gzip-compressed; directories expanded recursively; streaming stdin via `-`
|
|
- Input formats for `query`: FASTA, FASTQ, optionally gzip-compressed; streaming stdin via `-`
|
|
|
|
## Parameter constraints (enforced at CLI)
|
|
|
|
All constraints below are checked by `CommonArgs::validate()` at the start of `superkmer` and `index`. Invalid values exit immediately with an error.
|
|
|
|
| Parameter | Constraint | Reason |
|
|
|-----------|-----------|--------|
|
|
| k (`--kmer-size`) | odd | even k allows palindromic k-mers: kmer == revcomp(kmer), breaking the canonical form invariant |
|
|
| k (`--kmer-size`) | k ∈ [11, 31] | k > 31 overflows u64 at 2 bits/base; k < 11 gives insufficient specificity |
|
|
| m (`--minimizer-size`) | odd | same palindrome argument as k |
|
|
| m (`--minimizer-size`) | 3 ≤ m ≤ k−1 | minimizer must be strictly shorter than the kmer |
|
|
| z (`-z`, Findere, `index --approx` only) | z ≤ k−1 | effective indexed kmer size is k−z+1; z ≥ k would make it ≤ 0 |
|
|
|
|
## Genome label constraints
|
|
|
|
Genome labels are arbitrary Unicode strings with the following restrictions:
|
|
|
|
| Character | Forbidden | Reason |
|
|
|-----------|-----------|--------|
|
|
| `/` | yes | filesystem path separator |
|
|
| `=` | yes | `--new-label` parser separator |
|
|
| `\0` | yes | null byte |
|
|
| `\n` `\r` `\t` | yes | break CSV output |
|
|
| spaces | **allowed** | use shell quoting: `--new-label 'new label=old label'` |
|
|
|
|
Empty labels are also rejected. Labels derived automatically from the index directory name (when `--label` is omitted) are not validated since they come from the filesystem and are already safe.
|
|
|
|
## Priority operations
|
|
|
|
- Kmer counting (frequencies)
|
|
- Fast search / query
|
|
- Set operations: union, intersection, difference
|