OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

docs: add obikmer user guide and MkDocs build configuration

Browse Source 
Introduces a comprehensive documentation set covering theoretical foundations, CLI usage, installation, and system architecture. Adds MkDocs configuration and Makefile targets to generate, serve with live reload, and clean the documentation site. Includes citation styles and bibliography files for academic references.
This commit is contained in:
Eric Coissac
2026-08-13 17:19:01 +02:00
parent 0c86ea0385
commit 6acafa7f2c
28 changed files with 1497 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
UserDocMD/architecture.md
+33
View File
@@ -0,0 +1,33 @@
# Architecture notes for advanced use
This page describes execution-level behavior relevant to sizing and running `obikmer` on large datasets or multi-socket machines. It complements the [index format](formats/index_layout.md) and [theory](theory/indexing_architecture.md) pages.
## Sequence invariant
Every input sequence is treated purely as a compact representation of a set of overlapping kmers:
- Only the `A`/`C`/`G`/`T` alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.
- Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.
- Kmers are always handled in canonical form (see [DNA encoding](theory/encoding.md)), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.
## Index dimensioning
An index directory is organized as `KmerIndex → partitions → layers`, with a canonical kmer belonging to exactly one (partition, layer) pair. This is what makes set operations (merge, filter, distance) parallel and coordination-free across partitions.
- **Partition count** (`-p`/`--partitions`, rounded up to a power of 2) is the main dimensioning knob: more partitions means more independent parallel units and a smaller working set per partition, at the cost of more open files during construction.
- **Layers** accumulate as an index grows through successive merges; per-partition query cost grows with the number of layers (worst case linear, expected constant since most kmer lookups resolve in the first layer they could plausibly be in).
- Genome columns (count or presence data) are kept at a consistent width across every layer and partition after a merge, which is what allows whole-index aggregate distances (Jaccard, Bray-Curtis, Euclidean, Hellinger, …) to be computed as a two-pass cascade (local partial sums per partition, then a global combination) with no double counting.
## Parallel execution and NUMA awareness
Partition-level work (index construction, `merge`, `filter`, `reindex`, `select`, `distance`'s sibling-annex/Sankoff computations) is dispatched by a partition runner that adapts to the machine's memory topology, detected automatically at startup via hwloc:
- On a multi-socket / multi-NUMA-node machine, one thread pool is pinned per NUMA node, and each partition is processed entirely by threads pinned to one node — keeping the memory a partition touches local to that node's DRAM. This matters because touching kmer data across NUMA nodes without pinning can degrade throughput by an order of magnitude or more on large multi-socket machines.
- On a single-socket machine, Apple Silicon, or if hwloc cannot report NUMA topology, all cores are treated as one node with no pinning and negligible overhead — this is the default behavior on macOS.
- Within a node, the number of active worker threads ramps up progressively rather than being fixed up front: it starts conservatively and grows in steps, but only as long as measured CPU efficiency or disk I/O throughput keeps improving. If neither improves after a growth step, the runner stops adding workers — avoiding oversubscription on stages that are memory-bandwidth-bound rather than CPU- or I/O-bound. Ramp speed scales with the number of cores per node, so a single-node machine ramps just as fast as a large multi-node one.
No CLI flag controls this directly; it is fully automatic at runtime. NUMA-aware pinning can be compiled out (Cargo feature `numa`, on by default), in which case a plain global thread pool is used instead.
## Kmer filtering (`filter`)
[`filter`](usage/filter.md) evaluates predicates against the genome metadata matrix directly whenever every active filter can be expressed as a column-level test (e.g. "any outgroup column non-zero"), producing a per-slot keep/drop decision without touching kmer sequence data at all. If any active filter cannot be expressed this way, evaluation falls back to a per-kmer, row-level check. Either way, the result is always written as a single, freshly compacted layer (`unitigs.bin` and the MPHF are rebuilt from the surviving kmers), never as an additional layer on top of the source index.
UserDocMD/ecology-letters.csl
+230
View File
@@ -0,0 +1,230 @@
<?xml version="1.0" encoding="utf-8"?>
<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" demote-non-dropping-particle="sort-only" default-locale="en-US">
<info>
<title>Ecology Letters</title>
<id>http://www.zotero.org/styles/ecology-letters</id>
<link href="http://www.zotero.org/styles/ecology-letters" rel="self"/>
<link href="http://www.zotero.org/styles/apa" rel="template"/>
<link href="http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291461-0248/homepage/ForAuthors.html" rel="documentation"/>
<author>
<name>David Kaplan</name>
<email>david.kaplan@ird.fr</email>
</author>
<contributor>
<name>Sebastian Karcher</name>
</contributor>
<category citation-format="author-date"/>
<category field="biology"/>
<issn>1461-023X</issn>
<eissn>1461-0248</eissn>
<updated>2023-10-11T10:45:32+00:00</updated>
<rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
</info>
<macro name="container">
<choose>
<if type="chapter paper-conference" match="any">
<text term="in" text-case="capitalize-first" suffix=": "/>
<text variable="container-title" font-style="italic"/>
<text variable="collection-title" prefix=", "/>
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
<label form="short" suffix=" "/>
<name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
</names>
</if>
<else>
<group delimiter=", ">
<text variable="container-title" font-style="italic" form="short"/>
<text variable="collection-title"/>
</group>
</else>
</choose>
</macro>
<macro name="author">
<names variable="author">
<name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
<label form="short" prefix=" (" suffix=")" text-case="capitalize-first"/>
<et-al font-style="italic"/>
<substitute>
<names variable="editor"/>
<names variable="translator"/>
<text macro="title"/>
</substitute>
</names>
</macro>
<macro name="author-short">
<names variable="author">
<name form="short" and="symbol" delimiter=", " initialize-with=". "/>
<et-al font-style="italic"/>
<substitute>
<names variable="editor"/>
<names variable="translator"/>
<choose>
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
<text variable="title" form="short" font-style="italic"/>
</if>
<else>
<text variable="title" form="short" quotes="true"/>
</else>
</choose>
</substitute>
</names>
</macro>
<macro name="access">
<choose>
<if type="webpage">
<group>
<text term="available at" text-case="capitalize-first" suffix=": "/>
<text variable="URL" suffix="."/>
</group>
<text value="Last accessed" prefix=" " suffix=" "/>
<date variable="accessed">
<date-part name="day" suffix=" "/>
<date-part name="month" suffix=" "/>
<date-part name="year"/>
</date>
</if>
</choose>
</macro>
<macro name="title">
<choose>
<if type="report" match="any">
<text variable="title" font-style="italic"/>
<group prefix=" (" suffix=")">
<text variable="genre"/>
<text variable="number" prefix=" No. "/>
</group>
</if>
<else-if type="bill book graphic legal_case legislation motion_picture report song speech" match="any">
<text variable="title" font-style="italic"/>
</else-if>
<else-if type="webpage">
<text variable="title" font-style="italic"/>
</else-if>
<else>
<text variable="title"/>
</else>
</choose>
</macro>
<macro name="publisher">
<choose>
<if type="report" match="any">
<group delimiter=", ">
<text variable="publisher"/>
<text variable="publisher-place"/>
</group>
</if>
<else>
<text variable="genre" suffix=". "/>
<group delimiter=", ">
<text variable="publisher"/>
<text variable="publisher-place"/>
</group>
</else>
</choose>
</macro>
<macro name="event">
<choose>
<if variable="event">
<text term="presented at" text-case="capitalize-first" suffix=" "/>
<text variable="event"/>
</if>
</choose>
</macro>
<macro name="issued">
<choose>
<if variable="issued">
<date variable="issued">
<date-part name="year"/>
</date>
</if>
<else-if variable="accessed">
<choose>
<if type="webpage">
<date variable="accessed">
<date-part name="year"/>
</date>
</if>
<else>
<text term="no date" form="short"/>
</else>
</choose>
</else-if>
<else>
<text term="no date" form="short"/>
</else>
</choose>
</macro>
<macro name="edition">
<choose>
<if is-numeric="edition">
<group delimiter=" ">
<number variable="edition" form="ordinal"/>
<text value="edn"/>
</group>
</if>
<else>
<text variable="edition" suffix="."/>
</else>
</choose>
</macro>
<macro name="locators">
<choose>
<if type="article-journal article-magazine article-newspaper" match="any">
<group prefix=", " delimiter=", ">
<group>
<text variable="volume"/>
</group>
<text variable="page"/>
</group>
</if>
<else-if type="bill book graphic legal_case legislation motion_picture report song thesis" match="any">
<group delimiter=". " prefix=". ">
<text macro="edition"/>
<text macro="event"/>
<text macro="publisher"/>
</group>
</else-if>
<else-if type="chapter paper-conference" match="any">
<group delimiter=", " prefix=". ">
<text macro="event"/>
<text macro="publisher"/>
<group>
<label variable="page" form="short" suffix=" "/>
<text variable="page"/>
</group>
</group>
</else-if>
</choose>
</macro>
<citation et-al-min="3" et-al-use-first="1" disambiguate-add-year-suffix="true" collapse="year-suffix" year-suffix-delimiter=", ">
<sort>
<key macro="author"/>
<key macro="issued"/>
</sort>
<layout prefix="(" suffix=")" delimiter="; ">
<group delimiter=" ">
<text macro="author-short"/>
<text macro="issued"/>
</group>
</layout>
</citation>
<bibliography et-al-min="7" et-al-use-first="6" entry-spacing="0" hanging-indent="true">
<sort>
<key macro="author"/>
<key macro="issued" sort="ascending"/>
<key macro="title"/>
</sort>
<layout>
<group suffix=".">
<text macro="author" suffix="."/>
<text macro="issued" prefix=" (" suffix="). "/>
<group delimiter=". ">
<text macro="title"/>
<text macro="container"/>
</group>
<text macro="locators"/>
<text macro="access" prefix=". "/>
</group>
</layout>
</bibliography>
</style>
UserDocMD/formats/index_layout.md
+56
View File
@@ -0,0 +1,56 @@
# Index construction and on-disk layout
## Construction pipeline
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/indexing_architecture.md)):
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition.
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.
4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set.
5. **Local assembly.** The surviving kmers of each partition are assembled into unitigs — maximal non-branching runs of a local de Bruijn graph — such that every kmer appears exactly once, at one (unitig, offset) location.
6. **MPHF and evidence construction.** A minimal perfect hash function is built over the canonical kmers of each partition, together with the evidence structure needed to verify that a queried kmer was genuinely indexed (see below). Per-genome counts or presence bits are recorded alongside if requested.
Phases 1–5 are independent per partition and run in parallel; phase 6 finalizes each partition once its kmer set is fixed.
## Minimal perfect hash function (MPHF)
Each partition's surviving kmers are mapped to a dense range of integer slots by a minimal perfect hash function: no collisions, near-optimal space (a few bits per key), O(1) lookup. Because an MPHF maps *any* input to some slot — including kmers that were never indexed — a lookup alone cannot distinguish a genuinely indexed kmer from an arbitrary one; every lookup is followed by an evidence check.
## Evidence: exact vs. approximate
Two verification modes are available, selected at build time (`index --approx`) and convertible afterwards ([`reindex`](../usage/reindex.md)):
- **Exact** (default): the hashed slot stores a pointer back into the partition's unitig data. At query time the kmer is reconstructed from that location and compared directly to the query. Zero false positives, at the cost of one extra random read per lookup.
- **Approximate** (`--approx`): the slot stores a short fingerprint (`--evidence-bits` bits) instead of a pointer; verification is a single fingerprint comparison. This trades a small, bounded false-positive rate ($1/2^b$ per kmer, reduced further to about $1/2^{b \cdot z}$ for a read requiring $z$ consecutive matching kmers via the `-z`/`--findere-z` parameter) for lower memory and disk usage, since no reconstruction index is needed. See [`estimate`](../usage/estimate.md) to explore this trade-off before building.
## On-disk layout
```
<index_root>/
index.meta global configuration (k, minimizer size, partition count,
evidence mode, whether counts are stored) and genome list/metadata
scatter.done / count.done / index.done build-progress sentinels
spectrums/<label>.json per-genome kmer frequency histogram
partitions/
part_00000/ ... part_NNNNN/
index/
meta.json number of layers in this partition
layer_0/
unitigs.bin reconstructible kmer sequence data — always kept
unitigs.bin.idx random-access index into unitigs.bin (exact evidence only)
mphf.bin the minimal perfect hash function
evidence.bin exact evidence (exact mode only)
fingerprint.bin approximate evidence (approximate mode only)
counts/ per-genome kmer counts (if counts were requested)
presence/ per-genome presence/absence bits
layer_1/, layer_2/, ... added by later merges, same internal structure
```
`unitigs.bin` is the only file from which the indexed kmer content can be fully recovered; it is always retained. Every other file (MPHF, evidence, counts) is derived from it.
A **layer** corresponds to one increment of kmer content added to a partition — most commonly, one [`merge`](../usage/merge.md) operation that introduces kmers not already present in the index. Genomes already present in the index simply gain new columns in the existing layers' count/presence data; only genuinely new kmer content is assembled into a new layer. Because of this, merging cost scales with the novel kmer content being added, not with the accumulated size of the index. A query against an index with several layers checks each layer's MPHF in turn.
Sources merged together must share the same kmer size, minimizer size, partition count, and evidence mode (including matching approximate-mode parameters); mismatches are rejected rather than silently reconciled — [`reindex`](../usage/reindex.md) one of the sources first if needed.
`obikmer pack` consolidates a partition's per-column files (counts/presence) into a single file, reducing the number of file opens needed at query time.
UserDocMD/index.md
+68
View File
@@ -0,0 +1,68 @@
# obikmer
`obikmer` is a command-line tool for counting, indexing, querying and comparing DNA sequences represented as kmer sets. It targets individual genome datasets of tens of gigabases, with an emphasis on computational, memory, and disk efficiency.
All functionality is exposed through a single binary, `obikmer`, organized as subcommands.
## Core principles
- Kmers are of fixed, odd length $k$, chosen at index-construction time in the range $[11, 31]$ (see [Kmers and super-kmers](theory/kmers_and_superkmers.md)).
- Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see [DNA encoding](theory/encoding.md)).
- Kmers are handled in **canonical form** ($\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))$), making counting strand-independent.
- Sequences are decomposed into **super-kmers** before storage, anchored on a hash-selected **minimizer** (see [Minimizer selection](theory/minimizer_selection.md)), then routed to one of several **partitions** for parallel, memory-bounded processing (see [Partitioning and indexing architecture](theory/indexing_architecture.md)).
- Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see [Low-complexity kmer filter](theory/entropy_filter.md)).
## Commands
| Command | Purpose |
|---|---|
| [`superkmer`](usage/superkmer.md) | Extract super-kmers from a sequence file and write them to stdout |
| [`index`](usage/index_command.md) | Build a genome index |
| [`merge`](usage/merge.md) | Merge multiple indexes into one |
| [`filter`](usage/filter.md) | Retain only kmers matching ingroup/outgroup predicates |
| [`select`](usage/select.md) | Project and/or aggregate genome columns of an index |
| [`query`](usage/query.md) | Query an index with sequences and annotate matches |
| [`dump`](usage/dump.md) | Dump indexed kmers as CSV |
| [`annotate`](usage/annotate.md) | Add, update, or dump genome metadata |
| [`distance`](usage/distance.md) | Compute pairwise distance matrices and phylogenetic exports |
| [`unitig`](usage/unitig.md) | Dump the unitigs of an index as FASTA |
| [`estimate`](usage/estimate.md) | Estimate approximate-index parameters before indexing |
| [`reindex`](usage/reindex.md) | Convert an index's evidence representation (exact ↔ approximate) |
| [`utils`](usage/utils.md) | Miscellaneous index maintenance and inspection utilities |
| [`pack`](usage/pack.md) | Pack per-column matrix files into a single-file format |
See [Genome predicates and taxonomy paths](usage/predicates.md) for the selection language shared by `filter`, `select`, `dump`, and `unitig`.
## Further reading
- [Index construction and on-disk layout](formats/index_layout.md)
- [Architecture notes for advanced use](architecture.md) — parallel execution, NUMA awareness, index dimensioning
## Input formats
- `superkmer` and `index`: FASTA (`.fa`, `.fasta`), FASTQ (`.fq`, `.fastq`), GenBank flat file (`.gb`, `.gbk`, `.gbff`), all optionally gzip-compressed; directories are expanded recursively; streaming stdin via `-` or when no input path is given.
- `query`: FASTA or FASTQ, optionally gzip-compressed; streaming stdin the same way.
## Parameter constraints
These constraints are checked at startup; an invalid value exits immediately with an error.
| Parameter | Constraint | Reason |
|---|---|---|
| $k$ (`--kmer-size`) | odd, $k \in [11, 31]$ | odd length guarantees the canonical form is always well defined; the range keeps a kmer within a 64-bit word while retaining specificity |
| $m$ (`--minimizer-size`) | odd, $3 \le m \le k-1$ | same palindrome argument as $k$; must be strictly shorter than the kmer |
| $z$ (`-z`, approximate evidence only) | $z \le k-1$ | the effective indexed kmer size is $k-z+1$ |
## Genome label constraints
Genome labels are arbitrary Unicode strings, with the following restrictions:
| Character | Forbidden | Reason |
|---|---|---|
| `/` | yes | filesystem path separator |
| `=` | yes | separator used by `--new-label` |
| `\0` | yes | null byte |
| `\n`, `\r`, `\t` | yes | would break CSV output |
| spaces | allowed | quote in the shell, e.g. `--new-label 'new label=old label'` |
Empty labels are rejected. A label derived automatically from the input file name (when `--label` is omitted) is not validated, since it is already filesystem-safe.
UserDocMD/installation.md
+84
View File
@@ -0,0 +1,84 @@
# Installation
## Prerequisites
### Rust toolchain
`obikmer` requires **Rust 1.85 or later** (edition 2024). Install or update via [rustup](https://rustup.rs):
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
rustup update stable
```
### C build environment (required for hwloc)
`obikmer` embeds [hwloc](https://www.open-mpi.org/projects/hwloc/) (Hardware Locality) for NUMA-aware thread placement on multi-socket machines. hwloc is built from source at compile time, which requires a standard C build environment.
#### Linux (Debian/Ubuntu)
```bash
apt install build-essential automake libtool autoconf pkg-config
```
#### Linux (RHEL/Rocky/AlmaLinux)
```bash
dnf install gcc make automake libtool autoconf pkgconfig
```
#### HPC clusters
Most HPC clusters provide these tools via the module system:
```bash
module load gcc automake libtool autoconf
```
If in doubt, check that `autoreconf --version` and `libtool --version` return successfully.
#### macOS
```bash
brew install automake libtool autoconf pkg-config
```
## Building
```bash
git clone <repository-url>
cd obikmer/src
cargo build --release
```
The compiled binary is at `target/release/obikmer`.
### Building on HPC clusters (network filesystems)
HPC home directories are typically on a network filesystem (Lustre, NFS) optimized for large sequential reads, not for the many small file operations Cargo generates during compilation. Building directly on such a filesystem can be extremely slow.
Redirect the build directory to a local scratch disk:
```bash
CARGO_TARGET_DIR=/scratch/$USER/cargo-target cargo build --release
```
Adapt the path to the scratch space available on your cluster (`/var/tmp`, `/tmp`, `/scratch/local`, etc.). Once built, copy the binary to a permanent location:
```bash
cp /scratch/$USER/cargo-target/release/obikmer ~/bin/
```
## NUMA support
NUMA-aware thread placement is active automatically on multi-socket Linux machines, detected at runtime via hwloc. No build flag is required — it falls back gracefully to a single-pool strategy on:
- macOS (Apple Silicon, unified memory)
- single-socket Linux machines
- any system where hwloc reports only one NUMA node
## Verifying the installation
```bash
obikmer --help
```
UserDocMD/references.bib
+261
View File
@@ -0,0 +1,261 @@
%% This BibTeX bibliography file was created using BibDesk.
%% https://bibdesk.sourceforge.io/
%% Created for Eric Coissac at 2026-04-18 08:19:36 +0200
%% Saved with string encoding Unicode (UTF-8)
@article{Zheng2020-ji,
abstract = {MOTIVATION: Minimizers are methods to sample k-mers from a
string, with the guarantee that similar set of k-mers will be
chosen on similar strings. It is parameterized by the k-mer
length k, a window length w and an order on the k-mers.
Minimizers are used in a large number of softwares and pipelines
to improve computation efficiency and decrease memory usage.
Despite the method's popularity, many theoretical questions
regarding its performance remain open. The core metric for
measuring performance of a minimizer is the density, which
measures the sparsity of sampled k-mers. The theoretical optimal
density for a minimizer is 1/w, provably not achievable in
general. For given k and w, little is known about asymptotically
optimal minimizers, that is minimizers with density O(1/w).
RESULTS: We derive a necessary and sufficient condition for
existence of asymptotically optimal minimizers. We also provide a
randomized algorithm, called the Miniception, to design
minimizers with the best theoretical guarantee to date on density
in practical scenarios. Constructing and using the Miniception is
as easy as constructing and using a random minimizer, which
allows the design of efficient minimizers that scale to the
values of k and w used in current bioinformatics software
programs. AVAILABILITY AND IMPLEMENTATION: Reference
implementation of the Miniception and the codes for analysis can
be found at https://github.com/kingsford-group/miniception.
SUPPLEMENTARY INFORMATION: Supplementary data are available at
Bioinformatics online.},
author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
doi = {10.1093/bioinformatics/btaa472},
issn = {1367-4803,1367-4811},
journal = {Bioinformatics (Oxford, England)},
language = {en},
month = jul,
number = {Suppl_1},
pages = {i119--i127},
pmc = {PMC8248892},
pmid = 32657376,
publisher = {Oxford University Press (OUP)},
title = {Improved design and analysis of practical minimizers},
url = {http://dx.doi.org/10.1093/bioinformatics/btaa472},
volume = 36,
year = 2020,
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btaa472}}
@article{Zheng2021-cc,
abstract = {MOTIVATION: Minimizers are efficient methods to sample k-mers
from genomic sequences that unconditionally preserve sufficiently
long matches between sequences. Well-established methods to
construct efficient minimizers focus on sampling fewer k-mers on
a random sequence and use universal hitting sets (sets of k-mers
that appear frequently enough) to upper bound the sketch size. In
contrast, the problem of sequence-specific minimizers, which is
to construct efficient minimizers to sample fewer k-mers on a
specific sequence such as the reference genome, is less studied.
Currently, the theoretical understanding of this problem is
lacking, and existing methods do not specialize well to sketch
specific sequences. RESULTS: We propose the concept of polar
sets, complementary to the existing idea of universal hitting
sets. Polar sets are k-mer sets that are spread out enough on the
reference, and provably specialize well to specific sequences.
Link energy measures how well spread out a polar set is, and with
it, the sketch size can be bounded from above and below in a
theoretically sound way. This allows for direct optimization of
sketch size. We propose efficient heuristics to construct polar
sets, and via experiments on the human reference genome, show
their practical superiority in designing efficient
sequence-specific minimizers. AVAILABILITY AND IMPLEMENTATION: A
reference implementation and code for analyses under an
open-source license are at
https://github.com/kingsford-group/polarset. SUPPLEMENTARY
INFORMATION: Supplementary data are available at Bioinformatics
online.},
author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
doi = {10.1093/bioinformatics/btab313},
issn = {1367-4803,1367-4811},
journal = {Bioinformatics (Oxford, England)},
language = {en},
month = jul,
number = {Suppl\_1},
pages = {i187--i195},
pmc = {PMC8686682},
pmid = 34252928,
publisher = {Oxford University Press (OUP)},
title = {Sequence-specific minimizers via polar sets},
url = {http://dx.doi.org/10.1093/bioinformatics/btab313},
volume = 37,
year = 2021,
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btab313}}
@article{Pan2024-hb,
abstract = {MOTIVATION: The minimizer concept is a data structure for
sequence sketching. The standard canonical minimizer selects a
subset of k-mers from the given DNA sequence by comparing the
forward and reverse k-mers in a window simultaneously according
to a predefined selection scheme. It is widely employed by
sequence analysis such as read mapping and assembly. k-mer
density, k-mer repetitiveness (e.g. k-mer bias), and
computational efficiency are three critical measurements for
minimizer selection schemes. However, there exist trade-offs
between kinds of minimizer variants. Generic, effective, and
efficient are always the requirements for high-performance
minimizer algorithms. RESULTS: We propose a simple minimizer
operator as a refinement of the standard canonical minimizer. It
takes only a few operations to compute. However, it can improve
the k-mer repetitiveness, especially for the lexicographic order.
It applies to other selection schemes of total orders (e.g.
random orders). Moreover, it is computationally efficient and the
density is close to that of the standard minimizer. The refined
minimizer may benefit high-performance applications like binning
and read mapping. AVAILABILITY AND IMPLEMENTATION: The source
code of the benchmark in this work is available at the github
repository https://github.com/xp3i4/mini\_benchmark.},
author = {Pan, Chenxu and Reinert, Knut},
doi = {10.1093/bioinformatics/btae045},
issn = {1367-4803,1367-4811},
journal = {Bioinformatics (Oxford, England)},
language = {en},
month = feb,
number = 2,
pmc = {PMC10868324},
pmid = 38269626,
publisher = {Oxford University Press (OUP)},
title = {A simple refined DNA minimizer operator enables 2-fold faster computation},
url = {http://dx.doi.org/10.1093/bioinformatics/btae045},
volume = 40,
year = 2024,
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btae045}}
@article{Kille2023-px,
abstract = {MOTIVATION: The Jaccard similarity on k-mer sets has shown to be
a convenient proxy for sequence identity. By avoiding expensive
base-level alignments and comparing reduced sequence
representations, tools such as MashMap can scale to massive
numbers of pairwise comparisons while still providing useful
similarity estimates. However, due to their reliance on minimizer
winnowing, previous versions of MashMap were shown to be biased
and inconsistent estimators of Jaccard similarity. This directly
impacts downstream tools that rely on the accuracy of these
estimates. RESULTS: To address this, we propose the minmer
winnowing scheme, which generalizes the minimizer scheme by use
of a rolling minhash with multiple sampled k-mers per window. We
show both theoretically and empirically that minmers yield an
unbiased estimator of local Jaccard similarity, and we implement
this scheme in an updated version of MashMap. The minmer-based
implementation is over 10 times faster than the minimizer-based
version under the default ANI threshold, making it well-suited
for large-scale comparative genomics applications. AVAILABILITY
AND IMPLEMENTATION: MashMap3 is available at
https://github.com/marbl/MashMap.},
author = {Kille, Bryce and Garrison, Erik and Treangen, Todd J and Phillippy, Adam M},
doi = {10.1093/bioinformatics/btad512},
issn = {1367-4803,1367-4811},
journal = {Bioinformatics (Oxford, England)},
language = {en},
month = sep,
number = 9,
pmc = {PMC10505501},
pmid = 37603771,
publisher = {Oxford University Press (OUP)},
title = {Minmers are a generalization of minimizers that enable unbiased local Jaccard estimation},
url = {http://dx.doi.org/10.1093/bioinformatics/btad512},
volume = 39,
year = 2023,
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btad512}}
@incollection{Golan2025-xf,
address = {Cham},
author = {Golan, Shay and Shur, Arseny M},
booktitle = {Lecture Notes in Computer Science},
doi = {10.1007/978-3-031-82670-2\_25},
isbn = {9783031826696,9783031826702},
issn = {0302-9743,1611-3349},
language = {en},
pages = {347--360},
publisher = {Springer Nature Switzerland},
series = {Lecture Notes in Computer Science},
title = {Expected density of random minimizers},
url = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
year = 2025,
bdsk-url-1 = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-031-82670-2%5C_25}}
@article{Mohamadi2017-ok,
abstract = {Motivation: Many bioinformatics algorithms are designed for the
analysis of sequences of some uniform length, conventionally
referred to as k -mers. These include de Bruijn graph assembly
methods and sequence alignment tools. An efficient algorithm to
enumerate the number of unique k -mers, or even better, to build
a histogram of k -mer frequencies would be desirable for these
tools and their downstream analysis pipelines. Among other
applications, estimated frequencies can be used to predict genome
sizes, measure sequencing error rates, and tune runtime
parameters for analysis tools. However, calculating a k -mer
histogram from large volumes of sequencing data is a challenging
task. Results: Here, we present ntCard, a streaming algorithm for
estimating the frequencies of k -mers in genomics datasets. At
its core, ntCard uses the ntHash algorithm to efficiently compute
hash values for streamed sequences. It then samples the
calculated hash values to build a reduced representation
multiplicity table describing the sample distribution. Finally,
it uses a statistical model to reconstruct the population
distribution from the sample distribution. We have compared the
performance of ntCard and other cardinality estimation
algorithms. We used three datasets of 480 GB, 500 GB and 2.4 TB
in size, where the first two representing whole genome shotgun
sequencing experiments on the human genome and the last one on
the white spruce genome. Results show ntCard estimates k -mer
coverage frequencies >15× faster than the state-of-the-art
algorithms, using similar amount of memory, and with higher
accuracy rates. Thus, our benchmarks demonstrate ntCard as a
potentially enabling technology for large-scale genomics
applications. Availability and Implementation: ntCard is written
in C ++ and is released under the GPL license. It is freely
available at https://github.com/bcgsc/ntCard. Contact:
hmohamadi@bcgsc.ca or ibirol@bcgsc.ca. Supplementary information:
Supplementary data are available at Bioinformatics online.},
author = {Mohamadi, Hamid and Khan, Hamza and Birol, Inanc},
date-modified = {2026-04-18 08:19:36 +0200},
doi = {10.1093/bioinformatics/btw832},
issn = {1367-4803,1367-4811},
journal = {Bioinformatics (Oxford, England)},
language = {en},
month = may,
number = 9,
pages = {1324--1330},
pmc = {PMC5408799},
pmid = 28453674,
publisher = {Oxford University Press (OUP)},
title = {ntCard: a streaming algorithm for cardinality estimation in genomics data},
url = {http://dx.doi.org/10.1093/bioinformatics/btw832},
volume = 33,
year = 2017,
bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btw832}}
@misc{Mash-distances-doc,
author = {{Marbl Lab}},
howpublished = {Mash documentation},
title = {Mash Distance},
url = {https://mash.readthedocs.io/en/latest/distances.html},
urldate = {2026-07-09},
year = 2026}
@article{Fan2015-mash-formula,
author = {Fan, Huan and Ives, Anthony R and Surget-Groba, Yann and Cannon, Charles H},
doi = {10.1186/s12864-015-1647-5},
journal = {BMC Genomics},
number = 1,
title = {An assembly and alignment-free method of phylogeny reconstruction from next-generation sequencing data},
url = {https://doi.org/10.1186/s12864-015-1647-5},
volume = 16,
year = 2015}
UserDocMD/theory/encoding.md
+28
View File
@@ -0,0 +1,28 @@
# DNA encoding
## 2-bit nucleotide encoding
Every nucleotide is encoded on 2 bits, most-significant-bit first within each word:
| Base | Encoding |
|------|----------|
| A | `00` |
| C | `01` |
| G | `10` |
| T | `11` |
The Watson-Crick complement of a base is its bitwise NOT on 2 bits: $\text{complement}(base) = \lnot base \mathbin{\&} \texttt{0b11}$.
## Kmer encoding
A kmer of length $k$ ($k \le 31$) fits in a single 64-bit word. The first nucleotide occupies the two most significant bits, each following nucleotide occupies the next two bits, and unused low-order bits are zero. Extracting nucleotide i (0-indexed from the 5′ end) is a shift-and-mask operation.
Reverse complement is computed by bit manipulation directly on the packed word, without any lookup table: complement every base, reverse the byte order, then reverse the order of 2-bit groups within each byte in two more passes, and finally realign the result to the most-significant bits.
## Canonical form
The canonical form of a kmer is the lexicographic minimum of the kmer and its reverse complement:
$$\text{canonical}(kmer) = \min\big(kmer,\ \text{revcomp}(kmer)\big)$$
Using the canonical form halves the kmer space and makes counting strand-independent: a kmer and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.
UserDocMD/theory/entropy_filter.md
+36
View File
@@ -0,0 +1,36 @@
# Low-complexity kmer filter
Low-complexity kmers (homopolymer runs, tandem repeats) can dominate an index without carrying useful information. `obikmer` detects and excludes them during index construction using a normalized Shannon entropy score.
## Sub-word frequencies
For a kmer of length $k$ and a sub-word size $ws$ ($1 \le ws \le ws_{\max}$, default $ws_{\max} = 6$), the kmer is decomposed into its $k - ws + 1$ overlapping sub-words of length $ws$ by sliding a window across it. Each sub-word is tallied under its raw 2-bit-packed value, with no canonicalization.
## Corrected Shannon entropy
Let $f_j$ be the observed count of raw sub-word $j$, and $n_{\text{words}} = k - ws + 1$ the total number of sub-words. The entropy is:
$$H_{\text{corr}} = \log(n_{\text{words}}) - \frac{1}{n_{\text{words}}} \sum_j f_j \log f_j$$
## Small-sample correction
Because only $n_{\text{words}}$ sub-words are observed among up to $4^{ws}$ possible values, the achievable maximum entropy $H_{\max}$ is bounded below $\log(4^{ws})$ for small samples. $H_{\max}$ is computed from the most uniform integer distribution achievable with $n_{\text{words}}$ observations over $4^{ws}$ categories. The normalized entropy is:
$$\hat{H}(ws) = \frac{H_{\text{corr}}}{H_{\max}} \in [0, 1]$$
A value near 0 indicates low complexity (e.g. a homopolymer run); near 1 indicates high complexity, characteristic of a random sequence.
## Final score
The filter evaluates $\hat{H}(ws)$ for every word size from 1 to ws_max and keeps the minimum:
$$\text{entropy}(kmer) = \min_{ws=1}^{ws_{\max}} \hat{H}(ws)$$
Taking the minimum across word sizes ensures that repetition at any scale is detected: a homopolymer is caught at $ws=1$, a dinucleotide repeat at $ws=2$, and so on. A kmer is rejected if its entropy score falls below a threshold $\theta$ (default 0.7), a configurable collection parameter.
## Properties
The entropy score depends only on the kmer sequence itself, not on where or how many times it occurs:
- **Orientation invariance**: a kmer and its reverse complement always receive the same score.
- **Context independence**: a given kmer is always accepted or always rejected, regardless of which genome or read it appears in. The filter defines a fixed partition of the kmer space into low-complexity and valid kmers.
UserDocMD/theory/indexing_architecture.md
+32
View File
@@ -0,0 +1,32 @@
# Partitioning and indexing architecture
An index is split into a fixed number of **partitions**, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.
## Routing
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition:
```
canonical minimizer → hash(minimizer) → p-bit value → partition index
```
The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
## Why hashing is necessary
A canonical minimizer is an m-mer ($m \in \{9, 11, 13, 15\}$), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.
Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits $p$ is chosen well below the number of bits available in the minimizer ($2m$): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.
## Parameter guidance
| Minimizer size $m$ | Minimizer bits ($2m$) | Typical partition-index bits $p$ | Partitions |
|----|-----------|-----------|------------|
| 9 | 18 | 6–8 | 64–256 |
| 11 | 22 | 8–10 | 256–1 024 |
| 13 | 26 | 10–12 | 1 024–4 096|
| 15 | 30 | 10–14 | 1 024–16 384|
The number of partitions must satisfy $p \le 2m$, and in practice $p$ is chosen well below that bound to leave a comfortable entropy margin. For $k=31$, $m=13$, $p=10$ (1024 partitions), partition load is well balanced on real genomic data.
UserDocMD/theory/kmers_and_superkmers.md
+24
View File
@@ -0,0 +1,24 @@
# Kmers and super-kmers
## Kmers
A **kmer** is a DNA subsequence of fixed length $k$. Two constraints apply to $k$, both enforced when a command starts (an invalid value exits immediately with an error):
- $k \in [11, 31]$: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base ($k \le 32$ is the hard limit; $k < 11$ gives insufficient specificity).
- $k$ **is odd**: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see [DNA encoding](encoding.md)) to be well defined.
## Super-kmers
A **super-kmer** is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see [Minimizer selection](minimizer_selection.md)). Each kmer in the run overlaps the next by $k-1$ nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
For a random minimizer of length $m$ over kmers of length $k$, the expected length of a super-kmer is approximately [@Zheng2020-ji; @Golan2025-xf]:
$$L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1$$
For $k=31$, $m=13$ this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.
### Canonical super-kmers
A **canonical super-kmer** is the lexicographic minimum of a super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.
Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.
UserDocMD/theory/minimizer_selection.md
+42
View File
@@ -0,0 +1,42 @@
# Minimizer selection
## Definition
A **minimizer** of a kmer window is the m-mer ($m < k$) that is smallest, among all $k - m + 1$ overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent.
The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
## Hash-based ("random") minimizer
`obikmer` selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions.
Instead, a well-distributed hash function $H$ is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest $H$ value wins. Because $H$ is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition.
The canonical form used as input to $H$ is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the *distribution of hash values themselves* toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.
### Hash function
The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed:
$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
```
H(x):
x ← x ⊕ 0x9e3779b97f4a7c15
x ← x ⊕ (x >> 30)
x ← x × 0xbf58476d1ce4e5b9
x ← x ⊕ (x >> 27)
x ← x × 0x94d049bb133111eb
return x ⊕ (x >> 31)
```
The XOR seed avoids the finalizer's fixed point at 0 ($\text{mix64}(0) = 0$), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison.
## Partition routing is independent of minimizer selection
The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately:
- **Selection** uses $H$ applied to every candidate m-mer in the window, keeping the minimum.
- **Partition routing** recomputes $H$ on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.
See [Partitioning and indexing architecture](indexing_architecture.md) for how the routing value is turned into a partition index.
UserDocMD/usage/annotate.md
+25
View File
@@ -0,0 +1,25 @@
# annotate
Add or update genome metadata of an index from a CSV file, or dump the current metadata as CSV.
```bash
obikmer annotate INDEX --csv FILE [OPTIONS]
obikmer annotate INDEX --dump
```
## Arguments
| Argument | Description |
|---|---|
| `INDEX` | Index directory to annotate (modified in place) |
## Options
| Option | Default | Description |
|---|---|---|
| `--csv` | — | CSV file of metadata to apply (must contain an id column); required unless `--dump` is used |
| `--sep` | `,` | CSV field separator |
| `--id-col` | `id` | Name of the column containing genome labels |
| `--na-value` | `NA` | Value meaning "remove this field" (deletes the existing key if present) |
| `--no-overwrite` | off | Do not overwrite existing metadata keys |
| `--dump` | off | Print all genome metadata as CSV to stdout instead of applying a file |
UserDocMD/usage/distance.md
+97
View File
@@ -0,0 +1,97 @@
# distance
Compute pairwise distance matrices between the genomes stored in an index, optionally build trees (NJ/UPGMA), and optionally derive a central-position SNP model with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
```bash
obikmer distance INDEX [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `INDEX` | Index directory |
## Distance matrix
| Option | Default | Description |
|---|---|---|
| `--metric` | `jaccard` | One of `jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` |
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for Jaccard/Mash on a count index |
| `--shared-kmers` | off | Also write the shared-kmer count matrix |
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
| `-o, --output` | none (stdout) | Output file prefix; without it, the distance matrix is printed to stdout as CSV |
`hamming` requires a presence/absence index. All other metrics work on either index type; on a presence index, `jaccard`/`mash`/`hamming` are the only ones available.
### Metric definitions
- **jaccard**: $D = 1 - \dfrac{|A \cap B|}{|A \cup B|}$ over the sets of kmers present in each genome.
- **mash**: derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$.
- **hamming**: number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$.
- **bray-curtis**: $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts.
- **relfreq-bray-curtis**: the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts.
- **euclidean**: $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts.
- **relfreq-euclidean**: the same formula on relative frequencies.
- **hellinger**: $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$.
- **hellinger-euclidean**: the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$.
## Central-position SNP model
This is a separate operation from the distance-matrix computation above: if any option below is used, no `--metric` matrix is computed in the same invocation.
A **family** is the set of up to 4 kmers that share identical flanking sequence and differ only at the exact central base. Because $k$ is odd, the central position is well defined and maps to itself under reverse complementation. All computations below first require building the **sibling annex**, an index-wide record of which of the 4 possible central bases are observed at each family, across every genome.
| Option | Default | Description |
|---|---|---|
| `--sibling-annex` | off | Build the sibling presence-mask annex (prerequisite for every option below) |
| `--exclude-genome LABEL` | none | Exclude a genome (repeatable) from every SNP/Sankoff/export computation below |
| `--sibling-stats` | off | Write the family-size (sibling count) distribution, per genome and globally |
| `--raw-snp-distance` | off | Write the single-copy central-SNP p-distance matrix |
| `--raw-snp-counts` | off | Write per-pair diagnostic counts (n_snp, n_shared, n_eligible) instead of a matrix |
| `--snp` | off | Write a SNP-only pseudo-alignment in FASTA, IUPAC-coded |
### Locus eligibility
A family is eligible for a genome pair $(i, j)$ only if genome $i$ carries exactly one of the family's observed forms (single-copy, unambiguous) and genome $j$ also carries exactly one. A genome carrying more than one form at a locus makes that locus ineligible for any pair involving it.
`--raw-snp-distance` tallies, over every eligible locus of every genome pair, $n_{\text{snp}}$ (the two genomes' single forms differ) versus $n_{\text{shared}}$ (they agree — this includes invariant families). The output ratio is $\hat{p} = \dfrac{n_{\text{snp}}}{n_{\text{snp}} + n_{\text{shared}}}$.
`--snp` restricts itself to *variable* families (family size $\ge 2$) and writes one FASTA record per genome, one column per family, IUPAC-coded from each genome's presence mask at that family (a single form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`).
`--exclude-genome` removes a genome from these computations, re-checking column variability among the remaining genomes so that a column made monomorphic by the exclusion is dropped rather than kept artificially. It does not affect the `--metric` distance-matrix path.
## Sankoff calibration and phylogenetic exports
| Option | Default | Description |
|---|---|---|
| `--sankoff` | off | Calibrate a 16-state parsimony cost matrix and matching pseudo-alignment |
| `--sankoff-ratio-ceiling` | `0.5` | Exclude genome pairs whose raw SNP ratio exceeds this value from the calibration |
| `--tnt` | off | Also write a TNT script (implies `--sankoff`) |
| `--phyg` | off | Also write PhyG input files (implies `--sankoff`) |
| `--iqtree` | off | Also write an IQ-TREE custom model and alignment (implies `--sankoff`) |
| `--sankoff-cost-scale` | `100` | Integer scaling factor applied to costs before rounding (required by TNT/PhyG's integer-only cost commands) |
### The 16-state model
Each family is treated as a character with 16 possible states: one per subset of the 4 possible central bases actually observed (including the empty subset). Calibration combines two tallies, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`:
- a $5 \times 5$ transition matrix over family cardinality (0–4 observed forms) between paired genomes, and
- a $4 \times 4$ base-substitution transition matrix from unambiguous single-copy loci,
which are combined into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character, to avoid ambiguity with external tools' own gap semantics). It does not run any external tool.
### Exports
All three exports reuse the `--sankoff` calibrated matrix and pseudo-alignment, recoded for the target tool:
- **`--tnt`**: a self-contained TNT script (alignment recoded to TNT's fixed 16-symbol alphabet, integer-scaled cost matrix re-closed to a metric, a default search block).
- **`--phyg`**: a custom cost-matrix file plus a PhyG script reusing the `--sankoff` alignment directly.
- **`--iqtree`**: a custom substitution-model file (exchangeability matrix recovered as $R(a,b) = e^{-\text{cost}(a,b)}$, plus empirical state frequencies) and a matching alignment, for maximum-likelihood inference with real branch lengths (unlike the parsimony step-counts from TNT/PhyG). Only states actually occurring in the alignment are kept and compactly renumbered.
## Output files
With `-o/--output PREFIX`, the relevant subset of the following files is written: `<prefix>_dist.csv`, `<prefix>_shared.csv`, `<prefix>_nj.nwk`, `<prefix>_upgma.nwk`, `<prefix>_siblings.csv`, `<prefix>_rawsnp.csv`, `<prefix>_rawsnp_counts.csv`, `<prefix>_snp.fasta`, `<prefix>_sankoff_matrix.csv`, `<prefix>_sankoff_params.yaml`, `<prefix>_sankoff.fasta`, `<prefix>_sankoff.tnt`, `<prefix>_sankoff.tcm`, `<prefix>_sankoff.pg`, `<prefix>_iqtree.model`, `<prefix>_iqtree.fasta`. Without `-o`, only the plain `--metric` distance matrix is produced, on stdout.
UserDocMD/usage/dump.md
+25
View File
@@ -0,0 +1,25 @@
# dump
Dump all kmers of an index as CSV, one row per kmer, with per-genome counts or presence.
```bash
obikmer dump INDEX [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `INDEX` | Index directory to dump |
## Options
| Option | Default | Description |
|---|---|---|
| `--force-presence` | off | Output presence/absence (0/1) even if the index stores counts |
| `--debug` | off | Prefix each row with the partition and layer columns |
| `--head N` | none | Limit output to the first N kmers |
`dump` also accepts the shared [predicate options](filter.md#predicate-options) (`--ingroup`, `--outgroup`, `--min-count`, etc.) to restrict which kmers are dumped.
Output is CSV on stdout.
UserDocMD/usage/estimate.md
+18
View File
@@ -0,0 +1,18 @@
# estimate
Estimate approximate-index parameters (z, evidence bits, false-positive rate) before building an index with `--approx`, without touching any files.
```bash
obikmer estimate [OPTIONS]
```
## Options
| Option | Default | Description |
|---|---|---|
| `-k, --kmer-size` | `31` | Kmer size used at query time (matches `index`'s `--kmer-size`) |
| `-z, --findere-z` | none | Findere z parameter |
| `--evidence-bits` | none | Fingerprint bits per slot (b) |
| `--fp` | none | Target false-positive rate per z-window |
Any two of `-z`, `--evidence-bits`, `--fp` may be given; the third is derived using the same model as `index --approx` and `reindex --approx` ($FP = 1 / 2^{b \cdot z}$). The report printed to stdout includes: query $k$, effective indexed $k$ ($k-z+1$), $z$, evidence bits, per-kmer false-positive rate, and per-z-window false-positive rate.
UserDocMD/usage/filter.md
+47
View File
@@ -0,0 +1,47 @@
# filter
Apply row-level selection to an index: retain only kmers matching ingroup/outgroup predicates over genome membership, plus optional total-count and complexity thresholds. The output is a new, single-layer index.
```bash
obikmer filter SOURCE -o OUTPUT [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `SOURCE` | Source index directory |
## Options
| Option | Default | Description |
|---|---|---|
| `-o, --output` | — (required) | Output index directory |
| `-f, --force` | off | Overwrite an existing output directory |
| `--presence` | off | Output presence/absence instead of counts |
| `--min-total-count` | none | Minimum total count across all genomes (count index only) |
| `--max-total-count` | none | Maximum total count across all genomes |
| `--min-complexity` | none | Minimum normalized entropy (same score as `--theta` at index build time), recomputed from the stored unitig sequences |
| `--complexity-level-max` | `6` | Maximum sub-word size for the complexity score (used only with `--min-complexity`) |
## Predicate options
| Option | Default | Description |
|---|---|---|
| `--ingroup` | none | Ingroup predicate (repeatable; each occurrence is ANDed) |
| `--outgroup` | none | Outgroup predicate (repeatable; each occurrence is ORed) |
| `--min-count` | 0, or group size + N if negative | Minimum number of ingroup genomes carrying the kmer |
| `--max-count` | ingroup group size | Maximum number of ingroup genomes carrying the kmer |
| `--min-frac` | `1.0` if `--ingroup` given without an explicit quorum, else `0.0` | Minimum fraction of ingroup genomes |
| `--max-frac` | `1.0` | Maximum fraction of ingroup genomes |
| `--min-outgroup-count` | `0` | Minimum number of outgroup genomes carrying the kmer |
| `--max-outgroup-count` | `0` if `--outgroup` given without an explicit quorum, else outgroup group size | Maximum number of outgroup genomes |
| `--min-outgroup-frac` | `0.0` | Minimum fraction of outgroup genomes |
| `--max-outgroup-frac` | `1.0` | Maximum fraction of outgroup genomes |
| `--presence-threshold` | `0` | Minimum count for a genome to be considered a carrier of a kmer |
See [Genome predicates and taxonomy paths](predicates.md) for the predicate syntax used by `--ingroup`/`--outgroup`.
A negative `--min-count`/`--max-count` is interpreted as an offset from the group size — e.g. `--min-count=-1` means "all but one".
Declaring `--ingroup` with no explicit ingroup quorum flag implicitly sets `--min-frac 1.0` (present in every ingroup genome). Declaring `--outgroup` with no explicit outgroup quorum flag implicitly sets `--max-outgroup-count 0` (absent from every outgroup genome). Any explicit quorum flag for a group disables that group's implicit default.
UserDocMD/usage/index_command.md
+50
View File
@@ -0,0 +1,50 @@
# index
Build a genome index from one or more sequence files. Construction proceeds in phases (scatter → dereplicate → count → layered MPHF), described in [On-disk storage](../formats/index_layout.md).
```bash
obikmer index -o OUTPUT [OPTIONS] [INPUTS...]
```
## Arguments
| Argument | Description |
|---|---|
| `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. |
## Options
| Option | Default | Description |
|---|---|---|
| `-o, --output` | — (required) | Output index directory |
| `--force` | off | Overwrite an existing output directory |
| `--label` | input file name without extension | Genome label stored in the index |
| `--meta KEY=VALUE` | none | Attach a categorical metadata field to the genome (repeatable) |
| `-k, --kmer-size` | `31` | Kmer size (odd, in [11, 31]) |
| `-m, --minimizer-size` | `11` | Minimizer size (odd, in $[3, k-1]$) |
| `--theta` | `0.7` | Entropy threshold for the low-complexity filter |
| `--level-max` | `6` | Maximum sub-word size for the entropy score |
| `-p, --partitions` | `256` | Number of partitions (rounded up to a power of 2) |
| `-T, --threads` | detected core count | Number of worker threads |
| `--max-open-files` | `threads / 4` (min 1) | Maximum number of input files open simultaneously |
| `--min-abundance` | `1` | Minimum abundance (inclusive) for a kmer to be retained |
| `--max-abundance` | none | Maximum abundance (inclusive) |
| `--with-counts` | off | Store per-kmer counts; otherwise only presence/absence is stored |
| `--keep-intermediate` | off | Keep intermediate build files instead of deleting them after construction |
| `--approx` | off | Use approximate evidence (Findere fingerprint) instead of exact evidence |
| `-z, --findere-z` | see below | Findere z parameter: number of consecutive kmers that must all match (approximate evidence only) |
| `--evidence-bits` | see below | Fingerprint bits per slot (b), approximate evidence only |
| `--fp` | see below | Target false-positive rate per z-window, approximate evidence only |
| `--block-size` | `1` | Block size, in unitigs, for the exact on-disk index (rounded up to a power of 2) |
## Exact vs. approximate evidence
By default, an index stores **exact** evidence: a kmer is either present or absent (or has an exact count with `--with-counts`), with no false positives.
With `--approx`, evidence is stored as a compact **fingerprint** instead, trading a small, tunable false-positive rate for reduced memory/disk usage. The false-positive model is:
$$FP = \frac{1}{2^{b \cdot z}}$$
where $b$ is `--evidence-bits` and $z$ is `--findere-z`. Any two of `-z`, `--evidence-bits`, `--fp` can be given and the third is derived; if none are given, defaults are $b=8$, $z=1$ ($FP \approx 1/256$). See [`estimate`](estimate.md) to explore this trade-off before building an index, and [`reindex`](reindex.md) to convert an existing index between the two representations.
`z` must be strictly less than k: the effective indexed kmer length under approximate evidence is k−z+1.
UserDocMD/usage/merge.md
+29
View File
@@ -0,0 +1,29 @@
# merge
Merge multiple built indexes into a single index.
```bash
obikmer merge -o OUTPUT SOURCE... [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `SOURCE...` | Index directories to merge (at least one required) |
## Options
| Option | Default | Description |
|---|---|---|
| `-o, --output` | — (required) | Output index directory |
| `--force` | off | Overwrite an existing output directory |
| `--force-presence` | off | Store the merged index as presence/absence even if all sources have counts |
| `--rename-duplicates` | off | Disambiguate duplicate genome labels (`.1`, `.2`, …) instead of failing |
| `--budget-fraction` | `0.5` | Fraction of available RAM reserved as the memory budget for parallel partition merging |
## Behaviour
The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. `--force-presence` forces presence/absence regardless of the sources.
By default, merging two indexes that share a genome label fails with an error; `--rename-duplicates` instead appends a numeric suffix to keep both copies.
UserDocMD/usage/pack.md
+15
View File
@@ -0,0 +1,15 @@
# pack
Pack an index's per-column matrix files into a single-file format to reduce query-time I/O (fewer file opens per query).
```bash
obikmer pack INDEX
```
## Arguments
| Argument | Description |
|---|---|
| `INDEX` | Index directory to pack (modified in place) |
The index directory is locked for exclusive access while packing.
UserDocMD/usage/predicates.md
+46
View File
@@ -0,0 +1,46 @@
# Genome predicates and taxonomy paths
Several commands ([`filter`](filter.md), [`select`](select.md), [`dump`](dump.md), [`unitig`](unitig.md)) select or group genomes using the same predicate language over genome metadata (see [`annotate`](annotate.md) for attaching metadata to a genome).
## Predicate syntax
| Form | Meaning |
|---|---|
| `*` or `all` | Matches every genome (case-insensitive) |
| `key=v1\|v2` | Genome's `key` metadata equals one of the listed values |
| `key!=v` | Genome's `key` metadata does not equal `v` |
| `key~path` | Genome's `key` metadata (a taxonomy path) matches `path` (ancestry match) |
| `key!~path` | Genome's `key` metadata does not match `path` |
A genome whose metadata does not contain `key` at all cannot be classified by that predicate and is excluded from the relevant group's quorum count.
Multiple `--ingroup` predicates are combined with AND; multiple `--outgroup` predicates are combined with OR. When both an ingroup and an outgroup predicate would match the same genome, ingroup classification wins.
## Taxonomy paths
A metadata value is treated as a taxonomy path when it starts with the literal prefix `taxonomy:/`; any other value is treated as a plain string and only supports `=`/`!=`.
```
taxonomy:/segment1@rank1/segment2@rank2/...
```
Each segment is a name, optionally annotated with a rank (e.g. `@family`, `@genus`, `@species`); ranks are optional and can be mixed within a path. The `@` character is reserved inside taxonomy paths and cannot appear in segment names or rank labels.
### Path matching (`~` / `!~`)
Matching compares segment names only (ranks are informational, not part of the match), with anchoring controlled by leading/trailing `/`:
| Pattern | Matches |
|---|---|
| `A/B` | anywhere in the path |
| `/A/B` | at the start of the path (prefix) |
| `A/B$` | at the end of the path (suffix) |
| `/A/B$` | the entire path (exact) |
A rank-qualified query, `key@rank=value`, matches only when the path's segment at that specific rank equals `value`.
### Example
```bash
obikmer filter source -o output --ingroup "taxon~/Betulaceae/Betula"
```
UserDocMD/usage/query.md
+38
View File
@@ -0,0 +1,38 @@
# query
Query an index with sequences and annotate each query with the kmer matches found.
```bash
obikmer query INDEX INPUTS... [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `INDEX` | Index directory to query against |
| `INPUTS...` | Input sequence files (FASTA/FASTQ, gzip optional); at least one required |
## Options
| Option | Default | Description |
|---|---|---|
| `--detail` | off | Report per-position, per-genome coverage vectors in the output |
| `--count-missing` | off | Also count query kmers absent from the index |
| `--force-presence` | off | Report presence (0/1) per genome instead of raw counts |
| `--presence-threshold` | `1` | Minimum accumulated count to declare a genome present (implies `--force-presence`) |
| `-z, --findere-z` | derived from the index metadata | Override the Findere z parameter |
| `-T, --threads` | detected core count | Number of worker threads |
| `--chunk-size` | auto-sized (available RAM ÷ threads, clamped to 4–256 MiB) | I/O chunk size, in MiB |
| `--max-open-files` | `threads / 4` (min 1) | Maximum number of input files open simultaneously |
## Output
FASTA on stdout, one record per query, annotated in the OBITools-style header format `>id {"key":value,...}`:
- `kmer_count`: total number of kmers matched
- `kmer_missing`: number of query kmers absent from the index (only with `--count-missing`)
- `kmer_strict_matches`: per-genome match counts
- `coverage`: per-position, per-genome coverage vectors (only with `--detail`)
`--mismatch` is accepted by the CLI but not currently functional; using it produces a warning and is ignored.
UserDocMD/usage/reindex.md
+25
View File
@@ -0,0 +1,25 @@
# reindex
Convert an existing index's evidence representation in place, between exact and approximate.
```bash
obikmer reindex INDEX [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `INDEX` | Index directory to convert (modified in place) |
## Options
| Option | Default | Description |
|---|---|---|
| `--approx` | off | Convert to approximate evidence (default direction is approximate → exact); requires `-z`/`--evidence-bits`/`--fp` |
| `-z, --findere-z` | none | Findere z parameter (≥ 1) |
| `--evidence-bits` | none | Fingerprint bits per slot (b) |
| `--fp` | none | Target false-positive rate per z-window |
| `--block-size` | `1` | Block size for the exact on-disk index (ignored when converting to approximate) |
See [`index`](index_command.md#exact-vs-approximate-evidence) for the exact/approximate trade-off and the underlying false-positive model, and [`estimate`](estimate.md) to explore parameters beforehand. The index directory is locked for exclusive access during conversion.
UserDocMD/usage/select.md
+35
View File
@@ -0,0 +1,35 @@
# select
Project and/or aggregate the genome columns of an index into a new (or in-place) index. Where [`filter`](filter.md) selects rows (kmers), `select` operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.
```bash
obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `SOURCE` | Source index directory |
## Options
| Option | Default | Description |
|---|---|---|
| `--output` | — | Output index directory (mutually exclusive with `--in-place`) |
| `--in-place` | off | Rewrite the source index in place (mutually exclusive with `--output`) |
| `-f, --force` | off | Overwrite an existing output directory |
| `--group NAME:PRED` | none | Define a named group of genomes by predicate (repeatable; mutually exclusive with `--aggregate-by`) |
| `--group-op NAME:OP` | none | Aggregation operator for a named group |
| `--aggregate-by KEY` | none | Automatically create one group per distinct value of a metadata key (mutually exclusive with `--group`) |
| `--aggregate-op OP` | none | Aggregation operator applied to every auto-generated group |
| `--select COL,...` | all columns | Output columns, in order (group names or genome labels) |
| `--presence-threshold` | `0` | Minimum count for a genome to be considered a carrier (logical operators only) |
## Aggregation operators
`any`, `all`, `none` (logical, evaluated against `--presence-threshold`), `sum`, `min`, `max` (numeric, count index only). If a group's operator is left unspecified, it defaults to `any` when the source is a presence/absence index and `sum` when it stores counts.
A `select` never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in `--select`) is a cheap copy.
At least one of `--output`/`--in-place` is required, and at least one output column must be defined; every name listed in `--select` must resolve to either a defined group or an existing genome label. See [Genome predicates and taxonomy paths](predicates.md) for the predicate syntax used by `--group`.
UserDocMD/usage/superkmer.md
+27
View File
@@ -0,0 +1,27 @@
# superkmer
Extract super-kmers from one or more sequence files and write them to stdout, without building a full index. Useful for inspecting or piping the super-kmer decomposition of a dataset.
```bash
obikmer superkmer [OPTIONS] [INPUTS...]
```
## Arguments
| Argument | Description |
|---|---|
| `INPUTS...` | Input sequence files or directories (FASTA/FASTQ/GenBank, gzip optional). If omitted, reads from stdin. |
## Options
| Option | Default | Description |
|---|---|---|
| `-k, --kmer-size` | `31` | Kmer size (must be odd, in [11, 31]) |
| `-m, --minimizer-size` | `11` | Minimizer size (must be odd, in $[3, k-1]$) |
| `--theta` | `0.7` | Entropy threshold; kmers with a normalized entropy at or below this value are excluded |
| `--level-max` | `6` | Maximum sub-word size used for the entropy score |
| `-p, --partitions` | `256` | Number of partitions (rounded up to the next power of 2) |
| `-T, --threads` | detected core count | Number of worker threads |
| `--max-open-files` | `threads / 4` (min 1) | Maximum number of input files open simultaneously |
Output is written to stdout in the internal scatter format used by `index`; it is primarily intended to be piped into other tools or inspected for debugging.
UserDocMD/usage/unitig.md
+19
View File
@@ -0,0 +1,19 @@
# unitig
Dump the unitigs of an index as FASTA. A unitig is a maximal non-branching path through the de Bruijn graph implied by the index's kmers; the concatenation of every unitig reconstructs every stored kmer exactly once.
```bash
obikmer unitig INDEX [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `INDEX` | Index directory |
## Options
`unitig` accepts the shared [predicate options](filter.md#predicate-options) (`--ingroup`, `--outgroup`, `--min-count`, etc.) to restrict which kmers are included before the unitigs are enumerated.
Output is FASTA on stdout.
UserDocMD/usage/utils.md
+26
View File
@@ -0,0 +1,26 @@
# utils
Miscellaneous index maintenance and inspection utilities.
```bash
obikmer utils INDEXES... [OPTIONS]
```
## Arguments
| Argument | Description |
|---|---|
| `INDEXES...` | One or more index directories |
## Options
| Option | Scope | Description |
|---|---|---|
| `--new-label NEW=OLD` | single index only | Rename a genome label |
| `--upgrade-index` | single index only | Add any missing layer metadata files to an older index |
| `--bits-per-kmer` | single index only | Print bits-per-kmer statistics |
| `--stats` | single index only | Print per-genome kmer counts as CSV |
| `--partition-stats` | one or more indexes | Print a partition-size distribution report |
| `--csv FILE` | with `--partition-stats` | Also write raw per-(partition, source) data to FILE as CSV |
At least one operation option must be given. All options except `--partition-stats` require exactly one index directory.