Refactor the handling of genome absence to be a distinct state unless `--free-loss` is specified. This change introduces `QuartetConvention` to pair ambiguity policies with the free-loss flag, updating file naming conventions and logic across sampling, masking, and stats modules. - Update `--exclude-genome` description in `pair_coverage_sampling.md` to clarify absence as a state unless `--free-loss` is specified. - Move multi-member genome handling convention from `pair_coverage_sampling.md` to file name conventions. - Clarify persisted annex structure in `pair_coverage_sampling.md` as `PersistentBitVec` per non-monomorphic family with counting convention carried by file name. - Fix initial version's absence handling in `has_bipartition`, the annex, and `--sample-stats` to correctly count index 0 when absence is a state. - Replace entropy-only builder with `AnnexRequest` and `ensure_layer_family_annexes`. - Consolidate sampling parameters into a single `SamplingSpec`. - Update family constraint description in `sampling_design.md`. - Modify uniform draws and `--quartet` removal text in `sampling_design.md`. - Clarify state counting in tests for `--free-loss` option in `sampling_design.md`. - State that absence forms a pair without `--free-loss` but never with it in `sampling_design.md`. - Update `--quartet` option to require at least two distinct states, each carried by at least two genomes. - Define absence as a state unless `--free-loss` is given for family carrier definition. - Clarify `--pair-balance` favors families covering pairs with fewest shared families. - Modify logic for families fitting every tree topology equally well in `phylo.md`. - Change `--quartet` restriction logic to apply after pre-checks involving `--no-ambiguity` and `--exclude-genome`. - Update `obikmer` dependency version from 1.4.4 to 1.4.5 in `Cargo.lock` and `Cargo.toml`. - Modify `annex_selection` in `args/mod.rs` to derive `free_loss`, `entropy`, `quartet`, and `overlap` from `SamplingSpec`. - Update `SamplingArgs::quartet` documentation in `sampling.rs` to describe state determination based on `--free-loss`. - Add `free_loss` field to `AnnexSelection` struct in `family_annexes.rs`. - Change `AnnexRequest` to use `Vec<QuartetConvention>` instead of `Vec<AmbiguityPolicy>`. - Update `request` logic in `family_annexes.rs` to conditionally create `QuartetConvention` based on `self.quartet`. - Modify `dedup` signature in `family_annexes.rs` to accept iterator of type `T: Ord`. - Update `ensure_layer_family_annexes` in `family_annexes.rs` to use `missing_files` and `QuartetConvention`. - Add `missing_files` definition to `family_annexes.rs`. - Change `has_bipartition` in `masking.rs` to filter counts where count is `>= 2`. - Update `is_quartet_informative` in `masking.rs` to accept `free_loss` boolean parameter. - Modify `two_pairs_of_distinct_states_is_informative` and `singleton_variant_is_not_informative` in `masking.rs` to test both `free_loss` policies. - Add `under_free_loss_absence_is_missing_and_forms_no_pair` and `without_free_loss_absence_is_a_state_that_can_form_a_pair` definitions to `masking.rs`. - Update `no_ambiguity_drops_multi_member_genomes_from_the_count` in `masking.rs` to check `is_quartet_informative` under `free_loss=true`. - Add `the_annex_rule_and_the_exact_check_agree_on_the_same_family` definition to `masking.rs`. - Remove `absent_genomes_are_not_a_state` definition from `masking.rs`. - Update `restrict_to_quartet` in `quartet.rs` to accept `QuartetConvention` instead of `AmbiquityPolicy`. - Modify `add_site` in `sample_stats.rs` to handle state 0 as a state participating in bipartition. - Add `absence_as_a_state_makes_a_site_informative` definition to `sample_stats.rs`. - Add `quartet_convention` function to `SamplingSpec` in `subsample.rs`. - Update `sample_index` in `subsample.rs` to use `spec.quartet_convention()` for `AnnexRequest`. - Update `restrict_to_quartet` call in `subsample.rs` to use `spec.quartet_convention()`. - Add comment regarding `--no-ambiguity` counting convention in `helpers.rs`. - Import `QUARTET_ANNEX_FILE_NAMES` and `QuartetConvention` from `quartet_annex` in `mod.rs`. - Add `QuartetConvention` struct with `policy` and `free_loss` fields to `quartet_annex.rs`. - Modify `quartet_annex_file_name` in `quartet_annex.rs` to take `QuartetConvention`. - Add `the_conventions_never_share_a_file` function to `quartet_annex.rs`. - Remove `the_policies_never_share_a_file` definition from `quartet_annex.rs`.
obikmer
obikmer is a Rust toolkit for indexing, querying, and comparing DNA sequences
represented as sets of k-mers. It targets individual genome datasets (tens of
Gbases) with maximum efficiency in computation, memory, and disk usage.
Key principles
Compact k-mer encoding. Each k-mer is stored in a u64 at 2 bits/base.
k is odd, k ∈ [11, 31], fixed at runtime. The canonical form min(kmer, revcomp(kmer))
halves the effective space by collapsing both strands.
Superkmer-based partitioning. Sequences are decomposed into superkmers — maximal runs of k-mers sharing the same minimizer. Superkmers route naturally to partitions via the minimizer hash, enabling partition-parallel indexing and querying with no cross-partition communication.
Layered MPHF index. Each partition holds a stack of layers. Each layer is a minimal perfect hash function (MPHF) over the k-mers of one input genome, paired with a per-genome presence/count matrix. Queries scatter k-mers to their partition, probe each layer in order, and aggregate results.
Approximate indexing (Findere). With -z Z, the index stores k-mers of size
s = k − z + 1 instead of k. At query time, results are produced at size s, then
a per-genome sliding window of size z aggregates z consecutive s-mer hits into one
confirmed k-mer answer. This reduces the false-positive rate from 1/2^b per s-mer
to 1/2^(b·z) per k-mer, at the cost of z−1 unconfirmed positions at each sequence
break. The aggregation window spans the full query sequence, not individual superkmers,
to avoid false negatives at superkmer boundaries.
Multi-genome. A single index can hold any number of genomes. Each k-mer slot carries a per-genome count or presence vector. Distance matrices, NJ/UPGMA trees, and classification are derived from these vectors without rebuilding the index.
Input formats
Command Formats accepted
─────────────────── ──────────────────────────────────────────────────────────────
index, superkmer FASTA (.fa .fasta), FASTQ (.fq .fastq), GenBank flat file
(.gb .gbk .gbff), all optionally gzip-compressed.
Directories expanded recursively. Streaming stdin via -.
query FASTA, FASTQ, optionally gzip-compressed. Stdin via -.
Non-ACGT characters act as hard breaks between k-mer segments in all formats.
Commands
Command Role
───────── ────────────────────────────────────────────────────────────────────
index Build a genome index from sequence files.
Runs scatter → dereplicate → count → layered MPHF.
Resumes automatically if interrupted.
merge Merge multiple independently built indexes into one.
Schedules partitions largest-first under a memory budget semaphore
to avoid OOM on machines with many cores. The worst partition runs
alone first to calibrate the expansion estimator; subsequent
partitions run in parallel within the budget.
--budget-fraction F fraction of available RAM to use as budget
(default 0.5; reduce if OOM persists).
filter Filter and compact an existing index: apply count thresholds,
drop layers, rewrite as a single-layer index.
reindex Convert evidence in-place across all layers:
exact (evidence.bin) ↔ approximate (fingerprint.bin).
Does not touch the MPHF or unitigs.
query Query an index with FASTA/FASTQ sequences.
Annotates each sequence with per-genome k-mer match counts
and optional per-position coverage vectors (--detail).
Parallel over sequence chunks.
phylo Compute pairwise evolutionary-distance proxies between all
indexed genomes (Bray-Curtis, Jaccard, etc.), optionally
a Newick NJ/UPGMA tree, and optionally a central-position
SNP/Sankoff calibration with exports for external
phylogenetic tools (TNT, PhyG, IQ-TREE).
annotate Add or update genome metadata (taxonomy, etc.) from a CSV
file; or dump the current metadata as CSV.
estimate Dry-run: resolve and print approximate-index parameters
(z, evidence bits b, FP rates) given any two of (b, z, fp).
Does not touch any index.
dump Dump all indexed k-mers as CSV with per-genome counts or
presence flags.
superkmer Extract superkmers from a sequence file and write to stdout.
Diagnostic / pipeline use.
unitig Dump the unitig sequences stored in a built index. Debug use.
utils Miscellaneous utilities.
--new-label NEW=OLD rename a genome label in-place.
--bits-per-kmer print MPHF / evidence / matrix size breakdown.
--stats per-genome k-mer counts as CSV.
--partition-stats partition size distribution across one or more
indexes (markdown report to stdout). Useful to
diagnose minimizer imbalance before a large merge.
--csv FILE write per-(partition, source) raw data to FILE
(used with --partition-stats).
Quick start
# Build an exact index for each genome independently
obikmer index --kmer-size 31 --label genome_a genome_a.fa --output index_a/
obikmer index --kmer-size 31 --label genome_b genome_b.fa --output index_b/
# Merge into a single multi-genome index
obikmer merge --output index/ index_a/ index_b/
# Convert to approximate index (z=5, 8-bit fingerprints)
obikmer reindex --approx -z 5 --evidence-bits 8 index/
# Query reads
obikmer query index/ reads.fq.gz > annotated.fa
# Pairwise distances
obikmer phylo index/ > distances.tsv
Parameter constraints
Parameter Constraint
───────────────────── ──────────────
k (--kmer-size) odd, 11 ≤ k ≤ 31
m (--minimizer-size) odd, 3 ≤ m ≤ k−1
z (-z, --approx only) 1 ≤ z ≤ k−1
Documentation
Extended architecture and implementation notes are in DevDocMD/. Build with
make doc (requires Python + MkDocs Material). User-facing documentation is
in UserDocMD/, built with make doc-user.