# obikmer

`obikmer` is a Rust tool for manipulation, counting, indexing, and set operations on DNA sequences represented as kmer sets.

## Constraints

- Target scale: metagenomic data, tens of Gbases, billions of kmers
- Maximum efficiency in computation, memory, and disk usage
- k is odd, k ∈ [11, 31], fixed at runtime
- Input formats: FASTA, FASTQ, gzip, streaming stdin

## Input processing

For sequence files (FASTA/FASTQ):

- Cut sequence on non-ACGT nucleotides.
- Discard fragments < k.
- For sequences > 256 nt: cut at 256, start new with k-1 overlap (preserves all kmers, no duplicates).

## Super-kmers

The fundamental stored unit is a canonical super-kmer: a maximal run of consecutive kmers sharing the same non-canonical (raw forward) minimizer, canonized at the end.

### Construction

Super-kmers are built using the non-canonical minimizer (minimum forward m-mer, no revcomp comparison). Two consecutive kmers belong to the same super-kmer if and only if their raw forward minimizer is identical.

At the end of each super-kmer, the minimizer is canonized: `canonical(M) = min(M, revcomp(M))`.

- If `canonical(M) == M`: the super-kmer is already in canonical orientation — keep as-is.
- If `canonical(M) == revcomp(M)`: reverse-complement the entire super-kmer (and its minimizer).

The reverse-complement step does not cause any additional splitting — it only flips the orientation.

### Why non-canonical minimizer?

Using the raw forward minimizer (rather than the canonical minimizer) means fewer candidates are considered at each position → the minimizer changes more often → super-kmers are shorter or equal, never longer, compared to the canonical-minimizer definition.

The key benefit is at dereplication: a forward read and its reverse-complement covering the same genomic region both produce the same canonical super-kmer after the canonization step. Under the canonical-minimizer definition, they would share the same canonical minimizer but have different sequences (one being the revcomp of the other) — they would not dereplicate. Here they do, giving a compression factor of ~2× on paired or double-stranded data.

### Length

For a random minimizer of length m over k-mers of length k, the density of minimizer positions is approximately 2/(k−m+2) [@Zheng2020-ji; @Golan2025-xf], giving an expected super-kmer length of (k+m)/2 nucleotides. For k=31, m=13: expected ≈ 22 nt. In practice super-kmers rarely exceed a few dozen nucleotides — the 256 nt cap is almost never reached. When it is, the super-kmer is split at 256 nt with a k−1 overlap, preserving all kmers without duplication.

## Binary storage format

Memory-mappable binary format for sequence collections: store super-kmers end-to-end with u64-aligned headers (add padding if needed for 8-byte alignment). Include index blocks every B super-kmers (B=1000-10000) for fast jumping to position J (O(N/B + B) scan vs. separate index O(1) direct access), where N is total super-kmers in file. Represent header as union/struct for field access.

## Partitioning and indexing architecture

The canonical minimizer of a super-kmer is hashed to produce a p-bit routing value (p is a collection-level parameter):

    canonical minimizer → hash(minimizer) → p-bit value → PART → partition directory

PART is computed once at phase 1 to open the correct partition file, then discarded. It is recomputed on the fly at query time. It is never stored in the super-kmer header.

Each partition holds one BBHash instance (phase 6) that indexes kmers as plain u64 values — the minimizer plays no role inside the partition.

Properties: - 2^p partitions: p freely tunable at collection creation; not constrained to powers of 4 - Uniform distribution: hashing the canonical minimizer corrects the lexicographic bias inherent in minimizer selection [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf] - The odd-length constraint on m only ensures canonical form correctness during minimizer computation — it does not affect the partition layout

Entropy constraint: the canonical minimizer has 2m bits of entropy. Requesting p bits from the hash requires 2m ≥ p, i.e. p ≤ 2m:

m | p max | Partitions max
--- | --- | ---
9 | 2 | 4
11 | 6 | 64
13 | 10 | 1 024
15 | 14 | 16 384

For the typical configuration k=31, m=13: up to 1 024 partitions. Choosing p > 2m−16 is possible but produces artificially colliding PART values with no gain in routing granularity.

## Construction pipeline

All phases after scatter are embarrassingly parallel across partitions.

### Phase 1 — Scatter

Single streaming pass over raw input files (FASTA/FASTQ, gzip). FASTQ quality scores are ignored. For each read:

1. Ambiguous base filter: cut at any non-ACGT base; discard fragments shorter than k.
2. Entropy filter: scan each fragment with a sliding window of size k. When the kmer $K_i = S[i \mathinner{..} i+k-1]$ ended by nucleotide S[j] (with j=i+k− 1) has entropy below threshold θ, emit the current segment and start a new one (see algorithm below). Ki belongs to neither segment, and no valid kmer is lost.
3. Super-kmer extraction: for each clean segment, slide a minimizer window and group consecutive kmers sharing the same raw minimizer; canonise at the end.
4. Partition routing: `hash(canonical_minimizer) → PART` → append super-kmer to `partition/superkmers.bin.gz`.

Entropy filter algorithm:

When Ki (ended by S[j], j=i+k− 1) fails the entropy threshold:

- Current segment $S[\textit{seg_start} \mathinner{..} j-1]$ is emitted (last valid kmer = Ki− 1)
- New segment starts at S[i+ 1] (first new kmer = Ki+ 1)
- Ki is excluded: current segment lacks S[j], new segment lacks S[i]
- Overlap = $S[i+1 \mathinner{..} j-1]$ = k− 2 nucleotides

    \begin{algorithm}
    \caption{Entropy filter — sliding window segmentation}
    \begin{algorithmic}
    \PROCEDURE{EntropyFilter}{S, N, k, theta}
        \STATE seg_start <- 0
        \STATE window <- []
        \FOR{j <- 0 \TO N-1}
            \STATE window.push(S[j])
            \IF{|window| < k}
                \STATE continue
            \ENDIF
            \STATE i <- j - k + 1
            \IF{entropy(window) <= theta}
                \STATE emit S[seg_start .. j-1]
                \STATE seg_start <- i + 1
                \STATE window <- S[i+1 .. j]
            \ELSE
                \STATE window.pop_front()
            \ENDIF
        \ENDFOR
        \STATE emit S[seg_start .. N-1]
    \ENDPROCEDURE
    \end{algorithmic}
    \end{algorithm}

Writes are sequential and append-only — IO-friendly. Gzip applied at write time. Data volume ≈ raw genome size (2 bits/nt compaction offsets header overhead).

### Phase 2 — Dereplication

Performed independently per partition. Identical super-kmers are consolidated and their COUNT accumulated — analogous to amplicon dereplication in metabarcoding. Uses external bucket sort to stay within RAM bounds:

Pass 1 (streaming): hash the nucleotide payload of each super-kmer, route to one of B bucket files:

    hash(sequence) % B → bucket_i.bin

B ≈ 100 is tunable; RAM needed ≈ partition_size / B.

Pass 2: for each bucket, load into an in-memory `HashMap<sequence, COUNT>`, dereplicate by summing COUNT values, write consolidated super-kmers.

After dereplication: at Nx coverage the partition shrinks by ~x (errors aside). The COUNT field in each super-kmer header = number of times that exact super-kmer sequence was observed across all input reads.

Important: super-kmer COUNT ≠ individual kmer count. A kmer can appear in multiple distinct super-kmers (same partition, different flanking context); its true count = sum of COUNT of all super-kmers containing it. A super-kmer with COUNT=1 may contain only high-abundance kmers, each appearing in many other super-kmers. Abundance filtering therefore cannot be applied at this phase.

### Phase 3 — Per-kmer count aggregation and quorum filtering

For each dereplicated super-kmer, enumerate its kmers and accumulate counts:

    for each super-kmer (sequence, COUNT):
        for each kmer in sequence:
            kmer_counts[canonical(kmer)] += COUNT

Implemented as an external sort or a temporary HashMap, depending on partition size. At the end of this phase, each distinct canonical kmer has its exact total count.

Abundance filter applied here: kmers with `total_count < q` are discarded. `q` is a collection parameter (0 = keep all, including singletons for ≤1x data).

No pre-filter on super-kmer COUNT is possible at phase 2: a super-kmer with COUNT=1 may contain only high-abundance kmers, each present in many other super-kmers across the partition.

### Phase 4 — Super-kmer compaction

The valid kmer set from phase 3 is used as a mask to rewrite the super-kmer files:

    for each dereplicated super-kmer:
        scan kmer by kmer
        kmer not in valid set → break point (terminates current super-kmer)
        kmer in valid set     → extend current super-kmer

Three cases per super-kmer: - All kmers valid → copied as-is - No kmer valid → discarded - Mixed → split into sub-super-kmers at invalid boundaries; each sub-super-kmer inherits the original COUNT

After splitting, re-apply dereplication (bucket sort, phase 2 method) — splitting can produce new identical super-kmers. This re-dereplication is cheap: the volume is already greatly reduced.

Output: a clean super-kmer file where every kmer passes quorum. This file feeds phase 5.

### Phase 5 — Local de Bruijn graph and unitig construction

Within each partition, build a local de Bruijn graph from the valid kmer set and compute its unitigs. All operations are local to the partition — no cross-partition communication.

    valid kmers → HashSet<u64>

    for each kmer K:
        out_degree = |{K[1:]+b | b ∈ {A,C,G,T}} ∩ HashSet|
        in_degree  = |{b+K[:-1] | b ∈ {A,C,G,T}} ∩ HashSet|

    internal node ↔ in_degree=1 AND out_degree=1
    branching / dead-end → unitig start or end

Traverse non-branching paths to assemble unitigs. Kmers whose neighbours fall in other partitions appear as dead ends locally — they terminate the unitig. The result: each kmer appears in exactly one unitig within the partition.

The partition size (controlled by p) must be calibrated so that the HashSet fits in RAM during this phase.

Output: `unitigs.bin` — the permanent evidence structure for the partition. Each kmer in the partition appears at exactly one (unitig_id, offset) location.

Scope of local unitigs: these are unitigs of the partition’s local de Bruijn graph, not global unitigs. A kmer whose k-1 successor or predecessor falls in another partition appears as a dead end locally and terminates the unitig. This does not affect correctness of verification but means partition-local unitigs cannot be directly reused for global assembly.

### Phase 6 — BBHash construction and index finalisation

Built once on the definitive kmer set (all kmers in all unitigs of the partition):

    kmers from unitigs → BBHash → bbhash.bin
                       → counts.bin : packed n-bit array (or 1-bit for presence mode)
                       → refs.bin   : (unitig_id: u32, kmer_offset: u8) per kmer

BBHash is built once — no rebuild. The n-bit width for `counts.bin` is chosen from the observed count distribution (n=5 covers ~97% of kmers at 15x; n=1 for presence mode). Counts exceeding 2ⁿ−1 go into `overflow.bin` as sorted `(bbhash_index: u32, count: u32)` pairs.

Exact verification via unitig evidence:

`unitigs.bin` serves as the evidence structure: for any query kmer, the stored unitig provides the ground truth to confirm or deny its presence. BBHash maps every input to [0, N) including absent kmers — the unitig read-back is the only way to guarantee exactness.

    query kmer q
      → minimizer(q) → hash → PART, KEY → partition
      → BBHash(q) → index i
      → refs[i] = (unitig_id, offset)
      → read unitig unitig_id from unitigs.bin
      → extract kmer at offset → compare with q
      → match   : return counts[i]   ← exact hit
      → no match: kmer absent        ← BBHash collision on absent kmer

One random disk access into `unitigs.bin` per query; the unitig is the minimal, non-redundant evidence (each kmer stored once).

## On-disk collection structure

Collections are too large to hold in RAM (hundreds of genomes, billions of kmers). The collection lives on disk as a directory of memory-mapped files:

    collection/
      metadata.toml          — collection parameters (see below)
      superkmers.bin         — super-kmer sequences, end-to-end, memory-mappable
      superkmers.idx         — index blocks every B super-kmers for fast seeking
      part_XXXX/
        bbhash.bin           — BBHash function for this partition
        counts.bin           — packed n-bit count array (or 1-bit presence array)
        refs.bin             — back-references for exact verification

Collection parameters (stored in `metadata.toml`):

Parameter | Role
--- | ---
k | kmer length
m | minimizer length (odd, < k)
p | partition bits (0 ≤ p ≤ min(14, 2m−16))
mode | `presence` (1 bit/kmer) or `count` (n bits/kmer)
n | bits per kmer in count mode (chosen at construction)
min_count | singleton filtering threshold (0 = keep all)
hash_fn | hash function identifier
hash_seed | seed for the hash function

Exact verification via back-reference:

BBHash maps any input to [0, N) — including absent kmers. To avoid false positives, `refs.bin` stores at each BBHash index a back-reference `(superkmer_id: u32, kmer_offset: u8)` = 5 bytes. Query protocol:

    query kmer q
      → minimizer(q) → hash → PART, KEY → partition
      → BBHash(q) → index i
      → refs[i] = (sk_id, offset)
      → read super-kmer sk_id from superkmers.bin
      → extract kmer at offset → compare with q
      → if match: return counts[i]   (exact)
      → if no match: kmer absent

Count storage — two modes:

Presence mode (coverage ≤ 1x, or when only presence/absence matters): - `counts.bin` is a packed 1-bit array — all bits set to 1 for indexed kmers - Singletons are the signal, not filtered

Count mode (coverage > 1x): - `counts.bin` is a packed n-bit array; n chosen at construction from the observed distribution - Value 0: absent sentinel; values 1..2ⁿ−2: direct count; value 2ⁿ−1: overflow - Overflow counts stored in a separate `overflow.bin` as sorted `(index: u32, count: u32)` pairs - Empirically (k=31, 15x coverage): n=5 covers 97% of real kmers, n=6 covers 99% - min_count threshold filters low-frequency kmers (errors) before indexing; for ≤1x, min_count=0

BBHash: operates on kmer u64 values directly within each partition. The minimizer plays no role inside the partition — BBHash receives the kmer and builds the perfect hash naturally.

## Kmer entropy filter

Low-complexity kmers (polyA, polyT, tandem repeats) are detected and excluded during phase 1. The filter computes a normalized Shannon entropy over sub-words of multiple sizes, corrected for two sources of bias: the small number of observations within a single kmer, and the unequal sizes of circular equivalence classes.

### Sub-word frequencies

For a kmer of length k and a sub-word size ws (1 ≤ ws ≤ ws_max, typically ws_max = 6), extract the nwords=k−w s+ 1 overlapping sub-words by sliding a window of length ws:

$$w_i = \text{kmer}[i \mathinner{..} i+ws-1], \quad i = 0, \ldots, n_{\text{words}}-1$$

Each sub-word is mapped to its circular canonical form: the lexicographic minimum among all cyclic rotations of the word. Let sj be the size of equivalence class j (number of distinct raw words mapping to canonical form j), and fj the count of canonical form j among the nwords sub-words (∑jfj=nwords).

### Corrected Shannon entropy

The circular equivalence classes have unequal sizes: under a uniform distribution over all 4w s raw words, class j is visited with probability sj/4w s, not 1/na. Computing entropy directly over canonical classes therefore underestimates the entropy of a random sequence.

The correction “unfolds” each canonical class back to its member raw words, redistributing each observation of class j equally among its sj members:

$$H_{\text{corr}} = \log(n_{\text{words}}) - \frac{1}{n_{\text{words}}} \sum_j f_j \log f_j + \frac{1}{n_{\text{words}}} \sum_j f_j \log s_j$$

The last term is the correction for unequal class sizes. For a uniformly random sequence (fj≈nwords⋅sj/4w s), this gives Hcorr≈ log (4w s) = 2 ⋅w s⋅ log 2, the maximum entropy over raw words.

### Maximum entropy correction for small samples

With only nwords observations over 4w s possible raw words, the achievable maximum entropy is bounded by the most uniform integer distribution over 4w s categories.

Let c= ⌊nwords/4w s⌋ and r=nwords mod  4w s. The most uniform integer distribution assigns frequency c+ 1 to r categories and c to the remaining 4w s−r, with the convention 0log 0 = 0:

$$H_{\max} = -\left[(4^{ws} - r)\,\frac{c}{n_{\text{words}}}\log\frac{c}{n_{\text{words}}} + r\,\frac{c+1}{n_{\text{words}}}\log\frac{c+1}{n_{\text{words}}}\right]$$

When nwords< 4w s: c= 0, r=nwords, and the formula reduces to Hmax= log (nwords) — a single unified expression covers both regimes. A truly random sequence achieves Hcorr≈Hmax.

### Normalized entropy

$$\hat{H}(ws) = \frac{H_{\text{corr}}}{H_{\max}} \in [0, 1]$$

### Final score

The filter computes Ĥ(w s) for each word size ws from 1 to ws_max and returns the minimum:

$$\text{entropy}(kmer) = \min_{ws=1}^{ws_{\max}} \hat{H}(ws)$$

A value near 0 indicates low complexity (e.g. AAAA…); near 1 indicates high complexity. A kmer is rejected if entropy(k m e r) ≤θ, where θ is a collection parameter. The minimum across word sizes ensures that any scale of repetition is detected independently: polyA is caught at ws=1, dinucleotide repeats at ws=2, etc.

## Priority operations

- Kmer counting (frequencies)
- Fast search / query
- Set operations: union, intersection, difference

## Data types

Two types of DNA are represented:

- Super-kmers: stored as a nucleotide sequence (2 bits/base, byte-aligned, no padding). Max length: 256 nucleotides (512 bits, 64 bytes). Each super-kmer has a 32-bit header:

Field | Bits | Role
--- | --- | ---
COUNT | 24 | Occurrence count (≤ 16 M)
SEQL | 8 | Sequence length in nucleotides (≤ 256)

Bit layout (MSB to LSB):

    [31:8] COUNT  [7:0] SEQL

Getters:

    fn seql(&self)         -> u8  { self.0 as u8 }
    fn count(&self)        -> u32 { self.0 >> 8 }
    fn increment(&mut self)       { self.0 += 1 << 8; }

PART and KEY were considered as routing fields derived from `hash(minimizer)` but are single-use at phase 1 only — neither is stored. The nucleotide sequence is always byte-aligned (no offset): after reverse-complementing a super-kmer, a single bit-shift realigns the sequence so that nucleotide 0 always starts at bit 0 of word[0]. This shift is paid once at canonisation (phase 1, at most half of super-kmers) and eliminates any need for an alignment offset field. The direct consequence for dereplication: the byte array can be hashed directly without any offset adjustment.

Note: minimizer selection (window minimum) biases the raw minimizer distribution toward lexicographically small values [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]; hashing corrects this before partition routing.

- Kmers: short DNA subsequences (k bases), represented in `u64`.

## Kmer representation

Each base is encoded on 2 bits, so any kmer with k ≤ 31 fits in a `u64`.

### Bit layout

Nucleotide `i` occupies bits `64-i` and `63-i`. For a kmer of length k, 2k bits are used; the low-order 64 - 2k bits of the u64 are kept at 0 and have no meaning.

- Extraction of nuc[i]: `(word >> (63 - 2*i)) & 0b11`.

For kmers (k ≤ 31), the whole sequence fits in a single `u64`. For longer sequences up to 256 nucleotides (512 bits), the representation extends across multiple words.

### Complement and reverse complement

The encoding is chosen so that the Watson-Crick complement of any base is its bitwise NOT (on 2 bits):

Base | Encoding | Complement | NOT
--- | --- | --- | ---
A | `00` | T | `11`
C | `01` | G | `10`
G | `10` | C | `01`
T | `11` | A | `00`

This means `complement(base) = ~base & 0b11`, and the reverse complement of a kmer reduces to two cheap operations:

    revcomp(kmer) = reverse_bases(~kmer & mask_2k)

where `mask_2k = (1 << 2k) - 1` zeros the unused high bits, and `reverse_bases` reorders the 2-bit chunks from LSB to MSB. No lookup table needed.

DNA being double-stranded, each kmer and its reverse complement are equivalent. The canonical form is defined as:

    canonical(kmer) = min(kmer, revcomp(kmer))

This halves the kmer space and ensures strand-independent counting.

## Sequence operations

### Reverse complement

Reverse complement of a super-kmer sequence (used at most once per super-kmer, during phase 1 canonisation):

1. Reverse byte order
2. Apply 256-entry lookup table (precomputed reverse-complemented bytes) to each byte
3. Bit-shift the result to realign nucleotide 0 to bit 0 of word[0]

The shift amount depends only on the sequence length: `shift = (len % 4) * 2` bits. After the shift, the sequence is byte-aligned with no offset — the same invariant as the original.

    fn reverse_complement(words: &[u64], len: usize) -> Vec<u64> {
        let bytes = words_to_bytes(words);
        let mut rev: Vec<u8> = bytes.iter().rev().map(|&b| LOOKUP_TABLE[b as usize]).collect();
        let shift = (len % 4) * 2;
        if shift > 0 { bit_shift_left(&mut rev, shift); }
        bytes_to_words(rev)
    }

## Multithreading

For HPC with 100+ cores: use Rayon for data parallelism (parallel iterators), std::thread for custom threading, or async with Tokio for I/O-bound tasks. Equivalent to Go’s goroutines but with ownership/borrowing safety.

Rayon: Rust crate for automatic data parallelism. Provides parallel iterators (par_iter) that distribute work across CPU cores, e.g., vec.par_iter().map(|x| x*2).collect(). Handles thread spawning/synchronization transparently. Widely used and maintained by the Rust project. Orthogonal to Tokio (async runtime): Rayon for CPU parallelism, Tokio for async I/O.

Data pipelines: use Crossbeam channels (MPMC) with threads/async, equivalent to Go’s channels + goroutines.

Channels: std::sync::mpsc (built-in, MPSC, simple); crossbeam (crate, MPMC, faster, more features like select). Use Crossbeam for multi-producer/multi-consumer scenarios.

# Bibliographie

\bibliography