large refactoring

DevDocMD/theory/encoding.md

@@ -0,0 +1,42 @@
+# DNA encoding
+
+## 2-bit nucleotide encoding
+
+All nucleotides are encoded on 2 bits, MSB-first within each word. Nucleotides are numbered 0-based from the 5′ end across all sequence types:
+
+| Base | Encoding |
+|------|----------|
+| A    | `00`     |
+| C    | `01`     |
+| G    | `10`     |
+| T    | `11`     |
+
+The Watson-Crick complement of any base is its bitwise NOT on 2 bits: `complement(base) = ~base & 0b11`.
+
+## Kmer encoding
+
+A kmer fits in a single `u64`. Nucleotide 0 occupies bits 63–62, nucleotide i occupies bits 63−2i and 62−2i, and the low 64−2k bits are zero. Extraction of nucleotide i (0 ≤ i < k): `(kmer >> (62 - 2*i)) & 0b11`.
+
+Reverse complement is computed by **bit manipulation in four steps**, with no lookup table:
+
+!!! abstract "Algorithm — Kmer reverse complement"
+    ```text
+    procedure KmerRevcomp(kmer, k):
+        x ← ~kmer                                           -- complement all bases
+        x ← swap_bytes(x)                                   -- reverse byte order
+        x ← ((x >> 4) & 0x0F0F0F0F0F0F0F0F)
+           | ((x & 0x0F0F0F0F0F0F0F0F) << 4)               -- swap nibbles within each byte
+        x ← ((x >> 2) & 0x3333333333333333)
+           | ((x & 0x3333333333333333) << 2)                -- swap 2-bit pairs within each nibble
+        return x << (64 - 2*k)                              -- re-align to MSB
+    ```
+
+The three reorder passes together reverse the order of all 2-bit base codes across the 64-bit word. The bitwise NOT in the first step complements each base (A↔T, C↔G). The final left shift clears the low 64−2k padding bits.
+
+The **canonical form** is the lexicographic minimum of the kmer and its reverse complement:
+
+```
+canonical(kmer) = min(kmer, revcomp(kmer))
+```
+
+This halves the kmer space and ensures strand-independent counting.

DevDocMD/theory/encoding.refs.md

@@ -0,0 +1,11 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/encoding.md
+
+## Code couvert
+
+- `obikseq/src/kmer.rs` — encodage 2 bits/base, revcomp, forme canonique
+
+## Notes
+
+Document purement théorique. Peu de risque de dérive sauf si l'encodage interne de Kmer change.
+Vérifier que la table d'encodage A=00, C=01, G=10, T=11 est toujours celle du code.

DevDocMD/theory/entropy.md

@@ -0,0 +1,84 @@
+# Kmer entropy filter
+
+Low-complexity kmers (polyA, polyT, tandem repeats) are detected and excluded during phase 1. The filter computes a **normalized Shannon entropy** over sub-words of multiple sizes, corrected for one source of bias: the small number of observations within a single kmer relative to the number of possible sub-words.
+
+## Sub-word frequencies
+
+For a kmer of length k and a sub-word size ws (1 ≤ ws ≤ ws_max, typically ws_max = 6), extract the $n_{\text{words}} = k - ws + 1$ overlapping sub-words by sliding a window of length ws:
+
+$$w_i = \text{kmer}[i \mathinner{..} i+ws-1], \quad i = 0, \ldots, n_{\text{words}}-1$$
+
+Each sub-word is tallied under its own raw 2-bit-packed value — **no canonicalization**. Let $f_j$ be the count of raw word $j$ among the $n_{\text{words}}$ sub-words ($\sum_j f_j = n_{\text{words}}$), over the $4^{ws}$ possible raw words.
+
+An earlier version of this filter first folded each sub-word into a circular+reverse-complement equivalence class, then "unfolded" the observed class frequency back onto its members to correct for unequal class sizes. That machinery bought nothing it was claimed for — see *Why no equivalence classes* below — while measurably weakening detection of the very sequences the filter exists to catch, so it was removed.
+
+## Corrected Shannon entropy
+
+$$H_{\text{corr}} = \log(n_{\text{words}}) - \frac{1}{n_{\text{words}}} \sum_j f_j \log f_j$$
+
+This is a plain Shannon entropy over the observed raw-word frequencies.
+
+## Maximum entropy correction for small samples
+
+With only $n_{\text{words}}$ observations over $4^{ws}$ possible raw words, the achievable maximum entropy is bounded by the most uniform integer distribution over $4^{ws}$ categories.
+
+Let $c = \lfloor n_{\text{words}} / 4^{ws} \rfloor$ and $r = n_{\text{words}} \bmod 4^{ws}$. The most uniform integer distribution assigns frequency $c+1$ to $r$ categories and $c$ to the remaining $4^{ws} - r$, with the convention $0 \log 0 = 0$:
+
+$$H_{\max} = -\left[(4^{ws} - r)\,\frac{c}{n_{\text{words}}}\log\frac{c}{n_{\text{words}}} + r\,\frac{c+1}{n_{\text{words}}}\log\frac{c+1}{n_{\text{words}}}\right]$$
+
+When $n_{\text{words}} < 4^{ws}$: $c=0$, $r=n_{\text{words}}$, and the formula reduces to $H_{\max} = \log(n_{\text{words}})$ — a single unified expression covers both regimes. A truly random sequence achieves $H_{\text{corr}} \approx H_{\max}$.
+
+## Normalized entropy
+
+$$\hat{H}(ws) = \frac{H_{\text{corr}}}{H_{\max}} \in [0, 1]$$
+
+## Final score
+
+The filter computes $\hat{H}(ws)$ for each word size ws from 1 to ws_max and returns the **minimum**:
+
+$$\text{entropy}(kmer) = \min_{ws=1}^{ws_{\max}} \hat{H}(ws)$$
+
+A value near 0 indicates low complexity (e.g. AAAA…); near 1 indicates high complexity. A kmer is rejected if $\text{entropy}(kmer) < \theta$, where $\theta$ is a collection parameter (default 0.7). The minimum across word sizes ensures that any scale of repetition is detected independently: polyA is caught at ws=1, dinucleotide repeats at ws=2, etc.
+
+## Why no equivalence classes
+
+A prior design folded each sub-word into the canonical form of its circular-rotation + reverse-complement equivalence class before tallying, on the reasoning that (a) it guarantees $\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))$, and (b) collapsing phase-shifted repeats (e.g. `ATG` ≡ `TGA` ≡ `GAT`) into one class better reflects that they are "the same" low-complexity pattern.
+
+Both properties already hold for the raw, unfolded entropy above, without any class machinery:
+
+- **Reverse complement**: for any K of length n, window $j$ of $\text{revcomp}(K)$ equals $\text{revcomp}$ of window $(n{-}ws{-}j)$ of K. This is a bijection between the window sets under which each window maps to its own revcomp — and revcomp is itself a bijection (involution) on the space of raw ws-mers. So the multiset of raw-word frequencies for $\text{revcomp}(K)$ is exactly a relabeling of the multiset for K, and Shannon entropy — a function of the frequency multiset alone — is exactly invariant. No folding required, for any K.
+- **Tandem repeats**: a period-p repeat sampled by a stride-1 sliding window naturally cycles through its own rotations as raw tokens (e.g. `ATGATGATG…` yields the raw words `ATG`, `TGA`, `GAT` in rotation as the window slides). The low diversity this represents (few distinct raw words out of $4^{ws}$ possible) is already visible in the raw frequency distribution — no folding needed to detect it.
+
+What the fold-then-unfold step actually did was credit each observed class with the frequency of equivalence-class members that were **never observed on the read strand**, inflating $H_{\text{corr}}$ for genuine repeats. Worked example: k=31, ws=3, kmer = `ATG` repeated ($n_{\text{words}}=29$, all 29 windows fall into one class of size 6 under the old scheme — 3 rotations × forward/revcomp):
+
+| | $H_{\text{corr}}$ | normalized |
+|---|---|---|
+| old (folded, class size 6) | $\log 6 \approx 1.79$ | $\approx 0.53$ |
+| current (raw, unfolded) | $\log 3 \approx 1.10$ | $\approx 0.33$ |
+
+The gap is not a rounding artifact: per sub-word order, the folded score for this same repeat swings from 0.53 (ws=3, aligned with the period) up to **1.03** (ws=5, misaligned with the period) — i.e. a period-3 repeat could score *above* the theoretical maximum for a random sequence, depending on which ws happens to divide the repeat's period. The raw formula stays flat at ≈0.33–0.40 across ws=2..6 regardless of alignment, which is the robustness the "minimum across ws" design was meant to provide in the first place.
+
+## Interpretation as an effective number of classes
+
+$H_{\text{corr}}$ is a standard Shannon entropy over raw words, so the classical perplexity interpretation holds directly: $N_{\text{eff}} = e^{H_{\text{corr}}}$ is the number of equiprobable raw words that would yield the same entropy.
+
+For the normalised score $\hat{H}$, dividing by $H_{\max}$ changes the logarithm base:
+
+$$\hat{H} = \frac{\log N_{\text{eff}}}{\log N_{\max}} = \log_{N_{\max}} N_{\text{eff}} \quad \Longleftrightarrow \quad N_{\text{eff}} = N_{\max}^{\,\hat{H}}$$
+
+The property is preserved: $\hat{H}$ is the logarithm (in base $N_{\max}$) of the effective number of equi-represented raw words.
+
+In the large-sample limit ($n_{\text{words}} \gg 4^{ws}$), $N_{\max} \approx 4^{ws}$, giving:
+
+$$N_{\text{eff}} \approx 4^{ws \cdot \hat{H}}$$
+
+This has a clean interpretation: $ws \cdot \hat{H}$ is the **effective word length** (in bases) of a perfectly uniform distribution that would produce the same entropy. At $\hat{H} = 1$ the full space of $4^{ws}$ words is used; at $\hat{H} = 0.5$ with ws=2, only $4^1 = 4$ effective words out of 16 are occupied.
+
+In our actual regime, $n_{\text{words}}$ is small and $4^{ws}$ can exceed $n_{\text{words}}$, so $H_{\max} < \log(4^{ws})$ due to the small-sample correction. The exact effective count is $N_{\max}^{\hat{H}}$, not $4^{ws \cdot \hat{H}}$.
+
+## Properties
+
+The entropy score is a function of the kmer sequence alone — it does not depend on the surrounding context or on the position within any genome. Two consequences:
+
+- **Orientation invariance**: $\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))$ — see *Why no equivalence classes* above for why this holds without any explicit strand-folding step.
+- **Context independence**: the same kmer is always rejected or always kept, regardless of which genome it occurs in, where in that genome it appears, or which strand is considered. The filter defines a fixed partition of the kmer space into low-complexity and valid kmers.

DevDocMD/theory/entropy.refs.md

@@ -0,0 +1,16 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/entropy.md
+
+## Code couvert
+
+- `obikentropy/src/table.rs`, `obikentropy/src/tracker.rs` — formule d'entropie et tables de correction petits effectifs
+- `obikentropy/src/kmer_entropy.rs` — entropie d'un kmer isolé (`KmerEntropy`)
+- `obiskbuilder/src/rolling_stat.rs` — composition de `obikentropy::EntropyTracker` dans le suivi streaming (sélection de minimiseur + entropie)
+- `obiskbuilder/src/iter.rs`, `obiskbuilder/src/stream_iter.rs` — application du filtre lors du scatter (phase 1)
+
+## Notes
+
+Le repli en classes d'équivalence circulaires + brin inverse (décrit dans une version antérieure de ce document) a été supprimé : voir la section « Why no equivalence classes » de `entropy.md` pour la justification théorique et numérique.
+
+Vérifier que les paramètres `theta` et `level_max` dans le CLI
+(`obikmer/src/cli.rs` → `CommonArgs`) correspondent bien à ce qui est décrit.

DevDocMD/theory/indexing.md

@@ -0,0 +1,28 @@
+# Partitioning and indexing architecture
+
+The canonical minimizer of a super-kmer is hashed to produce a **p-bit routing value** (p is a collection-level parameter):
+
+```
+canonical minimizer → hash(minimizer) → p-bit value → PART → partition directory
+```
+
+PART is computed once at phase 1 to open the correct partition file, then discarded. It is recomputed on the fly at query time. It is never stored in the super-kmer header.
+
+Each partition holds one MPHF instance (phase 6) that indexes kmers as plain u64 values — the minimizer plays no role inside the partition.
+
+## Why hashing is necessary
+
+The canonical minimizer is an m-mer (m ∈ {9, 11, 13, 15}), encoded in 2m bits (18 to 30 bits). Its distribution over the $4^m$ possible values is **not uniform**: because the minimizer is the lexicographic minimum of a window of m-mers, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly by the raw minimizer value would produce severely unbalanced partitions.
+
+A hash function with good avalanche properties redistributes this skewed distribution uniformly over the $2^p$ partition slots. The key reason this works well is the **entropy gap**: p is chosen to be much smaller than 2m, so the hash compresses many distinct minimizer values into each partition slot. Even under strong bias in the minimizer distribution, as long as its effective entropy exceeds p bits — which holds comfortably since the set of distinct minimizers in any real dataset is far larger than $2^p$ — the load imbalance across partitions is negligible.
+
+## Parameter choices
+
+| m  | 2m (bits) | Typical p | Partitions |
+|----|-----------|-----------|------------|
+| 9  | 18        | 6–8       | 64–256     |
+| 11 | 22        | 8–10      | 256–1 024  |
+| 13 | 26        | 10–12     | 1 024–4 096|
+| 15 | 30        | 10–14     | 1 024–16 384|
+
+The hard constraint is p ≤ 2m: one cannot extract more bits of uniform randomness from a source than it contains. In practice p is chosen well below 2m, leaving a large entropy margin that absorbs the minimizer bias. For k=31, m=13, p=10: 1 024 partitions with comfortable balance.

DevDocMD/theory/indexing.refs.md

@@ -0,0 +1,12 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/indexing.md
+
+## Code couvert
+
+- `obikpartitionner/src/partition.rs` — routage par hash de minimiseur, choix des paramètres
+- `obikpartitionner/src/lib.rs` — structure KmerPartition, nombre de partitions
+
+## Notes
+
+Vérifier que la doc mentionne bien que le nombre de partitions est une puissance de 2
+(converti par `partitions_to_bits` dans `obikmer/src/cli.rs`).

DevDocMD/theory/minimizer.md

@@ -0,0 +1,73 @@
+# Minimizer selection
+
+## Definition
+
+A **minimizer** of a k-mer window is the m-mer (m < k) with the smallest value under some total order ≺ among all k − m + 1 overlapping m-mers in the window. The minimizer is always taken in **canonical form** (lexicographic minimum of forward and reverse complement) to ensure strand-independence.
+
+The minimizer partitions the sequence into **super-kmers**: maximal contiguous runs of overlapping k-mers that share the same minimizer. A single minimizer anchors each super-kmer, enabling partitioned storage and indexing.
+
+## Lexicographic ordering and its bias
+
+The classical definition uses lexicographic order on the canonical m-mer value. In 2-bit encoding (A=00, C=01, G=10, T=11), the canonical form is $\min_{\text{lex}}(\text{fwd}, \text{rc})$, so AT-rich m-mers have systematically small values:
+
+$$\text{canonical}(\text{AAAA}\cdots\text{A}) = \text{canonical}(\text{TTTT}\cdots\text{T}) = 0$$
+
+Since small values always win the lex comparison, low-complexity AT-rich m-mers dominate as minimizers across large genomic regions. On real metagenomics data with k=31, m=11 and 256 partitions, this produces a max/min partition ratio of ≈ 2.75 — and a single pathological partition when the hash function has a fixed point at 0.
+
+## Random minimizer
+
+A **random minimizer** replaces lex order with a hash order: define $H : \{0,1\}^{2m} \to \{0,1\}^{64}$ and select the m-mer with the **minimum $H$ value** in the window.
+
+The key property: because $H$ is a bijection with well-distributed outputs, each distinct m-mer in the window has equal probability of holding the minimum hash value. Selection probability is no longer correlated with nucleotide composition.
+
+## Why the canonical form remains lexicographic
+
+An apparent alternative is to redefine the canonical form of each m-mer as the strand with the smaller hash value:
+
+$$\text{canonical}_H(v) = \arg\min(H(\text{fwd}),\ H(\text{rc}))$$
+
+This must be rejected. The hash of this new canonical is $\min(H(\text{fwd}), H(\text{rc}))$ — the minimum of two i.i.d. Uniform$[0, 2^{64})$ values. Its distribution is:
+
+$$F(x) = 1 - \left(1 - \frac{x}{2^{64}}\right)^2$$
+
+with density $f(x) = 2(1 - x/2^{64})$, which is approximately **twice as large near 0 than near $2^{64}$**. The low-order partition bits inherit this bias: partition 0 receives roughly twice as many super-kmers as the last partition.
+
+The lex canonical form does not have this problem: $\text{canonical}_{\text{lex}}(v)$ is a fixed, deterministic representative of each equivalence class, and $H(\text{canonical}_{\text{lex}})$ is uniformly distributed over $[0, 2^{64})$ independently of the min/max relationship between the two strands.
+
+## Partition key independence
+
+A further subtlety arises when the selection hash is used directly as the partition key. The selected minimizer is the m-mer with the **minimum** $H$ value in a window of $W = k - m + 1$ positions. The minimum of $W$ i.i.d. Uniform$[0,2^{64})$ values has distribution:
+
+$$F(x) = 1 - \left(1 - \frac{x}{2^{64}}\right)^W \approx \frac{Wx}{2^{64}}$$
+
+concentrated near 0 relative to the full range. Using this minimum-hash directly as the partition key creates the same bias as lex ordering, just distributed differently.
+
+The correct approach is to decouple selection from partition routing:
+
+- **Selection** uses $H(\text{canonical}_{\text{lex}}(m\text{-mer}))$ to pick the minimizer in the window.
+- **Partition routing** recomputes $H(\text{canonical}_{\text{lex}}(\text{minimizer}))$ from the stored minimizer position. This is the hash of a specific kmer value, not the minimum of a window — it is uniformly distributed over $[0, 2^{64})$.
+
+## Seed and fixed-point elimination
+
+The splitmix64 finalizer has a fixed point at 0:
+
+$$\text{mix64}(0) = 0$$
+
+Since $\text{canonical}_{\text{lex}}(\text{AAAA}\cdots\text{A}) = 0$, using unseeded mix64 causes all-A m-mers to win every window comparison, recreating a pathological partition identical to the lex-ordering bias.
+
+The fix is a non-zero XOR seed applied before mixing:
+
+$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
+
+where $\varphi$ is the golden ratio. This maps 0 to $\text{mix64}(s)$, a well-distributed non-zero value. No canonical m-mer value has a systematically small $H$.
+
+!!! abstract "Hash function $H$"
+    ```
+    H(x):
+        x ← x  ⊕  0x9e3779b97f4a7c15
+        x ← x  ⊕  (x >> 30)
+        x ← x  ×  0xbf58476d1ce4e5b9
+        x ← x  ⊕  (x >> 27)
+        x ← x  ×  0x94d049bb133111eb
+        return x ⊕ (x >> 31)
+    ```

DevDocMD/theory/minimizer.refs.md

@@ -0,0 +1,12 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/minimizer.md
+
+## Code couvert
+
+- `obiskbuilder/src/lib.rs` — sélection du minimiseur par hash seedé (splitmix64 finalizer)
+- `obikseq/src/superkmer.rs` — forme canonique du minimiseur, fenêtre glissante
+
+## Notes
+
+Vérifier que la fonction de hash décrite (splitmix64 finalizer avec graine) correspond
+au code actuel. Vérifier aussi que la définition de « minimiseur canonique » est toujours cohérente.