large refactoring

DevDocMD/architecture/index_architecture.md

@@ -0,0 +1,313 @@
+# Kmer index architecture
+
+## Fundamental invariant
+
+A given canonical kmer belongs to **exactly one partition** and **exactly one layer** within that partition. This property makes all aggregation operations decomposable and parallelisable without coordination.
+
+---
+
+## Three-level hierarchy
+
+```
+KmerIndex  (index.meta + KmerPartition)
+├── partition_0/index/          one directory per minimiser bucket
+│   ├── meta.json               PartitionMeta { n_layers }
+│   ├── layer_0/
+│   │   ├── layer_meta.json     LayerMeta { evidence: EvidenceKind }
+│   │   ├── mphf.bin            PtrHash MPHF
+│   │   ├── unitigs.bin         unitig spine (never overwritten)
+│   │   ├── evidence.bin        exact evidence  (Exact only)
+│   │   ├── unitigs.bin.idx     block index     (Exact only)
+│   │   ├── fingerprint.bin     fingerprints    (Approx only)
+│   │   ├── counts/             PersistentCompactIntMatrix  (with_counts = true)
+│   │   └── presence/           PersistentBitMatrix
+│   └── layer_1/
+│       └── ...
+└── partition_1/index/
+    └── ...
+```
+
+**KmerIndex**: root entry point. Owns `IndexMeta` (written to `index.meta`) and a `KmerPartition` that routes canonical kmers to partition directories. All partition-level operations are dispatched in parallel via rayon.
+
+**Partition directory**: one directory per minimiser bucket. `PartitionMeta` (stored as `meta.json`) records `n_layers`. Layers within a partition cover disjoint kmer sets.
+
+**Layer directory**: one `MphfLayer` plus optional data stores. `LayerMeta` (stored as `layer_meta.json`) records which `EvidenceKind` was used. The MPHF and `unitigs.bin` are immutable once built; evidence files are the only part replaced by `reindex`.
+
+---
+
+## IndexConfig and IndexMeta
+
+```rust
+pub struct IndexConfig {
+    pub kmer_size:      usize,
+    pub minimizer_size: usize,
+    pub n_bits:         usize,       // log2(n_partitions)
+    pub with_counts:    bool,
+    pub evidence:       EvidenceKind,
+    pub block_bits:     u8,          // .idx granularity: 2^block_bits unitigs/block; 0 = one entry per unitig
+}
+
+pub struct IndexMeta {
+    pub version: u32,
+    pub config:  IndexConfig,
+    pub genomes: Vec<GenomeInfo>,    // ordered; index = genome column number
+}
+
+pub struct GenomeInfo {
+    pub label: String,
+    pub meta:  HashMap<String, String>,  // arbitrary categorical metadata
+}
+```
+
+`IndexMeta` is serialised as `index.meta` (JSON). It is the authority for the ordered list of genomes and for the parameters that govern all subsequent operations on the index.
+
+---
+
+## EvidenceKind
+
+```rust
+pub enum EvidenceKind {
+    Exact,
+    Approx { b: u8, z: u8 },
+}
+```
+
+Controls which files are written per layer and which query path is taken:
+
+| Variant | Files written | False-positive rate |
+|---|---|---|
+| `Exact` | `evidence.bin`, `unitigs.bin.idx` | 0 |
+| `Approx { b, z }` | `fingerprint.bin` | ≈ W / 2^(b·z) per read (Findere) |
+
+`EvidenceKind` is stored both in `IndexConfig` (index-wide default, updated by `reindex`) and in each `LayerMeta` (per-layer record of what was actually built).
+
+---
+
+## MphfLayer — autonomous kmer → slot mapping
+
+```rust
+pub struct MphfLayer {
+    mphf: PtrHash<…>,
+    ev:   LayerEvidence,   // Exact { evidence, unitigs } | Approx { fingerprint }
+    n:    usize,
+}
+```
+
+`MphfLayer::find(kmer)` dispatches transparently to `find_exact` or `find_approx` based on the evidence loaded at `open` time (read from `layer_meta.json`). Returns `Some(slot)` only if the kmer is confirmed present; `None` for absent or out-of-range.
+
+```
+find_exact:  slot = mphf(kmer); decode evidence → (chunk_id, rank); verify kmer in unitigs
+find_approx: slot = mphf(kmer); check fingerprint[slot] == seq_hash(kmer)
+```
+
+`block_bits` controls the `.idx` file written alongside `evidence.bin`. At `block_bits = 0`, every unitig chunk has an index entry, giving O(1) random access; larger values trade access time for a smaller `.idx`.
+
+The MPHF and `unitigs.bin` are never rebuilt by any post-build operation.
+
+---
+
+## Layer\<D\> — MPHF + data payload
+
+```rust
+pub struct Layer<D: LayerData = ()> {
+    mphf: MphfLayer,
+    data: D,
+}
+```
+
+`D` selects the attached data payload:
+
+| `D` | Data directory | `Item` returned by `query` |
+|---|---|---|
+| `()` | — | `()` (set membership only) |
+| `PersistentCompactIntMatrix` | `counts/` | `Box<[u32]>` (counts per genome) |
+| `PersistentBitMatrix` | `presence/` | `Box<[bool]>` (presence per genome) |
+
+`Layer::query(kmer)` delegates to `MphfLayer::find`, then calls `data.read(slot)` if a slot is returned. Both exact and approximate evidence are handled transparently; the caller sees only `Option<Hit<D::Item>>`.
+
+Build-time entry points:
+
+```rust
+Layer<()>::build(out_dir, block_bits)                         // set membership
+Layer<PersistentCompactIntMatrix>::build(out_dir, block_bits, count_of)
+Layer<PersistentBitMatrix>::build_presence(out_dir, block_bits, n_genomes, present_in)
+
+Layer::<()>::build_evidence(layer_dir, kind, block_bits)      // evidence only (reindex path)
+```
+
+---
+
+## DataStore — slot-indexed data
+
+`PersistentCompactIntMatrix` and `PersistentBitMatrix` are slot-indexed stores. They know nothing about kmers or MPHFs.
+
+| Type | `Item` | Aggregation method | Use |
+|---|---|---|---|
+| `PersistentCompactIntMatrix` | `Box<[u32]>` | `sum() → Array1<u64>` | counts per genome per slot |
+| `PersistentBitMatrix` | `Box<[bool]>` | `count_ones() → Array1<u64>` | presence per genome per slot |
+
+---
+
+## Aggregation traits — `obicompactvec::traits`
+
+Three traits unify the aggregation API across all hierarchy levels.
+
+```rust
+trait ColumnWeights: Send + Sync {
+    fn col_weights(&self) -> Array1<u64>;
+}
+
+trait CountPartials: ColumnWeights {
+    fn partial_bray(&self)                                       -> Array2<u64>;
+    fn partial_euclidean(&self)                                  -> Array2<f64>;
+    fn partial_threshold_jaccard(&self, threshold: u32)          -> (Array2<u64>, Array2<u64>);
+    fn partial_relfreq_bray(&self, global: &Array1<u64>)         -> Array2<f64>;
+    fn partial_relfreq_euclidean(&self, global: &Array1<u64>)    -> Array2<f64>;
+    fn partial_hellinger(&self, global: &Array1<u64>)            -> Array2<f64>;
+    // provided finalisation methods with default impls
+    fn bray_dist_matrix(&self)                                   -> Array2<f64> { … }
+    fn relfreq_bray_dist_matrix(&self)                           -> Array2<f64> { … }
+    // …
+}
+
+trait BitPartials: ColumnWeights {
+    fn partial_jaccard(&self)  -> (Array2<u64>, Array2<u64>);
+    fn partial_hamming(&self)  -> Array2<u64>;
+    // provided
+    fn jaccard_dist_matrix(&self) -> Array2<f64> { … }
+    fn hamming_dist_matrix(&self) -> Array2<u64> { … }
+}
+```
+
+Leaf implementors:
+
+| Type | Traits |
+|---|---|
+| `PersistentCompactIntMatrix` | `ColumnWeights`, `CountPartials` |
+| `PersistentBitMatrix` | `ColumnWeights`, `BitPartials` |
+
+---
+
+## LayeredStore\<S\> — recursive aggregation wrapper
+
+```rust
+pub struct LayeredStore<S>(Vec<S>);
+```
+
+Three blanket impls propagate all traits up the hierarchy:
+
+```rust
+impl<S: ColumnWeights> ColumnWeights for LayeredStore<S> { … }
+impl<S: CountPartials> CountPartials  for LayeredStore<S> { … }
+impl<S: BitPartials>   BitPartials    for LayeredStore<S> { … }
+```
+
+This makes `LayeredStore<LayeredStore<PersistentCompactIntMatrix>>` automatically implement `CountPartials` — no separate `PartitionedStore` type is needed:
+
+```
+PersistentCompactIntMatrix                         leaf (one layer)
+LayeredStore<PersistentCompactIntMatrix>           one partition (layers are disjoint)
+LayeredStore<LayeredStore<…>>                      whole index   (partitions are independent)
+```
+
+Normalised metrics require global column sums — computed in a two-pass cascade:
+
+```rust
+// on LayeredStore<LayeredStore<PersistentCompactIntMatrix>>
+fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
+    let global = self.col_weights();             // pass 1 — sums up hierarchy
+    let p      = self.partial_relfreq_bray(&global); // pass 2 — global broadcast read-only
+    p.mapv(|v| 1.0 - v)
+}
+```
+
+Because each kmer belongs to exactly one `(partition, layer)` pair, `col_weights()` has no double-counting across the hierarchy.
+
+---
+
+## Progressive aggregation principle
+
+No level reaches two levels down. Each level sums contributions from the level immediately below:
+
+```
+PersistentCompactIntMatrix::col_weights()              — one (partition, layer)
+        ↓ Σ across layers
+LayeredStore<PersistentCompactIntMatrix>::col_weights() — one partition
+        ↓ Σ across partitions
+LayeredStore<LayeredStore<…>>::col_weights()            — global
+```
+
+The same cascade applies to every partial method.
+
+---
+
+## Multi-genome column invariant
+
+After any merge, every layer in every partition has exactly `n_genomes` columns, where `n_genomes` is the current total in `index.meta`. This holds for both `PersistentCompactIntMatrix` and `PersistentBitMatrix`.
+
+Maintained by three coordinated operations:
+
+**Existing layers — column append.** `Layer::append_genome_column` appends one column to each existing layer. Slots matching the incoming genome receive its count or `true`; all other slots receive 0 or `false`.
+
+**New layers — absent columns prepended.** When a new layer is created for kmers unique to the incoming genome, `n_existing_genomes` absent columns are prepended before the incoming genome's column, so the new layer immediately has the same column count as all other layers.
+
+**First merge, Presence mode — `init_presence_matrix`.** The initial single-genome index has no `presence/` directory (presence is implicit). On the first merge, `Layer<()>::init_presence_matrix` materialises genome 0's presence column (all `true`) retroactively, raising the column count from 0 to 1 before appending column 1.
+
+This invariant is the precondition for correct progressive aggregation: every level can blindly sum matrices from below because all matrices have the same shape.
+
+---
+
+## Query model
+
+### Point query
+
+```
+minimiser(kmer) → partition p
+for each layer l in p:
+    if let Some(slot) = MphfLayer_l.find(kmer):
+        return data_l.read(slot)
+return None
+```
+
+O(n_layers) MPHF probes worst case; O(1) expected. The result comes from exactly one `(partition, layer)`.
+
+### Aggregation
+
+```
+result = reduce(
+    for p in partitions:       // parallel
+        for l in layers(p):    // parallel
+            partial(data_p_l)
+)
+```
+
+For normalised metrics, replace with the two-pass cascade.
+
+---
+
+## Parallelism model
+
+| Level | Unit | Coordination |
+|---|---|---|
+| Across partitions | inner stores of `LayeredStore<LayeredStore<S>>` | none |
+| Across layers within a partition | inner stores of `LayeredStore<S>` | none — disjoint kmer sets |
+| Normalised pass 1 (`col_weights`) | per inner store | none — additive |
+| Normalised pass 2 (partial) | per inner store | `global` broadcast read-only |
+| Within a matrix (distance) | upper-triangle pair `(i,j)` | none — rayon `par_iter` |
+
+---
+
+## reindex — evidence conversion in place
+
+`KmerIndex::reindex(target, block_bits)` converts every layer's evidence bundle to `target` without touching the MPHF or `unitigs.bin`:
+
+- `→ Exact`: builds `evidence.bin` + `unitigs.bin.idx`; removes `fingerprint.bin`
+- `→ Approx { b, z }`: builds `fingerprint.bin`; removes `evidence.bin` + `unitigs.bin.idx`
+
+On success, `IndexConfig::evidence` and `IndexConfig::block_bits` are updated in `index.meta`. Each layer's `layer_meta.json` is also rewritten with the new `EvidenceKind`.
+
+---
+
+## estimate — parameter dry-run
+
+`estimate` resolves approximate-evidence parameters (`z`, `b`, target FP rate) and prints the resulting effective kmer size and per-kmer / per-z-window false-positive rates without touching any index. Used to calibrate `Approx { b, z }` before building or reindexing.

DevDocMD/architecture/index_architecture.refs.md

@@ -0,0 +1,22 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: architecture/index_architecture.md
+
+## Code couvert
+
+- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix
+- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, EvidenceKind (Exact / Approx), LayerEvidence enum
+- `obilayeredmap/src/map.rs` — LayeredMap<D>
+- `obilayeredmap/src/meta.rs` — LayerMeta, PartitionMeta
+- `obikindex/src/meta.rs` — IndexConfig (kmer_size, n_bits, with_counts, evidence, block_bits), IndexMeta
+- `obikindex/src/index.rs` — KmerIndex, build_layers
+- `obicompactvec/src/` — PersistentCompactIntMatrix, PersistentBitMatrix (DataStore implementations)
+
+## Notes
+
+FORT RISQUE DE DÉRIVE. Nombreux changements récents :
+- Ajout de `EvidenceKind` (Exact / Approx { b, z }) dans `IndexConfig` et `LayerMeta`
+- Ajout de `block_bits` dans `IndexConfig`
+- `LayerEvidence` enum dans `mphf_layer.rs` remplace l'ancienne approche monolithique
+- Distinction `open()` vs `open_sequential()` dans `UnitigFileReader`
+- Commandes `reindex` et `estimate` ajoutées
+Vérifier que la hiérarchie à 3 niveaux décrite est toujours exacte et que les nouveaux paramètres sont documentés.

DevDocMD/architecture/numa_partition_runner.md

@@ -0,0 +1,323 @@
+# NUMA-aware partition runner
+
+## Problem
+
+All partition-level parallel loops in obikindex currently fall into two
+categories:
+
+**Naive Rayon** — used in `build_layers`, `pack_matrices`, `dump`, `select`,
+`stats`, `rebuild`, `reindex`:
+
+```rust
+(0..n).into_par_iter().for_each(|i| work(i));
+```
+
+Threads come from the global Rayon pool with no NUMA awareness.  On
+multi-socket machines this produces cross-socket memory traffic and degrades
+performance super-linearly (see [NUMA-aware worker pools](numa_worker_pools.md)).
+
+**Ad-hoc adaptive pool** — used in `merge`:
+
+A bespoke implementation with pre-spawned workers, channel-based dispatch, and
+activation control.  It handles NUMA correctly but is not reusable.
+
+Both cases should be replaced by a single generic mechanism.
+
+## Unified model
+
+The key insight is that **UMA is just the NUMA case with a single node**.  The
+runner always works the same way: one controller thread per node, each
+independently managing its own workers with the same adaptive logic.  The only
+difference between UMA and NUMA is the number of nodes and whether workers are
+pinned.
+
+```
+NUMA (k nodes)                    UMA (1 node)
+
+controller-0  controller-1  …     controller-0
+    │               │                  │
+workers[0]     workers[1]         workers[0]
+(pinned)       (pinned)           (global pool)
+    └───────────────┴──────────────────┘
+              shared work queue
+```
+
+On each node, the Rayon `ThreadPool` is pinned to that node's CPUs.
+`pool.install()` ensures all internal Rayon calls (inside the work function)
+use the node-local pool.  Linux first-touch then places heap allocations in
+local DRAM automatically.
+
+On UMA the global Rayon pool is used directly — no pinning, no overhead.
+
+## Adaptive mechanism
+
+Each controller follows the same logic regardless of node count:
+
+1. Pre-spawn `workers_per_node` dormant worker threads (blocked on `activate_rx`).
+2. Activate the first worker immediately.
+3. Loop on result channel with a `SPAWN_POLL` timeout:
+   - On result: call `on_done`; check whether to activate the next worker.
+   - On timeout: same check.
+   - Activation criterion: `should_spawn_worker(active, global_efficiency, prev_efficiency)`.
+4. Drop `activate_tx` when done — dormant workers exit cleanly.
+
+**Global CPU efficiency** (`CpuSample`, reads `/proc/stat` on Linux) is used by
+all controllers — no per-node measurement needed.  The signal is coarser than
+per-node efficiency but correct in practice: if any node saturates memory
+bandwidth, the global efficiency drops and all controllers stop activating
+workers.  Using a standard portable primitive avoids platform-specific CPU
+accounting and keeps the implementation clean.
+
+## Proposed API
+
+```rust
+pub struct PartitionRunner {
+    // One entry per NUMA node; one entry total on UMA.
+    nodes: Vec<NodeConfig>,
+}
+
+struct NodeConfig {
+    pool:       Option<Arc<rayon::ThreadPool>>,  // None = global Rayon pool (UMA)
+    cpu_ids:    Vec<usize>,                      // empty = no pinning (UMA)
+    max_workers: usize,
+}
+
+impl PartitionRunner {
+    /// Detect topology and build the runner.
+    /// Returns a single-node runner on UMA / macOS / hwloc failure.
+    pub fn new() -> Self;
+
+    /// Run `f(i)` for every index in `order`, collecting results.
+    ///
+    /// `on_done(i, result, elapsed)` is called under an internal mutex as
+    /// each partition completes — use it for progress bars and aggregation.
+    /// The runner serialises all calls to `on_done` via an internal
+    /// `Arc<Mutex<C>>`, so no `Sync` bound is required on the callback.
+    /// `Send` is required because the Arc clone crosses thread boundaries.
+    ///
+    /// Serialisation is free in practice: a partition takes seconds to
+    /// minutes; the callback takes microseconds.  Contention is negligible.
+    ///
+    /// Returns the first error from `f`, if any.
+    pub fn run<F, R, E, C>(
+        &self,
+        order:   &[usize],
+        f:       F,
+        on_done: C,
+    ) -> Result<(), E>
+    where
+        F: Fn(usize) -> Result<R, E> + Send + Sync,
+        R: Send,
+        E: Send,
+        C: FnMut(usize, R, Duration) + Send;   // Send required, Sync is not
+}
+```
+
+`order` is caller-supplied so each command chooses its scheduling strategy:
+largest-first for `merge`, sequential for `build_layers`, etc.
+
+## Migration examples
+
+### merge.rs (before: ~180 lines of bespoke machinery)
+
+```rust
+let runner = PartitionRunner::new();
+runner.run(
+    &order,
+    |i| dst_partition.merge_partition(i, srcs, mode, n_dst_genomes, block_bits, evidence)
+            .map_err(OKIError::Partition),
+    |i, g_len, dur| {
+        pb.inc(1);
+        debug!("partition {i}: done in {:.1}s — {g_len} new kmers", dur.as_secs_f64());
+        part_stats.push(PartStat { id: i, unitig_bytes: partition_sizes[i], g_len });
+    },
+)?;
+```
+
+### index.rs build_layers (before: naive into_par_iter)
+
+```rust
+let order: Vec<usize> = (0..n).collect();
+let runner = PartitionRunner::new();
+runner.run(
+    &order,
+    |i| self.partition.build_index_layer(i, min_ab, max_ab, with_counts, &evidence, block_bits)
+            .map_err(OKIError::Partition),
+    |_, n_kmers, _| {
+        total_kmers.fetch_add(n_kmers, Ordering::Relaxed);
+        pb.inc(1);
+    },
+)?;
+```
+
+All other sites (`pack_matrices`, `dump`, `select`, etc.) follow the same
+pattern.
+
+## Placement
+
+`PartitionRunner` lives in `obikindex/src/numa.rs` alongside `NumaSetup`.
+It depends only on standard library primitives and Rayon — no new dependencies.
+
+A single `PartitionRunner` instance can be built once per command invocation
+and reused across multiple `run()` calls (e.g. `merge` runs
+`merge_partitions` then `pack_matrices`).
+
+## Known issue: CPU-only activation signal stalls on I/O-bound stages
+
+Observed on a real `filter` run (109 genomes, 256 partitions, 8×24-core NUMA):
+`rebuild` (CPU-bound — k-mer construction) scales cleanly from 9 to 43 active
+workers as `CpuSample::do_i_activate` (`obisys::lib.rs`) sees efficiency climb.
+`pack_matrices` (I/O-bound — reopens and recomposes per-genome column files
+into `.pbmx`/`.pcmx`) activates one extra worker then flatlines at 10/192 for
+the rest of the stage, even though 256 partitions keep completing over several
+minutes. This matches the documented intent (§ Adaptive mechanism — "avoids
+over-provisioning ... I/O-bound ... workloads") but conflates two different
+things: *"CPU is not the bottleneck"* and *"more workers would not help"*. On
+storage with real queue depth (NVMe, RAID, parallel FS) the second stage could
+still benefit from more concurrent workers even with flat CPU usage — a signal
+the current mechanism cannot see.
+
+A one-off artefact was also found in the same log: right after a stage
+transition, `do_i_activate` produced a physically impossible spike (efficiency
+~94 cores on a 192-core box) because it has no minimum-window guard — unlike
+its sibling `cpu_efficiency`, which returns `0.0` if `wall < 0.1s`
+(`obisys::lib.rs:260`). `do_i_activate` unconditionally overwrites
+`self.wall`/`self.user_secs`/`self.sys_secs` even when the elapsed window is
+too short to be meaningful, so a burst of rapid completions right after
+activating a worker can divide a real CPU delta by a near-zero wall delta.
+
+### Implemented: I/O signal + shared debounce guard
+
+`IoSample` (`obisys::lib.rs`, alongside `CpuSample`) is fed by
+`read_bytes`/`write_bytes` from `/proc/self/io` on Linux (actual bytes
+submitted to the block layer — not `rchar`/`wchar`, which also count
+page-cache hits, and not `ru_inblock`/`ru_oublock`, unreliable on macOS), with
+a `proc_pid_rusage(RUSAGE_INFO_V4)` fallback on macOS
+(`ri_diskio_bytesread`/`ri_diskio_byteswritten`, FFI only via `libc`, no new
+dependency — same pattern as the existing `getrusage` bindings). Any other
+target degrades gracefully to a signal that never triggers (falls back to
+CPU-only activation), same pattern as `cgroup_v2_available`.
+
+`maybe_activate` (`numa.rs`) activates a worker if *either* signal still shows
+headroom, making `PartitionRunner` adapt to whichever resource is actually the
+bottleneck without per-call configuration. Both samplers are called
+unconditionally — no `||` short-circuit — so neither window starves behind
+whichever signal fires first:
+
+```rust
+let cpu_threshold = CPU_SPAWN_THRESHOLD * activation.last_step() as f64;
+let cpu_wants_more = cpu_sample.do_i_activate(cpu_threshold);
+let io_wants_more  = io_sample.do_i_activate(IO_SPAWN_THRESHOLD);
+if cpu_wants_more || io_wants_more {
+    activation.grow(GROWTH_DIVISOR, n_total);
+}
+```
+
+The CPU threshold is *not* the flat absolute delta it started as: it scales
+with `activation.last_step()` — the number of workers activated in the last
+growth step, tracked by `NodeActivation` (`numa.rs`) and updated every time
+`grow()` actually grows something. Growing by 8 workers should add ~8 cores of
+efficiency if the workload is truly CPU-bound; requiring only
+`CPU_SPAWN_THRESHOLD` (20 %) of that expected gain confirms the growth was
+useful without demanding perfect linear scaling. Scaling by the *last step's
+size* rather than the cumulative total keeps the bar equally meaningful
+whether it's the 2nd growth step or the 20th — a flat absolute threshold
+(0.2 core) is a strong signal at 8 active workers but pure noise at 150; a
+threshold scaled by the *cumulative* total instead (considered and rejected)
+would have made the bar essentially impossible to clear late in the ramp,
+strangling exactly the CPU-bound saturation the mechanism exists to allow.
+
+Unlike the CPU signal (an absolute delta in cores — a bounded, portable unit),
+raw I/O throughput has no natural scale across devices, so `IoSample` uses a
+**relative** growth threshold instead of an absolute one:
+
+```rust
+pub fn do_i_activate(&mut self, threshold: f64) -> bool {
+    let elapsed = self.wall.elapsed().as_secs_f64();
+    if elapsed < 0.1 { return false; }        // state untouched — window keeps accumulating
+
+    let n = Self::read_bytes();
+    let rate = n.saturating_sub(self.bytes) as f64 / elapsed;
+    let activate = if self.previous_rate == 0.0 {
+        rate > 0.0                            // bootstrap: any measured throughput is signal
+    } else {
+        (rate - self.previous_rate) / self.previous_rate >= threshold
+    };
+
+    self.bytes = n;
+    self.wall  = Instant::now();              // reset only on a real sample
+    activate
+}
+```
+
+The `elapsed < 0.1s → return false without mutating state` guard was also
+back-ported into `CpuSample::do_i_activate` (previously missing — source of
+the ~94-core artefact above) — one fix for both problems, and it removes the
+need for any arbitrary I/O-rate floor: a short/noisy window is rejected
+outright rather than papered over with a hardware-dependent constant.
+
+Both spawn thresholds (`CPU_SPAWN_THRESHOLD`, `IO_SPAWN_THRESHOLD`, module-level
+`const` in `numa.rs`, both `0.2`) are a starting point, not a derived value:
+`0.2` (20 % relative growth) for `IoSample` was chosen to match the CPU
+threshold's *implicit* relative sensitivity (in the observed log, an 8→9
+worker step raised efficiency by ~12 %) — but I/O throughput is lumpier than
+CPU time (buffered writes flush in bursts), so it needs empirical validation
+against a real `pack` run before being considered final.
+
+## Known issue: ramp-up too slow, and confused with node count
+
+The original design started `n_nodes` workers (one per node) and grew one
+worker at a time. On a real `filter` run this took ~10 minutes to climb from
+9 to ~40 active workers even on the CPU-bound `rebuild` stage — most of a
+35-minute stage spent under-provisioned while waiting for evidence to
+accumulate one worker at a time. There is no scale-down mechanism (`n_active`
+only grows), so the original caution was deliberate — but a quarter of
+available cores is still far from saturation, and the real risk zone (over-provisioning
+a memory-bandwidth-bound stage) only shows up much later in the ramp, near
+full occupancy — not at 25 %.
+
+The fix decouples ramp speed from node *count*: both the initial size and the
+growth step are a fraction of `workers_per_node` (node *size*), applied
+identically on every node. A single-NUMA-node (UMA) machine ramps exactly as
+fast as an 8-node one — growing by `n_nodes` per step, as first considered,
+would have degenerated to "grow by 1" on UMA, reproducing the original
+problem for exactly the machines that need the fix most.
+
+```rust
+// NodeActivation::grow — called both at startup (activate_initial) and on
+// every CPU/IO-triggered growth step, with a different divisor each time.
+let wanted = (self.caps[idx] / divisor).max(1);   // INITIAL_DIVISOR=4 at startup, GROWTH_DIVISOR=8 per step
+let room   = self.caps[idx].saturating_sub(self.active[idx]);
+let grow   = wanted.min(room).min(n_total.saturating_sub(self.total));
+```
+
+This also fixed a latent correctness gap: the original single shared
+`activate_tx`/`activate_rx` pair had *no* per-node addressing — sending one
+activation signal woke up whichever dormant worker (from any node) happened
+to win the race on that channel. `crossbeam_channel` gives no fairness
+guarantee across competing receivers, so "round-robin across nodes" was an
+assumption the code never actually enforced. `PartitionRunner::run` now opens
+one activation channel per node (`activate_txs`/`activate_rxs`, one pair per
+`NodeConfig`); `NodeActivation` (`numa.rs`) tracks how many of each node's
+dormant workers have been woken and grows every node by the same amount per
+step, capped by that node's remaining dormant workers and by the run's total
+budget (`n_total`) — balance across nodes is now guaranteed by construction,
+not incidental to channel implementation details.
+
+## Open questions
+
+- **Error handling**: `run` currently returns the first error; remaining errors
+  are dropped.  A `Vec<E>` return would give complete diagnostics.
+
+- **`INITIAL_DIVISOR` / `GROWTH_DIVISOR` tuning**: currently `4` and `8`
+  (start at 1/4 of a node's cores, grow by 1/8 per step), chosen to fix an
+  observed too-slow ramp — not yet validated against a real `pack` (I/O-bound)
+  run, where over-provisioning risk is different from the CPU-bound `rebuild`
+  case this was tuned against.
+
+- **`on_done` ordering**: the runner serialises calls to `on_done` via an
+  internal `Arc<Mutex<C>>`.  `Send` is required (the Arc clone crosses thread
+  boundaries); `Sync` is not (only one thread holds the lock at a time).
+  Contention is negligible because a partition takes seconds while the callback
+  takes microseconds.  The callback is therefore simple to write (plain
+  `Vec::push`, plain `FnMut`) with no measurable performance cost.

DevDocMD/architecture/numa_worker_pools.md

@@ -0,0 +1,97 @@
+# NUMA-aware worker pools for merge
+
+## Problem
+
+The merge command's bottleneck is `compute_degrees` in `obidebruinj`: a random pointer-chase over 20–70 M node hash maps that saturates DRAM bandwidth. When multiple partition workers run concurrently, they contend for the shared memory bus, causing super-linear slowdown (measured: 0.016 µs/node solo → 0.95 µs/node with 4–5 concurrent workers, ×60 degradation).
+
+Modern HPC nodes are multi-socket NUMA machines (observed: 2 sockets × 4 NUMA nodes × 24 cores = 192 cores). Cross-NUMA memory traffic compounds the contention:
+
+- Full 192-core run: ~15 min/partition (×10 worse than M3 Mac)
+- `taskset` restricted to 4 NUMA nodes (96 cores): ~90 s/partition
+- OAR job on 1 NUMA node (24 cores): ~80 s/partition, same throughput as 96 cores
+
+**Conclusion**: the bottleneck is memory bandwidth per NUMA node, not core count. 24 cores on one NUMA node achieve the same throughput as 96 cores across four.
+
+## Strategy
+
+Run N worker groups in parallel, one per NUMA node, each with its own Rayon thread pool whose threads are pinned to the NUMA node's CPUs. Linux's first-touch policy then places graph allocations on local DRAM automatically — no explicit NUMA allocator needed.
+
+Expected throughput: N × single-NUMA throughput. On the 8-NUMA-node HPC: 8 × ~80 s = 9–10 min total instead of >60 min with the current single-pool approach.
+
+## Rayon thread pool isolation
+
+Rayon provides `ThreadPool::install(|| { ... })`: any Rayon call (`par_iter`, `current_num_threads`, etc.) inside the closure uses *that* pool exclusively. Wrapping `merge_partition` in `pool.install()` redirects all downstream Rayon calls — including those in `debruijn.rs` and `partition.rs` — without touching those crates.
+
+```rust
+// worker thread, assigned to NUMA pool `pool`
+pool.install(|| {
+    dst_partition.merge_partition(i, srcs, mode, n_dst_genomes, block_bits, evidence)
+})
+```
+
+`rayon::current_num_threads()` inside `merge_partition` will return the pool size (e.g. 24), not the global thread count — which is the right value for buffer sizing.
+
+## Thread pinning
+
+`ThreadPoolBuilder::spawn_handler` provides a hook executed for each thread at creation. Inside, `libc::sched_setaffinity` pins the thread to a CPU set:
+
+```rust
+let cpus: Vec<usize> = numa_node_cpus(node); // from /sys/devices/system/node/nodeN/cpulist
+rayon::ThreadPoolBuilder::new()
+    .num_threads(cpus.len())
+    .spawn_handler(move |thread| {
+        let mut b = std::thread::Builder::new();
+        std::thread::Builder::new().spawn(move || {
+            pin_to_cpus(&cpus);   // sched_setaffinity via libc
+            thread.run()
+        })?;
+        Ok(())
+    })
+    .build()?
+```
+
+NUMA topology is read from `/sys/devices/system/node/node*/cpulist` — no `libnuma` dependency required. If the `numa` crate is linked, `numa_available()` / `numa_run_on_node()` are an alternative.
+
+## Memory locality
+
+Linux allocates pages on the NUMA node of the thread that first writes them (first-touch policy). Once Rayon threads are pinned to node N, all graph data built by those threads lands on node N's DRAM. No changes to the allocator, no explicit `numa_alloc_onnode` calls.
+
+## Adaptive spawn criterion
+
+The current criterion uses `std::thread::available_parallelism()` (returns total cores = 192) and `max_workers = n_cores / 2`. With NUMA pools:
+
+- `n_cores` per pool = cores per NUMA node (e.g. 24)
+- `max_workers` per pool = pool size / 2 (e.g. 12)
+- CPU efficiency is measured per pool, not globally
+
+Each NUMA group runs its own independent adaptive pool. Workers are distributed across NUMA groups round-robin or by workload (partition assignment can be pre-split by NUMA group index).
+
+## Required changes
+
+| File | Change |
+|------|--------|
+| `obikindex/src/merge.rs` | Detect NUMA topology; build N `ThreadPool`s with pinned threads; assign each pre-spawned worker to a pool; wrap `merge_partition` in `pool.install()` |
+| `obikindex/src/merge.rs` | Replace `available_parallelism()` with per-NUMA core count for spawn criterion |
+| `obikpartitionner/src/merge_layer.rs` | No change — `merge_partition` already works inside any Rayon context |
+| `obidebruinj/src/debruijn.rs` | No change — `par_iter` and `current_num_threads` are pool-context-aware |
+| `obikpartitionner/src/partition.rs` | No change — same reason |
+
+## Platform guard
+
+NUMA pinning is Linux-only. The fallback is the current single global pool:
+
+```rust
+#[cfg(target_os = "linux")]
+fn build_numa_pools() -> Option<Vec<rayon::ThreadPool>> { ... }
+
+#[cfg(not(target_os = "linux"))]
+fn build_numa_pools() -> Option<Vec<rayon::ThreadPool>> { None }
+```
+
+When `build_numa_pools()` returns `None` (macOS, UMA, or single-socket), `merge.rs` uses the existing code path unchanged.
+
+## Open questions
+
+- **Partition assignment**: split partitions by NUMA group up-front (static) or use a shared queue with per-group workers stealing from a common pool? Static split is simpler; stealing is better for load balance when partitions vary widely in size.
+- **Intra-NUMA adaptive criterion**: with 24 cores and ~3–5 effective workers per NUMA node, the current marginal-gain criterion needs re-tuning or can be left as-is with per-pool `n_cores = 24`.
+- **I/O**: partition data (unitig files) is on a shared filesystem. With 8 concurrent NUMA groups, I/O concurrency increases 8× — need to verify the filesystem (Lustre or local SSD) can absorb it without becoming the new bottleneck.

DevDocMD/architecture/query.md

@@ -0,0 +1,406 @@
+# Query system
+
+## Goal
+
+Given a set of query sequences, determine for each sequence how many of its k-mers are found in the index and, for each indexed genome, how many k-mers match. The query system is the foundation for read classification and sequence-to-genome mapping.
+
+---
+
+## Input
+
+- Query sequences in FASTA or FASTQ format (gzip supported, streaming stdin supported). GenBank flat files are not supported at query time (only at index time).
+- Sequences shorter than k bases are silently skipped.
+- Non-ACGT characters are handled by the superkmer decomposition layer: they act as hard breaks, producing shorter superkmers (identical to the behaviour at indexing time).
+
+---
+
+## Algorithm
+
+The query follows the same superkmer-based partitioning strategy used at indexing time. Everything below happens inside `process_chunk` (`query.rs`); there is no separate per-stage function, but the internal data flow is staged: k-mer-level dereplication, a two-part MPHF/column-major matrix lookup (`obikpartitionner::query_partition_with`), and a sparse Findere pass, each producing sparse intermediate structures rather than one dense allocation for the whole chunk.
+
+```
+for each chunk of sequences (parallel workers via obipipeline, one call to process_chunk):
+    build QueryBatch (QueryBatch::from_records):
+        decompose all sequences into superkmers (SuperKmerIter) — construction only,
+        not the dedup key
+        deduplicate at k-mer granularity, split by partition in the same pass:
+            by_partition: Vec<HashMap<CanonicalKmer, Vec<KmerDesc>>>   ← KmerDesc = (seq_idx, pos)
+    allocate SmerIndex (SmerIndex::new): in_index: Vec<bool>, sized total_smers —
+        NOT multiplied by n_genomes
+    allocate by_genome: Vec<Vec<(seq_idx, pos, value)>>, one empty Vec per genome —
+        stays empty (zero cost) for every genome this chunk never matches
+    for each partition p:
+        query_partition_with(p, kmers_for_p, on_event):
+            stage 1 (MPHF-only): for each unique k-mer, try each layer's MphfLayer::find
+                in turn, stop at the first hit; bucket confirmed hits by (layer, slot);
+                emit QueryHit::Found(descs) once per hit k-mer
+            stage 2 (column-major fetch): for each layer with ≥1 hit, for each genome
+                column g in 0..layer.n_cols(): scan that layer's bucketed slots, look up
+                col_value(g, slot); emit QueryHit::Value(descs, g, value) on nonzero
+        on_event dispatches: Found → SmerIndex::mark_found for every desc;
+                              Value → push (seq_idx, pos, value) into by_genome[g]
+    for each genome g with ≥1 hit (sparse_findere_for_genome):
+        sort by_genome[g] by (seq_idx, pos); detect maximal runs of consecutive pos
+        within one seq_idx; monotone-deque window-minimum scoped to each run →
+        confirmed_by_genome[g]: Vec<(seq_idx, pos_out, value)>
+    accumulate genome_totals per sequence from confirmed_by_genome (per genome, direct)
+    accumulate kmer_count / kmer_missing per (sequence, output position), O(1) each,
+        using only the confirmed-any bitmap and SmerIndex — independent of n_genomes
+    if --detail: densify confirmed_by_genome into per-(seq, genome) coverage arrays
+    emit annotated sequences (emit_batch)
+```
+
+Superkmers that appear more than once in the batch (same sequence or across sequences), or different superkmers that happen to share a k-mer (read overlaps, repeats, a SNP splitting an otherwise-identical run), are deduplicated at k-mer granularity: each unique `CanonicalKmer` triggers at most one MPHF lookup and, on hit, one matrix fetch, broadcast to every `KmerDesc` occurrence referencing it.
+
+**Findere requires full-sequence aggregation.** The sliding window (now per-run, not per-sequence — see [Findere z-window filter](#findere-z-window-filter)) only ever runs after all partitions have contributed their hits to `by_genome`. Applying it per superkmer would produce false negatives at superkmer boundaries, where the z-window spans two superkmers.
+
+Batches are processed in parallel via `obipipeline` workers; the `--threads` flag controls the number of worker threads.
+
+---
+
+## Findere z-window filter
+
+For approximate index modes, the index physically stores s-mers of size `s = k_user − z + 1`; `idx.kmer_size()` (bound to `k` in `process_chunk`) is this physically-indexed s-mer size, so decomposing the query at `k` naturally produces s-mer results.
+
+The z-window aggregation is **sparse**, per genome, implemented in `sparse_findere_for_genome` (`query.rs`) — a run-detection pass followed by a monotone-deque sliding-window minimum scoped to each run, not a dense scan over every s-mer position of every sequence:
+
+```
+sparse_findere_for_genome(hits, z, presence, threshold):
+    // hits: raw (seq_idx, pos_smer, value) triples for this genome, as delivered
+    // by query_partition_with's QueryHit::Value — only ever nonzero entries;
+    // a position with no hit for this genome simply has no entry at all.
+    sort hits by (seq_idx, pos_smer)
+
+    for each maximal run of consecutive pos_smer values within the same seq_idx:
+        dq: VecDeque<(run-relative index, value)>
+        for k, (_, pos, value) in enumerate(run):
+            maintain dq monotone non-decreasing (pop back while back.value >= value)
+            push (k, value)
+            evict dq entries with run-relative index <= k - z
+            if k + 1 >= z:
+                win_min = dq.front().value
+                if win_min > 0:
+                    pos_out = pos + 1 - z
+                    confirmed.push((seq_idx, pos_out, adjust(win_min)))
+    return confirmed
+```
+
+A window can only be confirmed (`win_min > 0`) when all `z` s-mers in it are present *and* nonzero for this genome — which, by construction, can only happen strictly inside one contiguous run of hits (any gap — an absent or zero-valued s-mer — forces `win_min = 0` for every window spanning it, exactly matching the old dense scan's "not in index counts as 0" rule, just never materialising the zero). The deque logic is otherwise identical to the pre-sparsification version; it's scoped to run-relative indices instead of the whole sequence.
+
+This runs once per genome that has at least one hit in the chunk (`process_chunk` iterates `by_genome`, one `Vec<(seq_idx, pos_smer, value)>` per genome, built from `QueryHit::Value` during the partition loop — genomes with zero hits in this chunk have an empty `Vec` and cost nothing beyond the iteration itself). Total work is `O(hits log hits)` per genome (the sort) rather than `O(n_smers)` per genome regardless of hit count — a genuine complexity win on top of the memory one, for the common case where most `(chunk, genome)` pairs have no or few hits.
+
+Output position `pos_out` is confirmed for genome `g` iff its run produced a nonzero `win_min` — equivalent to "all `z` consecutive s-mer values in the window are nonzero for `g`", same semantics as before.
+
+**The value reported per confirmed position is the window minimum, not the leftmost s-mer's raw value** — unchanged from the dense version. For presence indexes (0/1 values) this is equivalent to a logical AND either way. For count indexes it is not: the accumulated count for genome `g` at position `pos_out` is the minimum across the window, the weakest link — not the leftmost s-mer's own count. The presence/count adjustment (`u32::from(win_min >= threshold)` vs. raw `win_min`) is applied once, inside `sparse_findere_for_genome`, rather than later during accumulation.
+
+**`kmer_missing` bookkeeping is independent of the per-genome sparse structures**, by design (see roadmap point 9): a lightweight dense `SmerIndex` (`in_index: Vec<bool>`, sized `total_smers` — **not** multiplied by `n_genomes`) is populated from `QueryHit::Found` during the partition loop, one entry per hit k-mer regardless of which genome(s) it matched. A position with no genome confirmed counts as `kmer_missing` iff the leftmost s-mer of that window is absent from `SmerIndex` entirely (see [`kmer_missing` semantics](#kmer_missing-semantics)).
+
+**Coverage (`--detail`)** is built by re-scanning each genome's confirmed-hit list (already computed, no extra pass over raw data) and densifying into the `[u32; n_kmers_out]` arrays the JSON output format requires — but only when `--detail` is actually requested; the sparse structures cost nothing extra when it isn't.
+
+**Short sequences**: when a sequence's s-mer count is less than `z`, its run(s) — if any hits exist at all — can never reach length `z`, so no window is ever confirmed for it; no k_user-mer is emitted, same outcome as the dense version's `n_kmers_out == 0` early-skip, reached here as a natural consequence rather than a separate check.
+
+**Exact indexes**: `z = 1`, every single-hit "run" of length 1 immediately satisfies `k + 1 >= z`, so every hit is its own confirmed window with `win_min` equal to its own value — a passthrough, as before.
+
+### Effective z at query time
+
+`effective_z` is resolved at the start of `run()`:
+
+```rust
+let effective_z = args.findere_z.unwrap_or_else(|| match idx.meta().config.evidence {
+    IndexMode::Approx { z, .. } | IndexMode::Hybrid { z, .. } => z as usize,
+    IndexMode::Exact => 1,
+});
+```
+
+The `-z` CLI option overrides the index metadata value. A higher z increases stringency (lower FP, some true positives may be discarded at sequence ends); a lower z increases sensitivity.
+
+---
+
+## Layer lookup: `MphfLayer::find`
+
+`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read. The caller (`QueryLayer`) never chooses the dispatch path: it is fixed at open time by `LayerEvidence`. See [obilayeredmap](../implementation/obilayeredmap.md) for the full `find` / `find_strict` API.
+
+### `QueryLayer` variant selection
+
+`QueryLayer::open` (`obikpartitionner/src/query_layer.rs:28-45`) only ever returns two variants — `Presence` or `Count`, checked in this order:
+
+| Order | Condition | Variant | Data returned per k-mer |
+|---|---|---|---|
+| 1 | `with_counts=true` and `counts/` exists | `Count` | raw count per genome |
+| 2 | (else) `presence/` exists, or `counts/` doesn't exist at all | `Presence` | see below |
+| 3 | (else — `counts/` exists, `presence/` doesn't, `with_counts=false`) | `Count` | counts used as-is |
+
+There is no `QueryLayer::SetOnly` variant. The "no on-disk matrix at all" case is handled one level down: `Presence` wraps `PersistentBitMatrix`, whose own `open()` (`obicompactvec/src/bitmatrix.rs:260-288`) auto-detects among **three** internal representations — `Packed` (`presence/matrix.pbmx`), `Columnar` (`presence/meta.json`), or `Implicit { n_rows, n_cols }` when neither file exists (built from `layer_meta.json`, `fill_row` returning all-`1`s without touching disk). This is where "1 for every genome" actually happens — not at the `QueryLayer` level.
+
+**Worth double-checking, not confirmed as a bug**: `PersistentBitMatrix::open`'s `Implicit` branch constructs `Implicit { n_rows: meta.n, n_cols: 1 }` — `n_cols` is hardcoded to `1`, not to the layer's actual `n_genomes`. `fill_row` for `Implicit` only writes `buf[..1]`, leaving the rest of a longer `n_genomes`-sized buffer untouched (zeroed by the caller beforehand). If this path is ever reached for a layer covering more than one genome, only genome index 0 would read as present. Whether that's reachable in practice (layers might always be single-genome when they fall back to `Implicit`) wasn't verified here — flagging for follow-up, not fixing.
+
+---
+
+## Presence / count mode at query time
+
+The `--force-presence` flag and `--presence-threshold` control how per-genome values are accumulated, independently of what the index stores:
+
+```
+genome_totals[g] += if presence { u32::from(v >= threshold) } else { v }
+```
+
+`presence` is true when `--force-presence` is set or when the index has no counts (`!with_counts`). The default `presence_threshold` is 1, so any nonzero count counts as a match.
+
+---
+
+## Coverage vectors (`--detail`)
+
+When `--detail` is requested, a 3-D accumulator `cov[seq_idx][genome][kmer_pos]` is allocated after all partitions are queried, with dimensions derived from `n_kmers_out = n_smers − z + 1` (k_user-mer positions, not s-mer positions):
+
+```
+cov[seq_idx][g][pos] += contribution
+where pos is the k_user-mer index in the filtered (post-Findere) vector
+```
+
+Coverage reflects confirmed k_user-mers only. The vectors are emitted in the JSON annotation under the key `"coverage"`.
+
+---
+
+## `kmer_missing` semantics
+
+`kmer_missing` counts k_user-mer positions where the leftmost s-mer of the window (`smer_index.is_in_index(seq_idx, pos)`, `SmerIndex`) is `false` — i.e. absent from the index entirely. K-mers where the z-window fails because a later s-mer is absent or zero (but the leftmost one is present) are not counted as missing — the leftmost s-mer being present is used as proxy for index membership.
+
+---
+
+## Output format
+
+Output sequences are written in **OBITools4 format**: the original sequence with a JSON annotation map in the title line.
+
+```
+>read_id {"kmer_count":59,"kmer_strict_matches":{"genome_a":42,"genome_b":7}}
+ATCGATCG...
+```
+
+With `--detail`:
+
+```
+>read_id {"kmer_count":59,"kmer_strict_matches":{...},"coverage":{"genome_a":[0,1,2,...],...}}
+ATCGATCG...
+```
+
+Genome keys follow the iteration order of `meta.genomes`.
+
+---
+
+## Annotation schema
+
+| Key | Type | Condition | Semantics |
+|---|---|---|---|
+| `kmer_count` | int | always | k-mers confirmed (post-Findere) with at least one genome match |
+| `kmer_missing` | int | `--count-missing` | k-mers absent from the index entirely (leftmost s-mer of the window not found) |
+| `kmer_strict_matches` | object | always | per-genome accumulated value, non-zero entries only (label → count or 0/1) |
+| `coverage` | object | `--detail` | per-genome array of per-position contributions (label → [u32]) |
+
+`kmer_count + kmer_missing` ≤ total k_user-mers in the sequence. The gap corresponds to k_user-mers whose z-window was not fully confirmed (at least one s-mer absent or zero for all genomes) but whose first s-mer was present in the index.
+
+---
+
+## CLI
+
+```
+obikmer query <index> [--detail] [--mismatch] [--count-missing]
+              [--force-presence] [--presence-threshold <n>]
+              [-z <z>] [-T <threads>] [--chunk-size <MiB>]
+              <query.fa> [<query2.fa> ...]
+```
+
+| Option | Default | Semantics |
+|---|---|---|
+| `-z` / `--findere-z` | from index metadata | Override Findere z parameter |
+| `--detail` | off | Emit per-position coverage vectors in JSON |
+| `--count-missing` | off | Add `kmer_missing` field to JSON |
+| `--force-presence` | off | Report 0/1 per genome regardless of index counts |
+| `--presence-threshold` | 1 | Minimum count to declare genome present |
+| `-T` / `--threads` | all CPUs | Worker threads |
+| `--chunk-size` | auto (from available RAM and thread count) | I/O chunk size in MiB — see [Future work, point 3](#throughput--parallelism--identified-potential-not-yet-implemented) for why the auto-sizing formula currently under-estimates memory on indexes with many genomes |
+
+`--mismatch` is accepted but currently ignored with a warning on stderr.
+
+---
+
+## Future work
+
+- **`--mismatch`**: 1-mismatch approximate matching — generate `3·k` single-substitution variants per k-mer, look each up independently.
+- **Read classification** (`--classify`): assign each read to the genome with the highest match score.
+- **Whitelist / blacklist filtering**: threshold-based accept/reject on per-genome match scores.
+
+### Throughput & parallelism — identified potential (not yet implemented)
+
+Observed on a 192-core (8×24 NUMA) machine: `query` uses ~10 cores or fewer, and the default chunk size gets the process OOM-killed. Root causes and candidate fixes, in dependency order:
+
+**1. Single-threaded I/O source (main core-utilization bottleneck).**
+`run()` builds `all_chunks` via `paths.into_iter().flat_map(read_sequence_chunks_sized(...))` and passes it directly as the `input` iterator to `pipe.apply()`. In `obipipeline::Pipe::apply` (`scheduler.rs`), `input.next()` is called exclusively from the dedicated source thread — so file opening, decompression, and FASTA/FASTQ chunk-boundary parsing for *all* input files run serially in one thread, regardless of `--threads`. Compare with `steps::scatter` (used by `index`) and `cmd/superkmer.rs`: there, file opening + streaming is itself a `Flat` pipeline stage (`||?`), executed across the `n_workers` pool, with `obipipeline::throttle(paths, max_open)` bounding concurrently-open files in the source thread. That pattern parallelises I/O across files (and NUMA nodes); `query.rs` cannot.
+Fix direction: restructure `query`'s pipe with an initial `Flat` stage analogous to `scatter`'s, opening/chunking files across workers instead of in `flat_map`.
+
+**2. Gzip decompression is inherently single-threaded per file.**
+`niffler`/`flate2` (used by `xopen`) do standard DEFLATE, which has no parallel-decodable structure for an arbitrary stream. Fix (1) parallelises *across* files but not *within* one large gzip file. Parking a possible fix (`rapidgzip-rs`) is tracked in [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen).
+
+**3. Chunk-size memory formula ignores `n_genomes`.**
+`chunk_bytes = available_memory_bytes() / (n_workers * 16)` (`query.rs:407-414`) assumes a fixed ~8–16× overhead per raw input byte. But `KmerResults::new` (`query.rs:165-179`) allocates `data: Vec<u32>` sized `total_kmers_in_chunk × n_genomes` — dense, **for every k-mer position in the chunk, hit or not** — plus `win_min` and (with `--detail`) `cov`, same scaling. Real per-chunk memory is `O(n_genomes)`, not constant; the formula doesn't know `n_genomes` at all. This is the direct cause of the OOM kill on indexes with many reference genomes.
+
+**4. MPHF lookup and matrix-row fetch are fused, not staged.**
+`QueryLayer::find_into` (`obikpartitionner/src/query_layer.rs:48-67`) does the MPHF `find` *and* the `fill_row` matrix read in one call per k-mer, inside a single-threaded loop (`query_partition_with`). There is no separation between "is this k-mer indexed" (cheap, `O(1)`, independent of `n_genomes`) and "what are its per-genome values" (the expensive, `n_genomes`-scaling part).
+
+**5. Dereplication should happen at k-mer granularity, directly — not via an intermediate superkmer-level dedup.**
+`QueryBatch::from_records` currently dereplicates at the *superkmer* level (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, `query.rs:112`). This misses redundancy between k-mers shared by *different* superkmers (read overlaps, repeats, a SNP splitting an otherwise-identical run). Superkmer *construction* (`SuperKmerIter`) stays mandatory — it is the mechanism that computes minimizers/partition routing, not an optional dedup layer — but the dedup structure built on top of it should key directly on `CanonicalKmer`, in the same pass: `HashMap<CanonicalKmer, Vec<(seq_idx, pos)>>`. This also means the MPHF `find` itself runs once per **distinct** k-mer instead of once per occurrence — a win independent of the matrix-fetch cost below.
+
+**6. Stage 1 output: bucket confirmed hits by layer, keyed by MPHF slot.**
+For each unique canonical k-mer, MPHF lookup across a partition's layers stops at the first match (`query_partition_with:105-111`) — a k-mer belongs to at most one layer. So stage 1's output can be reshaped directly into:
+```
+HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>
+```
+replacing the `CanonicalKmer` key by the resolved `slot` (compact integer, and exactly what stage 2 needs to address the matrix). K-mers matching no layer simply have no entry here (they still count toward `in_index`/`kmer_missing` bookkeeping, which stays `O(1)` per position, independent of `n_genomes`).
+
+**7. Partition-level parallelism is currently absent — and a NUMA-aware mechanism for exactly this already exists, unused, in `obikindex`.**
+`process_chunk`'s partition loop (`query.rs:250-278`, `for (part_idx, part_sks) in by_part.iter().enumerate()`) processes every partition of a chunk sequentially on the single worker thread that owns that chunk. This is a parallelism axis on its own, independent of the column question below.
+More importantly: `docmd/architecture/numa_partition_runner.md` and `numa_worker_pools.md` document `PartitionRunner` (`obikindex/src/numa.rs`), **already implemented** and already used by `merge.rs`, `index.rs` (`build_layers`), `select.rs`, `reindex.rs`, `rebuild.rs` — one controller thread per NUMA node, a Rayon pool pinned to that node's CPUs (`hwlocality`, `numa` feature, default-on in `obikindex/Cargo.toml`), adaptive worker activation driven by *both* a CPU-efficiency signal and an I/O-throughput signal (`CpuSample`/`IoSample`, `/proc/self/io` on Linux). It exists precisely because a naive `into_par_iter()` on the global Rayon pool measurably degrades ×60 on this codebase's own 192-core/8-NUMA reference machine (`numa_worker_pools.md`, § Problem) once workers contend for cross-socket memory bandwidth on shared mmap'd/hashed structures — exactly the shape of the matrix-column scan in point 8 below.
+`obikmer` already depends on `obikindex` (`obikmer/Cargo.toml`, for `KmerIndex`), so `PartitionRunner` is directly reachable from `cmd/query.rs` — no new dependency. Both the partition-level loop and (see point 8) the genome-column scan should be driven through it rather than through ad-hoc `rayon::into_par_iter()`, to avoid reproducing the already-measured-and-fixed contention problem. Also relevant: the "CPU-only signal stalls on I/O-bound stages" issue documented for `pack_matrices` (mmap-heavy, page-fault-bound) applies just as much to a column-major mmap scan over persistent matrices — reuse the existing dual CPU/IO activation signal rather than re-deriving one.
+>
+> **Correction from implementation (Phase 4 below)**: this turned out not to be viable as described. `PartitionRunner::run()`'s actual body spawns roughly one OS thread per worker slot across every NUMA node **on every call** (confirmed by reading `numa.rs`, not just its doc comments) — fine for the one-call-per-command-invocation batch usage in `merge`/`build_layers`, but `query_partition_with` runs once per `(chunk, partition)`, far too frequently to absorb that spawn cost. Partition-level parallelism via `PartitionRunner` is deferred, not implemented. See Phase 4's "What did not ship, and why" for the detail.
+
+**8. Stage 2: column-major matrix fetch, parallel across genome columns — via `PartitionRunner`, not naive `rayon`.**
+Both persistent matrix formats are column-oriented on disk: `ColumnarCompactIntMatrix`/`ColumnarBitMatrix` (`obicompactvec/src/{intmatrix,bitmatrix}.rs`) mmap one file per genome column; `PackedCompactIntMatrix`/`PackedBitMatrix` mmap one region-offset per column in a single file. `fill_row(slot, buf)` as used today (`query.rs:262-272` via `on_hit`) reads **one slot across all `n_genomes` columns** per hit — the worst possible access pattern for this layout (up to `n_genomes` scattered mmap regions touched per single k-mer).
+Better: for each layer, walk the matrix **column by column** (genome by genome): for each genome, scan the `slot` keys collected in step 6 for that layer and call `col.get(slot)`, keeping only nonzero results, and broadcast to the associated `(seq_idx, pos)` list. Total `get()` calls are unchanged (`n_hits × n_genomes` in the worst case) — the win is locality (sequential access within one mmap'd column at a time, not scattered across all columns per hit), not fewer operations.
+Columns are independent (read-only, disjoint mmap regions) → embarrassingly parallel across genomes, *but* — per point 7 — `obicompactvec`'s existing `into_par_iter()` over `0..n_cols` (`sum()`, `count_nonzero()`, pairwise distance matrices) is the **naive, unpinned** pattern the rest of the codebase is actively migrating away from, not a model to copy here. Route this through `PartitionRunner` (or the same NUMA-pool machinery) instead. Two things to settle when this is designed: how the partition axis (point 7), the column axis, and the existing chunk-level `n_workers` `obipipeline` pool compose without oversubscribing the machine (three different concurrency mechanisms — raw-thread pipe workers, `PartitionRunner`'s pinned Rayon pools, and whatever drives the column scan — need a single reconciled thread budget, not three independent ones); and the threshold below which per-column dispatch overhead outweighs the gain (small `n_genomes` or small per-layer hit counts) — to be measured, not assumed.
+>
+> **Correction from implementation (Phase 4 below)**: column-major fetch is implemented — but as a plain sequential loop, not parallelised via `PartitionRunner`. Same reason as point 7's correction above. The column-major *locality* win (the actual claim of this point) does not depend on adding parallelism on top of it, and is validated independently. Column-level parallelism is deferred pending a mechanism that fits this call frequency (candidates noted in Phase 4).
+
+**9. Sparse per-genome representation, fed directly to Findere.**
+Stage 2's output should be `HashMap<genome_idx, Vec<(seq_idx, position, count)>>`, **sorted by `(seq_idx, position)`** once collected, instead of a dense `KmerResults`-style matrix — the key must carry `seq_idx`, not just `genome_idx`, because a chunk batches many sequences and `position` is only meaningful within one; a plain `Vec<(position, count)>` per genome would silently mix positions from different sequences and corrupt the sliding-window scan. This bounds retained memory by actual nonzero hits on both axes (position sparsity from non-matching k-mers, genome sparsity from a matched k-mer typically belonging to only a handful of genomes out of possibly many). The Findere sliding-window (`process_chunk`, the `win_min`/deque loop) would need reworking to run per `(sequence, genome)` over its sparse, sorted `(position, count)` list — detect runs of ≥`z` consecutive positions, window-min within each run — instead of today's dense `O(total_kmers × n_genomes)` scan. This is also a genuine complexity win (`O(hits log hits)` per genome vs. dense scan), not just memory.
+**Not covered by this sparsification**: `--detail`'s `cov` accumulator (`query.rs:304-308`) has the identical `n_genomes`-dense scaling problem and wasn't folded into points above. It doesn't need to be retained densely throughout processing, though — only the final JSON serialization (`emit_batch`) requires a dense `[u32]` per `(seq, genome)`, and only for the sequences actually being output with `--detail`. Densification can stay a late, output-time-only step, reconstructed from the sparse per-genome lists.
+
+**Secondary patterns available from `scatter.rs`/`superkmer.rs`, not yet in `query.rs`:**
+- `throttle()` + `CommonArgs::effective_max_open()` to bound concurrently-open input files (query.rs defines its own `QueryArgs`, doesn't reuse this).
+- Progress bar with EMA throughput + live active-worker gauges (`obisys::spinner`, `flat_active`/`transform_active` counters) — diagnostic value for locating the bottleneck.
+- `obisys::Reporter`/`Stage::start`/`stop` timing per phase (used by `index`, `filter`; absent from `query`).
+
+None of this is implemented yet — parked here as a coherent roadmap while the design is discussed further. Suggested dependency order: (1) I/O parallelism → (3) genome-aware chunk sizing → (4)–(9) staged/k-mer-deduped/NUMA-aware-partition-and-column-major/sparse query engine (larger refactor, biggest structural payoff — reuses `PartitionRunner` rather than inventing a new parallelism mechanism) → (2) parallel gzip (separate, orthogonal, tracked in chunkreader.md) → secondary diagnostics patterns.
+
+---
+
+## Implementation plan
+
+Concrete, phased translation of the roadmap above. Phases 0–2 are small, independent, low-risk, and each individually testable against current `query` output — land them first, in order, and measure on the reference 192-core/8-NUMA machine before deciding whether phases 3–5 (the staged/sparse engine, the larger structural payoff) are still worth their cost. Phases 3–5 are one coordinated change spanning `obikmer`, `obikpartitionner`, and `obicompactvec` — they should not be split across releases mid-way, because the intermediate state (e.g. k-mer-level dedup feeding the old dense `KmerResults`) has no correctness or performance benefit on its own. Phase 6 is unrelated to phases 0–5 and can happen any time, independently, if `rapidgzip-rs` is validated (see [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen)).
+
+Instrumentation is deliberately sequenced *before* the I/O fix (reordering the roadmap's own listed order), because every later phase's justification rests on a measurement ("to be measured, not assumed" appears throughout the roadmap above) — without it, phases 3–5 would be undertaken on faith.
+
+Performance measurement on the reference 192-core/8-NUMA machine is done by the project owner, not from this development environment (macOS, 16 cores — `PartitionRunner`'s NUMA pinning is Linux-only, so even phase 4's mechanism can't be functionally exercised for its actual purpose here). Each phase below is therefore written to be *self-measuring*: the debug-level logging it adds must be enough, on its own, to judge whether that phase's algorithmic choice paid off from a cluster run's logs, without needing to attach a profiler.
+
+### Conventions applied to every phase below
+
+**Debug logging.** Every phase that changes an algorithmic choice (not phase 0, which *is* the logging) adds `tracing::debug!`/`trace!` at points that let a cluster run's logs answer "did this help": counts, ratios, and timings that quantify the specific claim that phase makes — e.g. phase 3 must log how many MPHF `find` calls were saved by k-mer-level dedup (the whole justification for that phase), phase 4 must log per-column scan timings, phase 5 must log actual retained-memory / sparsity ratios achieved. Prefer one structured `debug!` per chunk (fields, not prose) over free-text — the cluster logs will be the only evidence available for judging these choices, so they need to be grep/awk-able, not just readable.
+
+**Unit tests.** This project's convention (`obiread`, `obikseq`, `obidebruinj`, `obicompactvec`, `obilayeredmap`, `obiskio`, `obifastwrite`) is `#[cfg(test)] #[path = "tests/<name>.rs"] mod tests;` at the bottom of the source file, with the actual test code in a sibling `src/tests/<name>.rs`. Neither `obikmer` nor `obikpartitionner` (the two crates phases 3 and 5 touch most) currently have a `src/tests/` directory at all — this needs creating, following the existing pattern exactly, not inventing a new one.
+
+**Workflow (`jj`).** Work happens in a fresh `jj` commit, easy to abandon. `jj new` between phases is reasonable where it helps isolate a phase for review, but only when the working copy compiles at that point (project convention) — phase 3's internal sub-steps (batch dedup change, then `query_layer.rs` split, then the new return shape) will likely not each compile independently since they're one coupled change, so treat "commit boundary" and "plan phase boundary" as related but not forced to match 1:1; use judgement per phase rather than mechanically splitting on every bullet.
+
+### Phase 0 — Instrumentation (prerequisite for measuring every later phase)
+
+**Goal**: make core utilization, throughput, and per-stage timing visible on a real run, so phases 1–5 can be justified with numbers instead of assumption.
+
+- `obikmer/src/cmd/query.rs`: wrap `run()`'s main loop with `obisys::Reporter`/`Stage::start("query")`/`.stop()`, printed at the end via `rep.print()` — same pattern as `index.rs`/`filter.rs`.
+- Add an `obisys::spinner("query")` progress bar around the `pipe.apply(...)` loop, with an EMA throughput readout (bases/s or k-mers/s, mirroring `steps::scatter`'s `ema_rate` computation, `scatter.rs:88-118`) and live gauges for "chunks in flight" / "workers busy" — reuse the `AtomicU32` counter pattern from `scatter.rs` (`flat_active`, `transform_active`) rather than inventing a new one.
+- Add `max_open_files: Option<usize>` to `QueryArgs` and a `effective_max_open()` method mirroring `CommonArgs::effective_max_open()` (`obikmer/src/cli.rs:90-94`) — needed by phase 1's `throttle()` call. (`QueryArgs` can't just embed `CommonArgs` — it doesn't take `kmer_size`/`minimizer_size`/`partitions`/`level_max`/`theta` from the CLI, those come from the index metadata — so this is a small standalone addition, not a flatten.)
+- Add one structured `debug!` per `process_chunk` call: chunk byte size, sequence count, s-mer count, wall time, and (once later phases exist) the fields they add — this single log line is the baseline every later phase's own logging gets compared against.
+- **Validation**: none needed beyond "the numbers appear and look sane" — this phase changes no query logic or output.
+- **Deliverable used by every phase below**: a before/after throughput and core-utilization measurement on the reference machine.
+
+### Phase 1 — Parallel per-file I/O (fixes root cause of low core utilization)
+
+**Goal**: file opening, decompression, and chunk-boundary parsing run across the `n_workers` pool instead of serially in the pipe's dedicated source thread.
+
+- `obikmer/src/cmd/query.rs`:
+  - Replace the `paths.into_iter().flat_map(read_sequence_chunks_sized(...))` construction (current `run()`, building `all_chunks`) with `obipipeline::throttle(paths.into_iter(), args.effective_max_open())`, passed as the pipe's `input`.
+  - Add a new `QueryData::Path(PathBuf)` variant (alongside `Chunk`/`Output`) to carry the throttled path through the pipe's type-erasure mechanism.
+  - Add a new **first** pipe stage, `Flat`/fallible (`||?`), modeled on `scatter.rs:60-86` and `superkmer.rs:54-65`: given a `Throttled<PathBuf>`, call `read_sequence_chunks_sized(path, chunk_bytes)` and yield each `Rope` chunk, keeping `pw.guard` alive until the file's iterator is exhausted (reuse or adapt `scatter.rs`'s `GuardedIter` wrapper — same lifetime problem, same fix).
+  - The existing `process_chunk` transform stage becomes the pipe's **second** stage, unchanged in its own logic — it still receives one `Rope` chunk at a time, just no longer all coming from one serial source.
+  - `make_pipe!` invocation grows from one stage (`Chunk => Output`) to two (`Path => Chunk => Output`).
+- Log, per file: time spent waiting on the `throttle()` slot (queueing due to `max_open`), and time spent opening/decompressing/producing the first chunk — this is what directly proves (or disproves) that I/O is now spread across workers instead of serialized.
+- **Validation**: run `query` on a small multi-file input, diff output against the pre-change version — content must be identical; **record order across files is not guaranteed to be preserved** even before this change (chunk-level dispatch across `n_workers` already reorders completions), so the diff must be order-insensitive (sort by read id, or compare as sets) if it wasn't already.
+- **Measure**: core utilization on the reference machine with several large input files, compare against phase 0's baseline.
+
+### Phase 2 — Genome-aware chunk-size formula (fixes OOM)
+
+**Goal**: `chunk_bytes` reflects actual per-chunk memory (`O(n_genomes)`), not a fixed multiplier.
+
+- `obikmer/src/cmd/query.rs`, `run()`: `n_genomes` and `args.detail` are already computed above the `chunk_bytes` calculation (`n_genomes` at the top of `run()`, before line 407 in the current file) — reorder if needed, then replace:
+  ```rust
+  let computed = avail / (n_workers as u64 * 16);
+  ```
+  with a formula that scales the divisor by `n_genomes` (and roughly doubles it when `--detail` is set, since `cov` duplicates the per-genome accumulation): e.g. `per_chunk_multiplier = base_overhead + n_genomes as u64 * BYTES_PER_KMER_PER_GENOME * if detail { 2 } else { 1 }`, replacing the flat `16`. `BYTES_PER_KMER_PER_GENOME` should be derived from `KmerResults`'s actual layout (`4` bytes per `u32` entry in `data`, plus the `bool` in `in_index`, plus `win_min`'s equal-sized buffer) rather than guessed.
+  - `args.chunk_size` (manual `--chunk-size` override) keeps taking priority, unchanged.
+- Log the resolved `chunk_bytes`, `n_genomes`, and the estimated peak per-chunk memory (`chunk_bytes` × the same multiplier used to derive it) once at startup — lets a cluster run confirm the estimate was actually respected, not just that the process didn't get OOM-killed (which could also happen to be true for the wrong reason).
+- **Validation**: build a test index with a large `n_genomes` (e.g. hundreds), run `query` with default chunk sizing under a memory limit (`ulimit -v` or a cgroup), confirm it no longer gets OOM-killed and that memory scales as predicted when `n_genomes` grows.
+- **Note**: this phase is superseded once phase 5 lands (sparse retained memory no longer scales with `n_genomes × total_kmers` at all) — but it's needed immediately regardless, since phases 3–5 are a bigger, riskier change and users need a working `query` in the meantime.
+
+### Phase 3 — K-mer-level dereplication, staged MPHF/matrix lookup
+
+**Goal**: replace superkmer-level dedup with k-mer-level dedup (roadmap point 5), and split the fused MPHF-find/matrix-fetch (point 4) so stage 1's output is bucketed by layer and MPHF slot (point 6).
+
+- `obikmer/src/cmd/query.rs`:
+  - Replace `QueryBatch::from_records`'s dedup map (`HashMap<RoutableSuperKmer, Vec<SKDesc>>`, current `query.rs:112`) with a per-partition `HashMap<CanonicalKmer, Vec<(seq_idx: u32, pos: u32)>>`, built in the same `SuperKmerIter` pass: superkmer construction and partition routing (`part_idx` from the superkmer's minimizer hash) are unchanged, only the granularity of what gets deduplicated changes — each `CanonicalKmer` within a superkmer is inserted individually instead of the whole superkmer being the dedup key.
+  - **Verified**: `CanonicalKmer` (`obikseq/src/kmer.rs:390`, `pub type CanonicalKmer = CanonicalKmerOf<KLen>`) — the underlying `CanonicalKmerOf<L>` derives `Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash` (`kmer.rs:269`). Usable as a `HashMap`/`HashSet` key as-is, no change needed.
+- `obikpartitionner/src/query_layer.rs`:
+  - Split `QueryLayer::find_into` (`query_layer.rs:48-67`) into two methods: `find_slot(&self, kmer: CanonicalKmer) -> Option<usize>` (MPHF only, no matrix touch) and keep `fill_row` as-is for phase 4 to call later.
+  - Replace `query_partition_with`'s inner loop (`query_layer.rs:103-113`) with a version that, for each unique `CanonicalKmer`, calls `find_slot` across the partition's layers (stopping at first hit, same as today), and instead of immediately filling a row, records `(layer_idx, slot)`.
+  - New return shape for the partition-level query, replacing today's `on_hit(sk_idx, kmer_idx, row)` callback: `HashMap<layer_idx, HashMap<slot, Vec<(seq_idx, pos)>>>` (roadmap point 6) — built directly from the k-mer dedup map's `Vec<(seq_idx,pos)>` values, keyed by the resolved slot instead of the k-mer.
+- **This phase alone has no throughput benefit yet** (matrix fetch still happens, just deferred) beyond the k-mer-level dedup itself (fewer MPHF calls when queries have overlapping/repeated k-mers) — its purpose is to produce the input phase 4 needs. Land phase 3+4 together, not phase 3 alone, per the "don't split 3–5 across releases" note above.
+- Log, per chunk: total k-mer occurrences vs. unique `CanonicalKmer` count (the dedup ratio — the entire justification for this phase) and the resulting MPHF `find` call count. If the dedup ratio is close to `1.0` on real query data (little redundancy), that's the cluster run telling us this phase wasn't worth it — the logging needs to be able to say that, not just confirm the happy path.
+- **Unit tests**: create `obikmer/src/cmd/tests/query.rs` (new `src/tests/` dir for this crate, following the project's `#[cfg(test)] #[path = "tests/query.rs"] mod tests;` convention) and `obikpartitionner/src/tests/query_layer.rs` (likewise new for this crate). Cover: the k-mer-level dedup map construction on synthetic sequences with known repeated/overlapping k-mers (assert unique-kmer count and occurrence lists); the `find_slot`/bucket-by-layer-and-slot construction against a small hand-built `QueryLayer` fixture, asserting the `(layer_idx, slot, seq_idx, pos)` tuples match what the old per-occurrence loop would have produced.
+
+### Phase 4 — Column-major matrix fetch (roadmap points 7–8) — implemented, NUMA parallelism deferred
+
+**Goal (revised during implementation)**: replace `fill_row`-per-hit (row-major, worst-case mmap locality) with a column-major scan. `PartitionRunner` turned out to be the wrong mechanism for this at this call granularity — see below; the column-major fetch itself is implemented and validated, without it.
+
+**What shipped:**
+- `obicompactvec`: the per-column accessors this phase needed **already existed** — `PersistentCompactIntMatrix::col_view(c)` and `PersistentBitMatrix::col_view(c)` are public, and `IntSliceView::get(slot)`/`BitSliceView::get(slot)` are public — the original plan underestimated how much of this plumbing the pairwise-distance code (`dump`/`select`/`stats`) had already required. The one real gap: `PersistentBitMatrix::col_view()` panics on the `Implicit` variant (the documented mono-genome fast path, `bitmatrix.rs`). Added `PersistentBitMatrix::get(c, slot) -> u32` (`bitmatrix.rs`), a non-panicking column-major point lookup that returns `1` for `Implicit` regardless of `c` — the smallest surface needed, not a new `col_get` API from scratch.
+- `obikpartitionner/src/query_layer.rs`: `query_partition_with` is now two explicit stages, matching roadmap points 6–8: **stage 1** (MPHF-only, per unique k-mer, bucket hits by `(layer_idx, slot)`, emits `QueryHit::Found`) then **stage 2** (per layer with ≥1 hit, column-major: for each genome column `g` in `0..layer.n_cols().min(n_genomes)`, scan that layer's bucketed slots and call `col_value(g, slot)`, emitting `QueryHit::Value(descs, g, value)` on nonzero). `QueryHit` is a single enum delivered through one `FnMut(QueryHit)` callback — an earlier two-closure design (`on_found` + `on_value`) didn't borrow-check, since the caller's single mutable accumulator (`KmerResults`) can't be captured by two separate `FnMut` closures passed to the same call.
+- `obikmer/src/cmd/query.rs`: `KmerResults::set` (row-major, whole-row-at-once) replaced by `mark_found` (stage 1: flag a position as indexed, independent of any genome's value) and `set_one` (stage 2: write one genome's value at one position). `QueryStats` extended with `n_columns_scanned`/`n_col_get_calls`, logged per chunk.
+- Total `get()`-equivalent calls are unchanged from the row-major version (`n_hits × n_cols` in the worst case, confirmed by `n_col_get_calls` in the debug log) — the win is locality (sequential access within one layer's column at a time, across `mmap`'d regions, instead of jumping across all columns per hit), exactly as predicted.
+
+**What did not ship, and why — `PartitionRunner` is architecturally the wrong tool here:**
+Reading `obikindex/src/numa.rs`'s actual `run()` body (not just its doc comments) shows every call spawns a timer thread **plus one OS thread per worker slot on every NUMA node** (`std::thread::scope` + one `s.spawn()` per node per `max_workers`) — on the 192-core/8-NUMA reference machine, that's on the order of 190+ fresh OS threads spawned **per call**. This is fine for its actual, established usage in this codebase (`merge.rs`, `index.rs`'s `build_layers`): one `PartitionRunner::new()` + one `run()` call per command invocation, amortised over a batch of ~256 long-running partitions. It is not fine for `query`'s call pattern: `query_partition_with` runs once per `(chunk, partition)`, potentially thousands of times per second — spawning ~190 OS threads that often to scan a handful of genome columns would very likely cost far more than the row-major approach it's meant to replace. This is exactly the "resolve empirically, don't assume" composition risk the roadmap flagged, just resolved by reading the mechanism's actual cost before wiring it in, rather than by measuring a regression on the cluster after the fact.
+The column-major loop in stage 2 is therefore a **plain sequential loop** for now — it captures the whole, provable locality win (roadmap point 8's actual claim) without adding any parallelism mechanism. Genome-column-level parallelism (point 8's "bonus" axis) and partition-level parallelism (point 7) are both deferred — not abandoned. Candidates for a follow-up, once there's a concrete profiling need: (a) `rayon`'s already-warm global pool (`into_par_iter()`) for the column axis specifically — cheap to invoke repeatedly since it doesn't spawn threads per call, though it's the same "naive rayon" pattern `numa_worker_pools.md` warns about for a *different* workload (random pointer-chasing over large hash maps); a column scan's access pattern (sequential reads within one `mmap`'d region) has a different contention profile and hasn't been shown to have the same problem — needs its own measurement, not an assumption either way; (b) restructuring so `PartitionRunner` is invoked once per whole `query` run (or per large batch of chunks) rather than per `(chunk, partition)`, amortising its spawn cost the way `merge`/`build_layers` do — a bigger structural change than this phase's scope.
+- Log (implemented): `QueryStats::n_columns_scanned`/`n_col_get_calls`, folded into the existing per-chunk `debug!("k-mer dedup + column-major fetch", ...)` line (`query.rs`) alongside phase 3's dedup counters.
+- **Unit tests**: extended `obikpartitionner/src/tests/query_layer.rs` (phase 3's file) — `query_partition_with`'s empty/missing-index paths updated for the new `QueryStats` fields and single-callback signature.
+- **Validation performed**: full workspace build + `cargo test --workspace`, zero failures. Functional validation against real indexes: (1) a single-genome index — output byte-identical to pre-phase-4 (same `kmer_count`/`kmer_strict_matches` on every record); (2) the existing 20-genome `benchmark/global_index_presence` index — runs correctly, `n_hits=0` for an unrelated query (expected: no shared k-mers between a plant read and a bacterial reference set), no panics, confirming the `Implicit`/multi-column bounds logic doesn't crash on a real multi-genome, mixed-format index; (3) **the critical correctness case**: built two single-sequence-pair test genomes, merged into one 2-genome index, queried with reads from both — reads from `genomeA` matched **only** `genomeA` (`kmer_count` identical to the pre-dedup occurrence count, zero leakage into `genomeB`'s column) and vice versa. This is the test that would have caught a column-index mixup, an off-by-one in `n_cols`, or cross-genome bleed from the stage-1/stage-2 split — it passed cleanly.
+- **Not yet done**: the microbenchmark comparing column-major vs. the old row-major access pattern's wall time / page-fault counters on a large-`n_genomes` layer — needs a realistically large multi-genome index and, for the page-fault counters specifically, Linux (not available from this development environment). Left for cluster validation alongside phases 1–3's own pending measurements.
+
+### Phase 5 — Sparse Findere rework (roadmap point 9)
+
+**Goal**: replace the dense `KmerResults`/`win_min` sliding-window scan with one operating on phase 4's sparse per-genome output.
+
+- `obikmer/src/cmd/query.rs`, `process_chunk`:
+  - Remove `KmerResults` (`query.rs:157-202`) and the dense `win_min` allocation (`query.rs:290-291`, sized `max_n_kmers × n_genomes`).
+  - Keep a lightweight dense `in_index: Vec<bool>` per chunk (sized `total_kmers`, independent of `n_genomes`) from phase 3's stage 1 — still needed for `kmer_missing` bookkeeping (leftmost-s-mer-of-window membership test), which phase 4's sparse structure doesn't carry (a k-mer with no genome hit has no entry there at all).
+  - New per-`(seq_idx, genome)` scan: for each genome's `Vec<(seq_idx, pos, count)>` (sorted, per phase 4), group by `seq_idx` (contiguous after sort), then within each sequence's positions detect runs of `pos, pos+1, pos+2, ...` of length ≥ `z`; within each run, the existing monotone-deque window-minimum logic (`query.rs`'s current `dq` loop, conceptually unchanged) applies — but the deque now only scans real entries in the run, never zero-filled gaps.
+  - Update `SeqAcc` accumulation and `emit_batch` to consume this per-genome sparse iteration instead of `results.val`/`results.is_in_index`.
+  - `--detail`/`cov`: build sparsely during the same scan (only positions with a confirmed contribution get an entry), densify into the `[u32]` JSON array only in `emit_batch`, only for genomes/sequences actually being serialized (per roadmap point 9's note, `query.rs:304-308`'s current dense allocation goes away).
+- Log, per chunk: total sparse entries retained vs. what the old dense `KmerResults` would have allocated (`total_smers × n_genomes`) — the sparsity ratio is this phase's entire reason for existing, so it must be directly visible in the logs, not inferred from process RSS. Also log the run-detection stats (number of runs found, average run length) — a low average run length relative to `z` would mean most positions still fail to form a full window, worth knowing.
+- **Unit tests**: `obikmer/src/cmd/tests/query.rs` (extended from phase 3) — the property test described below is the primary deliverable here, not an afterthought; write it as an actual `#[test]` (or a small internal fuzz/property-style loop over randomized fixtures if a property-testing crate isn't already a dependency — check before adding one, per this project's dependency-approval rule) rather than a one-off manual comparison.
+- **Validation — this is the correctness-critical phase**: property-test comparing old (dense, pre-phase-3) and new (sparse) implementations on the same randomized input/index fixtures, asserting identical `kmer_count`, `kmer_missing`, `kmer_strict_matches`, and (with `--detail`) `coverage` for every sequence. Keep both implementations compiled side by side (behind a debug-only flag or a temporary parallel code path) only for the duration of this validation; delete the dense path once parity is confirmed — per this project's own convention, superseded code is not kept "just in case."
+- **Update `docmd/architecture/query.md` itself**: once this phase lands, the "Findere z-window filter" section (which currently — correctly — describes the dense deque-over-`0..n_smers` scan) needs another pass to describe the sparse run-detection algorithm instead, as already flagged when this phase was discussed.
+
+**Implemented as planned, no deviations discovered this time.** What shipped:
+- `KmerResults` removed entirely, replaced by `SmerIndex` (`in_index: Vec<bool>` + `offsets`, unchanged size/purpose, renamed since it's no longer "results" — just the O(1)-per-position "was this k-mer found at all" bookkeeping) and `by_genome: Vec<Vec<(seq_idx, pos, value)>>` (one empty `Vec` per genome until a hit arrives — genomes with zero hits in a chunk cost nothing beyond the outer `Vec`'s own allocation).
+- New `sparse_findere_for_genome(hits, z, presence, threshold) -> (Vec<ConfirmedHit>, n_runs, total_run_len)` (`query.rs`): sorts one genome's raw hits by `(seq_idx, pos)`, detects maximal runs of consecutive `pos` within one sequence, runs the same monotone-deque window-minimum as before but scoped to each run (run-relative indices for eviction, absolute `pos` for computing `pos_out`). Presence/count adjustment (`u32::from(win_min >= threshold)` vs. raw) is applied inside this function, once per confirmed hit, rather than later during accumulation.
+- `process_chunk` restructured into three passes after the partition loop: (1) run `sparse_findere_for_genome` per genome, collecting `confirmed_by_genome` and run-detection stats; (2) accumulate `genome_totals` directly from `confirmed_by_genome` and mark a `confirmed_any: Vec<bool>` (sized `total_kmers_out`, not `× n_genomes`); (3) a position-only pass (`O(total_kmers_out)`, no genome factor) computing `kmer_count`/`kmer_missing` from `confirmed_any` + `SmerIndex`. `cov` (`--detail`) is populated by re-scanning `confirmed_by_genome` — only when `--detail` is actually set, otherwise skipped entirely.
+- Debug log added (`"sparse Findere"`): `n_dense_would_be` (`n_occurrences × n_genomes` — what the deleted dense path would have allocated), `n_sparse_entries` (what's actually retained), `n_runs`/`avg_run_len` (per the plan's ask, to see whether hits mostly fail to form complete windows).
+- **Unit tests**: `sparse_findere_matches_dense_reference_on_random_inputs` (`obikmer/src/cmd/tests/query.rs`) — 200 randomized cases (sequence count/length, `z`, presence/count mode, threshold, hit density from sparse to fully-dense) comparing `sparse_findere_for_genome` against `dense_reference_findere`, a faithful reimplementation of the deleted dense algorithm kept only as a test-local correctness oracle (no property-testing crate added — checked first, none was a workspace dependency; a small `std`-only xorshift64 PRNG stands in for one, deterministic and dependency-free). All 200 cases pass.
+- **Functional validation performed**: full workspace build + `cargo test --workspace`, zero failures. End-to-end against real indexes: baseline output (no flags) unchanged from pre-phase-5 recorded values on the same fixtures; `--count-missing` correct (`kmer_missing: 0` on a self-match); `--detail` correct — coverage array length matches `kmer_count`, and critically, re-ran the two-genome cross-contamination check from phase 4 with `--detail --count-missing`: `genomeA` reads show coverage sum `106` for `genomeA` and `0` for `genomeB` (and vice versa) — confirms the sparse-to-dense `cov` reconstruction doesn't leak across genomes either, not just the scalar `kmer_strict_matches` path.
+- This phase's roadmap item ("update the Findere z-window filter section") — done, see above; the "Algorithm" section's pseudocode was also updated, since it still named `KmerResults`/`SKDesc` from before phases 3–4.
+
+### Phase 6 — Parallel gzip decompression (independent, optional)
+
+Tracked separately in [chunkreader.md](../implementation/chunkreader.md#future-work--parallel-gzip-decompression-in-xopen); parked pending validation of `rapidgzip-rs` on real data. Not a dependency of, or a dependency for, phases 0–5 — `xopen` is shared infrastructure (`obiread`), phase 1 benefits from it but doesn't require it (phase 1 parallelises *across* files; this phase would additionally parallelise *within* one large file).
+
+### Cross-cutting risks
+
+- **Thread-budget oversubscription** (phase 4): the single biggest unresolved design question in this whole plan — see phase 4's composition note. Should be settled with real measurements early in phase 4, not assumed from the design alone.
+- **`obicompactvec` API surface growth** (phase 4): new public per-column accessors are additive (existing `fill_row`/`row` stay for other callers — `dump`, `select`, distance computations) — no breaking change expected, but worth checking `obicompactvec`'s other callers aren't already relying on `fill_row` being the only/cheapest access path in a way that would make maintaining two access patterns (row-major and column-major) a real maintenance cost rather than a one-off addition.
+- **`PersistentBitMatrix::Implicit`'s hardcoded `n_cols: 1` — resolved, not a bug.** `LayerMeta`'s own doc comment (`obicompactvec/src/layer_meta.rs:1-9`) states it is written "alongside `mphf.bin`" and read by `PersistentBitMatrix::open` "to determine `n_rows` for **the implicit (mono-genome presence/absence) case**" — i.e. `Implicit` is a documented single-genome fast path (no presence matrix needed when there is trivially one genome), not a generic "no matrix built yet" fallback. `n_cols: 1` is correct by design for the case it's meant to handle. Phase 4's column loop is safe as planned — this was worth checking once, doesn't need further action.

DevDocMD/architecture/query.refs.md

@@ -0,0 +1,16 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: architecture/query.md
+
+## Code couvert
+
+- `obikmer/src/cmd/query.rs` — commande query, format de sortie
+- `obikpartitionner/src/query_layer.rs` — routage de la requête à travers les partitions
+- `obiread/src/lib.rs` — lecture des séquences d'entrée pour la requête
+
+## Notes
+
+RISQUE DE DÉRIVE. Vérifier :
+- La commande `unitig` a été modifiée pour utiliser `open_sequential()` — vérifier si query est concerné
+- `find_exact` / `find_approx` / `find` générique ont été ajoutés dans `MphfLayer` — le chemin de requête a changé
+- Si l'index est approximatif (Approx), la requête peut produire des faux positifs : la doc le mentionne-t-elle ?
+- Format de sortie CSV (`obikindex/src/csv.rs` ou équivalent) à vérifier

DevDocMD/architecture/rebuild_filter.md

@@ -0,0 +1,105 @@
+# Rebuild / filter — column-first design
+
+## Problem with the current two-pass design
+
+`rebuild_partition` currently makes **two full passes** over source data:
+
+**Pass 1** — read unitigs → MPHF lookup (source) → read row (108 values) → apply filter → push kmer into `GraphDeBruijn`, **discard row**.
+
+**Pass 2** — read unitigs again → MPHF lookup again → read row again → for each passing kmer, look up slot in new MPHF → fill column builders.
+
+Both passes do random access into the source matrix: for each kmer, the MPHF returns a slot, then we read 108 values scattered across 108 column positions. This is cache-hostile even with a packed matrix (`.pbmx`), because the matrix is column-major: consecutive row reads jump across the file.
+
+## Memory budget
+
+The `keep` bitvector costs **1 bit per slot**. With 256 partitions and realistic kmer counts, each partition holds at most a few tens of millions of slots → a few MB per bitvector. Even in the absolute worst case (800 M slots), it stays under 100 MB. This is negligible.
+
+The `slot_map` option (Option B, 8–16 bytes per slot) is heavier but still bounded: at 15 M slots and 8 bytes, that is 120 MB per partition, acceptable for a single worker.
+
+## Key observation
+
+**The filter operates on column values, not on kmers.** A filter like `--max-outgroup-count 0` only needs to know, for each slot, whether any outgroup column is non-zero. It does not need to know which kmer occupies that slot.
+
+This means filtering can be done as a **sequential column scan** that produces a `keep: BitVec[n_slots]` — no MPHF lookups, no kmer knowledge, perfectly cache-friendly.
+
+## Proposed single-scan design
+
+### Step 1 — column scan → `keep` bitvector
+
+```
+for each column c in source matrix:
+    read column c sequentially (one mmap range)
+    update keep[slot] according to filter contribution of column c
+```
+
+For `GroupQuorumFilter` with ingroup/outgroup:
+- ingroup columns: count presence per slot → `ingroup_count[slot]`
+- outgroup columns: `keep[slot] &= (value[slot] == 0)` (early-exit possible)
+
+Result: `keep: BitVec` of size `n_slots`, computed with purely sequential IO.
+
+### Step 2 — unitig scan → kept kmers + new MPHF
+
+```
+for each kmer in unitig files:
+    old_slot = old_MPHF(kmer)
+    if keep[old_slot]:
+        push kmer into new GraphDeBruijn
+        record (old_slot, kmer)   ← or just old_slot in order
+```
+
+Build new MPHF from `GraphDeBruijn` via `materialize_layer`.
+
+### Step 3 — fill new matrix
+
+Two sub-options:
+
+**Option A — from recorded (old_slot, kmer) pairs:**
+
+```
+for each (old_slot, kmer) in recorded list:
+    new_slot = new_MPHF(kmer)
+    for each column c:
+        new_matrix[new_slot, c] = old_matrix[old_slot, c]
+```
+
+Memory cost: `n_kept × (8 + 8)` bytes for `(old_slot: usize, kmer: CanonicalKmer)`.
+For species-specific filters, `n_kept` is small. For unfiltered rebuild, `n_kept = n_slots`.
+
+**Option B — column-by-column copy using old→new slot mapping:**
+
+Precompute `slot_map: Vec<Option<usize>>` of size `n_slots`:
+- For each kmer in unitig file: `slot_map[old_MPHF(kmer)] = Some(new_MPHF(kmer))`
+
+Then for each source column:
+```
+read source column sequentially
+for each slot where slot_map[slot] = Some(new_slot):
+    write value to new column at new_slot
+```
+
+Memory cost: `n_slots × sizeof(usize)` for the slot map (one usize per source slot).
+IO pattern: sequential read of each source column → random write into new column builders.
+
+Option B avoids storing kmer values and works uniformly regardless of filter selectivity.
+
+## Comparison
+
+| | Current | Proposed |
+|---|---|---|
+| Disk reads | 2× unitigs + 2× random matrix | 1× columns (sequential) + 1× unitigs |
+| MPHF lookups (source) | 2× N_kmers | 1× N_kept (step 2) or 0 (option B, col scan only) |
+| Cache behavior | poor (random row access) | good (sequential column scan) |
+| Extra memory | none | slot_map (option B) or (old_slot, kmer) list (option A) |
+
+## Files to modify
+
+- `src/obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition` and `iter_src_layers`
+- Possibly `src/obicompactvec/` — add column iterator API if not already present
+- `src/obilayeredmap/` — check if per-column sequential access is exposed on `SrcLayerData`
+
+## Open questions
+
+- Does `SrcLayerData` expose per-column sequential iteration, or only `lookup(kmer, n_genomes)` random access?
+- For option B: are new column builders writable in random-slot order (i.e. `set_val(slot, value)` without sequential constraint)?
+- For `GroupQuorumFilter` specifically: can the filter be decomposed into independent per-column contributions, or does it need the full row?

DevDocMD/architecture/sequences/invariant.md

@@ -0,0 +1,9 @@
+# Manipulated sequences
+
+- We consider sequences only as compact form of a set of overlaping kmers
+- The largest kmers we considere are 31-mer
+- We only consider odd k
+
+- all sequences match /^[acgtACGT]+/
+- maximum length 256 nucleotides
+- minimum length k

DevDocMD/architecture/sequences/invariant.refs.md

@@ -0,0 +1,12 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: architecture/sequences/invariant.md
+
+## Code couvert
+
+- `obikseq/src/sequence.rs` — invariants de représentation des séquences (ACGT, longueur max)
+- `obikseq/src/unitig.rs` — type Unitig, contrainte MAX_KMERS_PER_CHUNK (255 kmers par chunk)
+
+## Notes
+
+Document court et stable. Vérifier que la limite de 256 nucléotides (ou 255 kmers) par chunk
+est toujours la même dans `obiskio::MAX_KMERS_PER_CHUNK`.

DevDocMD/architecture/siblings.md

@@ -0,0 +1,742 @@
+# Sibling annex — architecture (discussion)
+
+Status: architecture decided (2026-08-14). Implementation not yet mandated.
+
+## Two index spaces, uncorrelated
+
+Every kmer stored in a `Layer` lives in two independent index spaces:
+
+- **Iteration order**: its position when enumerating `unitigs.bin` (the
+  superkmer file), deterministic but arbitrary with respect to slot.
+- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns.
+
+The two are not correlated by any formula. Converting from one to the other
+requires either recomputing the MPHF (kmer → slot) or scanning the iteration
+stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a
+one-way function, not an invertible bijection with a stored inverse. Any
+method that reconstructs a kmer from a bare slot number is wrong by
+construction, regardless of the mechanism used (MPHF re-hash, or evidence
+decode + direct unitig read). See `MphfLayer::kmer_at`
+(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
+from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
+"Pending work" status below).
+
+## Two pipelines, never mixed
+
+| | origin of the kmer | membership known? | correct mapping |
+|---|---|---|---|
+| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check |
+| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence |
+
+Evidence exists solely to answer "is this external kmer a member of the
+layer" for the query pipeline. Using it (or the MPHF) to go the other way —
+recover a kmer from a slot, or re-verify a kmer that was just produced by
+iterating the layer — is a conceptual error: evidence can be probabilistic
+(`Approx` mode), so any slot→kmer attempt is unsound in general, and
+pointless even in `Exact`/`Hybrid` mode since the kmer was already known.
+
+## Sibling annex: an iteration-pipeline artifact only
+
+The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`,
+`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a
+family minorant and which family members are present in the index. Its only
+consumers (`obikphylo/siblings/stats.rs`, `family_scan.rs`) enumerate it
+exhaustively (`0..annex.len()`); no query-pipeline code path touches it.
+
+**Decision**: the annex must be persisted in iteration order, not slot
+order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex
+file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
+no `kmer_at`. It also enables specialized iterators building on this zip:
+minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.
+
+Today the annex is built and stored in **slot** order
+(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated
+via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is
+threaded through the whole cross-partition reconciliation pipeline (variant
+generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This
+must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading
+the **iteration index** instead of the slot end to end — eliminating
+`kmer_at` from the build path entirely, not just the read path. No
+slot-indexed intermediate is needed even during construction; the
+iteration-order id is sufficient throughout.
+
+The cross-partition side of the same pipeline is unaffected: checking
+whether a generated family-variant kmer exists in another partition is a
+genuine query-pipeline operation (the variant's membership in the *target*
+partition is unknown) and must keep going through
+`KmerPartition::query_partition_with` (MPHF + evidence), never a raw
+`index()`.
+
+## Pending work — done
+
+The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
+traits over `obikindex::KmerIndex`/`obilayeredmap::Layer<D>`, replacing the
+old `obikindex::siblings` module) builds and reads the annex purely in
+iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
+batch variants, mirroring `Layer<D>`'s own `KmerIter`/`KmerBatchIter`
+shape). `MphfLayer::kmer_at` has no remaining callers.
+
+A separate, unrelated bug surfaced during this work and was fixed
+(2026-08-14): `MphfLayer::enumerate_kmers_batch` computed its
+`batch_start_index` via the stdlib `.enumerate()` adapter, which counts
+*batches* (0, 1, 2…), not the cumulative k-mer offset the annex is actually
+keyed on — every batch past the first wrote its mask/annex entries at the
+wrong iteration-order position. Fixed by tracking a running offset instead;
+regression tests added (`sibling_annex_no_empty_masks_after_build`,
+`sibling_histogram_does_not_panic_on_partial_last_batch`).
+
+## Performance: `build_sibling_annex` parallelism (2026-08-14)
+
+Investigated on a real multi-genome run (`phyloskims_sal_vac`, k=31/m=11).
+Baseline: mostly one active core, with short multi-core bursts — average
+~3 cores.
+
+**Fixes that helped, kept:**
+
+- `CanonicalKmerOf::minimizer()` (`obikseq/src/kmer.rs`) — a direct O(k)
+  bit-arithmetic minimiser for a single isolated k-mer, replacing a
+  `RollingStat` instance fed byte-by-byte through an ASCII round-trip (used
+  by `helpers::partition_of`, called for every generated family variant).
+  ~3x wall-clock improvement on its own, confirmed by sampling
+  (`obiskbuilder::rolling_stat`/`obikentropy` frames disappeared from the
+  hot path). `CanonicalKmerOf::partition()` added alongside it (wraps
+  `minimizer().seq_hash() & mask`, the same routing rule
+  `KmerPartition`/`RoutableSuperKmer` use).
+- Cross-partition resolution (`outgoing.par_iter()` in
+  `build_layer_sibling_annex`) parallelised at the *partition* level — one
+  Rayon task per non-empty `outgoing[dest]` bucket. For k=31/m=11, a
+  central-base substitution changes the winning minimiser (and thus the
+  destination partition) only when that window overlaps the central base:
+  ~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants
+  route right back to the partition already being built. That self bucket
+  ends up far larger than any other, so the per-partition split pinned one
+  thread to it alone while the rest of the pool finished instantly —
+  confirmed by sampling: one thread solid in `MphfLayer::find`, everyone
+  else idle. Fixed by splitting each non-empty bucket into
+  `total_queries / n_workers` (capped 4096) chunks *before* `par_iter()`,
+  preserving per-partition mmap locality (each chunk stays contiguous
+  within one partition) while letting Rayon spread an oversized bucket
+  across several threads. Net effect of both fixes together: ~3 cores
+  average → ~10-13 cores average on the same run, and a projected total
+  build time of ~1h15 down to ~30min on the real `phyloskims_sal_vac` run
+  this was measured against.
+- `TracedBar`'s ETA (`obisys/src/progress.rs`) was silently starved: the
+  custom progress message and the self-computed ETA text used to share one
+  `pb.set_message()` slot, with the ETA holding off for 2s after any custom
+  message — fine when custom messages are rare, broken once
+  `build_sibling_annex`'s per-partition callback fires more often than
+  that. Fixed by keeping the two texts in separate fields, composed
+  together on every render instead of one overwriting the other.
+
+**Tried and reverted — do not repeat blindly:**
+
+- Parallelising the *outer* partition loop in `build_sibling_annex` with
+  `obikindex::PartitionRunner` (already used by `merge`/`build_layers`),
+  splitting a fixed core budget between outer (partition) and inner
+  (pipeline + resolution) concurrency so their product wouldn't exceed the
+  budget. Measured *worse*: throughput dropped over time (26
+  partitions/5min → 38/11-12min) and peak resolution concurrency fell from
+  ~11-12 cores to ~7-8. Cause: this capped the resolution burst — which
+  scales very well on its own — to make room for outer concurrency, and
+  running several partitions' resolution at once scatters access across
+  multiple partitions' mmap regions at once, working against the
+  locality `outgoing`'s per-partition grouping exists for. `PartitionRunner`
+  stayed exported from `obikindex` (`new_capped` too) since it's
+  general-purpose, but nothing in `obikphylo` calls it.
+- Splitting resolution chunks even finer (`/(n_workers*8)`, cap 1024,
+  instead of `/n_workers`, cap 4096) to smooth the residual sawtooth.
+  Measured ~10% *slower*, wider dips, not narrower. Reverted to the
+  original chunk sizing.
+
+**Known remaining limitation, not yet worth fixing:** within one layer, the
+four stages (sequential `unitigs.bin` read → parallel generation →
+parallel resolution → sequential annex write) never overlap — confirmed by
+1s-interval sampling: generation alone occupies ~17 threads evenly, but the
+next layer's read/generation never starts until the current layer's
+resolution and write are both done. This produces a real, periodic (~layer
+duration) alternation between "many cores" and "few cores" that neither of
+the fixes above touches, since both operate *within* one layer's resolution
+step. The only remaining lever is overlapping consecutive layers (e.g. a
+depth-2 pipeline: start layer N+1's read/generation while layer N's
+resolution/write is still running) — a real restructuring, not a parameter
+tweak, and explicitly *not* to be combined with the reverted
+budget-capping idea above (let each phase use however many cores it
+naturally wants; only the *scheduling* needs to overlap). Deferred, not
+started.
+
+## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)
+
+`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`,
+`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already
+implements most of a dispatch/accumulate/resolve pipeline: generation
+(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and
+buckets cross-partition queries) runs on an `obipipeline::throttle` +
+`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter`
+*across partitions*, one generated batch resolved at a time, never several
+concurrently — this ordering is deliberate, see the module's own docs on a
+reverted concurrent-batch-resolution attempt that scattered mmap access).
+The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`)
+already exists: `n_layers <= 7` (checked once from the first non-empty
+partition's `PartitionMeta::n_layers`, documented as identical across every
+partition of an index — a structural, build-time property, never a
+per-partition state) decides whether `FamilyMask`'s recorded `layer_value`
+can be trusted to skip straight to the right layer
+(`find_presence_batch_fast`) or must fall back to scanning every layer of
+the destination partition (`find_presence_batch`).
+
+**Real gap, confirmed not implemented**: resolution is triggered by the
+*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the
+scanned layer), not by an *output* accumulator filling up. Since most
+central-base variants of a family route back to the same partition being
+scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s
+`outgoing[dest]` is large for the local/self partition and thin for the
+other ~255 (or however many) destination partitions — each of those gets
+resolved at low query density every batch instead of being accumulated
+across several source batches until resolving it is worthwhile. This is
+distinct from, and not fixed by, the fast/slow layer gate above.
+
+Redesign sketched (not built): per-destination accumulators decoupled from
+`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch
+completion — a "hot" accumulator for the partition being scanned (sharded
+one-per-generation-worker, no lock, since all `n_workers` pipeline workers
+write to it concurrently — this differs from an earlier, simpler mental
+model of "one thread owns one layer's local collector," which doesn't hold
+here since `n_workers` threads cooperate on scanning *one* layer at a time,
+not one thread per layer) and "cold" mutex-per-partition accumulators for
+the rest, low contention expected since traffic to any single cold
+destination is a small fraction of total.
+
+This breaks the current strict-iteration-order delivery of `on_family`
+(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH`
+resolves atomically). With cross-batch accumulation, a family only becomes
+complete once *every* accumulator holding one of its outgoing queries has
+flushed, at unpredictable, independent times — no longer streamable
+strictly in order without a large, unbounded pending buffer. Resolution
+sketched: replace order-dependent consumers with coordinate-addressed
+writes instead of order-dependent appends (see `PseudoAlignment` idea
+below) wherever possible, since `sibling_annex_stats`'s reduction (plain
+counts) is already order-independent and needs nothing here.
+
+**Superseded 2026-08-15** by the `--subsample`/`--shannon` design below,
+which sidesteps the accumulator redesign for now: bounding the number of
+families actually resolved per layer (via sampling) keeps per-layer
+resolution volume small enough that the batch-density problem above stops
+mattering in practice for these two consumers. The accumulator redesign
+remains relevant for a future *unsampled, full-index* run, but is not
+required to ship `--subsample`/`--shannon`.
+
+## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15)
+
+The reference run (`phyloskims_sal_vac`-scale bacterial test set,
+`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites
+(4.8 MB) — trivially small. The in-progress plant index is expected to
+carry on the order of 9 billion minorant families; a dense byte-per-cell
+alignment at that column count is unbuildable regardless of genome count
+(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000–
+8,000 genomes on a large machine, which makes the per-cell cost dominant in
+the other dimension too. Pruning the retained family set before
+materializing anything is mandatory, not an optimization.
+
+**Already free**: `family_size() < 2` (no sibling variant registered at
+all) is a zero-cost structural filter, read directly off `FamilyMask` bits,
+already applied in `snp_pseudo_alignment`. Insufficient alone — per the
+~80-85% mono-family estimate from earlier discussion, this only brings 9
+billion down to roughly 1.3-1.8 billion, still unusable.
+
+**Entropy definition — settled 2026-08-15, correcting an earlier wrong
+turn.** The project does **not** encode families as IUPAC ambiguity codes
+interpreted the classical way (Fitch-parsimony subset-compatibility, or
+ML's "one true state, uncertain which"); see
+`docmd/theory/evolutionary_distances.md` ("Why the IUPAC/DNA encoding used
+for the first `--snp` test was wrong") and the Sankoff resolution that
+followed it. The real model is a genuine 16-state alphabet (the powerset of
+`{A,C,G,T}`, `∅` included as a real state) scored with a *calibrated
+pairwise cost matrix* (`obikphylo::cardcomp::pairwise_cost_matrix`,
+`cmd/phylo/sankoff.rs`), not a compatibility/subset relation between
+states. Under that model, each of the 16 states — including multi-bit ones
+like `AC` — is a first-class, independently-costed state, not an
+uncertainty encoding of a single true base. So: **entropy over the 15
+non-empty states (`∅` excluded, matching the earlier decision to exclude
+genomes where the family is absent) is the correct informativeness
+measure** for this project — not a 4-symbol reduction, which would discard
+exactly the cardinality/composition information the calibrated cost matrix
+is built to exploit.
+
+## `--subsample` / `--shannon` — sampling strategy (decided 2026-08-15)
+
+Goal: make both the pseudo-alignment (`--snp`) and a Shannon-entropy
+diagnostic usable at any index scale, from the 13-genome bacterial
+reference run up to the 9-billion-family plant index, without requiring the
+batched-accumulator redesign above.
+
+**`--subsample N`** (integer, families to retain): bounds the pseudo-alignment
+to `N` minorant families, sampled **proportionally per layer** among
+non-monomorphic minorants (`family_size >= 2`) — this sidesteps the need
+for a true global reservoir merge across layers while still approximating a
+uniform sample over the whole index, and directly answers the earlier open
+question of global-vs-per-layer selection scope.
+
+Three passes, in order:
+
+1. **Global count** (cheap, structural, parallel across layers — same shape
+   as the existing `sibling_family_size_histogram`, extended to report a
+   **per-layer** breakdown rather than one index-wide aggregate): for each
+   layer, `count_layer` = number of non-monomorphic minorants. Gives
+   `total_count = Σ count_layer`.
+2. **Per-layer proportional reservoir sampling** (cheap, structural, one
+   pass per layer, no cross-partition resolution): `N_layer = round(N ×
+   count_layer / total_count)`. Since `N_layer` is a proportion of
+   `count_layer`, it can never exceed it as long as `N <= total_count` — the
+   one edge case is `total_count <= N`, in which case sampling is skipped
+   entirely and *every* non-monomorphic minorant of every layer is kept
+   (no reservoir needed, `N` was never a real constraint). Otherwise:
+   Algorithm-R reservoir sampling over the layer's non-monomorphic minorant
+   indices, producing `N_layer` iteration-order indices directly, no
+   intermediate full list ever materialized.
+3. **Filtered resolution** (the expensive step, the existing
+   `scan_layer_families` engine, unchanged): re-scan the layer, generating
+   and resolving cross-partition queries **only** for the indices selected
+   in step 2 (cheap membership test against a small per-layer index set) —
+   this is what keeps `--subsample` cheap even on an unsampled-scale index,
+   since the cross-partition resolution volume is bounded by `N`, not by
+   the layer's true size.
+
+Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir
+sampling would waste step-3's expensive resolution work on candidates later
+evicted by the reservoir. Step 1 must fully complete (every layer) before
+step 2 can start for any layer, since `total_count` is a global quantity.
+
+**`--shannon`** (no argument): emits a CSV of per-family Shannon entropy
+(15 non-empty states, `∅`/absent genomes excluded from the denominator, per
+the settled definition above). Independent of `--subsample` — entropy is
+computed and written per family as soon as its `genome_mask` resolves,
+O(1) memory per family, so it streams fine even unsampled at full index
+scale (a time cost, not a memory one). Combined with `--subsample N`, it
+delivers the original exploratory diagnostic (e.g. `--subsample 1000000
+--shannon`) directly from this general machinery, rather than a
+purpose-built one-off script.
+
+Validated end-to-end (2026-08-15) against real data: `--sibling-hist` on
+`phyloskims_sal_vac` (91 real genomes, k=31/m=11, 256 partitions × 2
+layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238
+total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier
+guess, corrected here). `--subsample`/`--shannon` on the smaller 20-genome
+bacterial reference (`benchmark/global_index_presence`) produced a sample
+size within rounding of the request (99,742/100,000) and a
+[0.8,1.2)-bucket share (40.7%) matching the full unsampled population
+(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a
+real bug in `reservoir_sample_layer` (see next section), now fixed.
+
+**Bug found and fixed (2026-08-15): `family_idx` numbering mismatch.**
+`scan_layer_families`'s `family_idx` counts *every minorant* of a layer
+(monomorphic ones included, since `iter_minorants_batch` filters only on
+`is_minorant()`), not the raw annex slot (`SiblingAnnex::get(slot)` spans
+every k-mer, minorant or not) and not a counter over non-monomorphic
+minorants alone. `subsample.rs`'s `reservoir_sample_layer` originally
+stored raw slot numbers in its `HashSet<usize>` selection, which drifts
+away from `family_idx` as soon as *any* monomorphic minorant is seen —
+i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by
+tracking two separate counters: `family_idx` (every minorant, matching
+`scan_layer_families`) and `seen` (non-monomorphic minorants only, what
+Algorithm R actually samples over) — only `family_idx` values are ever
+stored in the selection set. The existing unit test never exercised this
+(its fixture has exactly one non-monomorphic family, always hitting the
+"keep everything" shortcut) — a stronger fixture with several interleaved
+monomorphic/non-monomorphic families would be needed to catch a regression
+here automatically; not yet written.
+
+## Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15)
+
+Motivation: on real data (bacterial reference, full unsampled run), only
+~5‰ of non-monomorphic minorants fall in the `[0.5, 1.5]` bit band judged
+phylogenetically interesting (entropy too low = uninformative near-invariant
+site; too high = saturated/noisy, see `family_entropy`'s 15-state
+discussion) — roughly 1 in 10,000 minorants overall. Computing exact
+entropy for every candidate just to discard 99.99% of them is wasteful at
+the full 9-billion scale.
+
+**The idea**: `PersistentBitMatrix::col_view(c)` gives a genome's whole
+presence column as a `BitSliceView` (sequential, no MPHF, no cross-partition
+routing — purely local to one layer's own matrix). Accumulating
+`TempCompactIntVecBuilder::inc_present(col)` (already exists in
+`obicompactvec/src/builder.rs:121`, along with `add`/`min`/`max`/`diff` on
+`IntSliceView` — no new low-level API needed) over every column of a layer
+produces `coverage[slot]`: how many genomes carry each exact k-mer, in one
+sequential per-layer pass, entirely decoupled from family/sibling
+structure. Persisted once per layer, a family's members' coverage could
+then be looked up via a plain local MPHF `index()` (cheap) instead of a
+full cross-partition presence resolution (`find_presence_batch`) — the
+expensive part today is specifically the cross-partition/cross-layer
+routing to a sibling's own matrix, not the bit-reading itself, and
+`coverage[slot]` sidesteps that routing entirely by moving the cost into a
+one-time, purely local, embarrassingly-parallel build step.
+
+**Why not implemented**: `coverage[slot]` is a per-member marginal —
+summing members' coverages to approximate a family's entropy silently
+assumes no genome carries more than one member at once. It cannot
+represent or detect joint co-occurrence (a genome carrying both `A` and
+`C` at once, i.e. a combined 15-symbol state) at all, which is exactly the
+phenomenon `family_entropy`'s 15-state definition exists to capture (see
+`CardinalityTally`/`cardinality_transition_probs`, the project's own
+existing machinery for this same co-occurrence structure, built for the
+Sankoff matrix calibration). A family that is in truth uniformly `AC`
+across every carrying genome would look like a well-balanced 2-state split
+under the marginal approximation (entropy ≈ 1) while its true 15-state
+entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the
+"saturated, uninformative" tail this pre-filter would need to catch,
+undermining the point. It stays plausible as a coarse filter for the
+*low* tail only (a dominant single member's marginal share reliably
+predicts low true entropy too), but not as a stand-in for the high tail —
+not pursued further for now.
+
+## `--free-loss`/`--tnt` pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below)
+
+Measured on `phyloskims_sal_vac` (91 genomes): `obikmer phylo --subsample 500000
+--free-loss --tnt` logs four sequential stages —
+`raw_snp_distance` (1413s), `base_pair_tally` (1456s),
+`cardinality_tally` (2078s), `snp_pseudo_alignment` (143s). Reading the
+code (`obikmer/src/cmd/phylo/mod.rs:210-242`,
+`obikphylo/src/siblings/distance.rs`, `cardinality.rs`, `alignment.rs`)
+surfaced two compounding problems, not one:
+
+1. **Four separate full scans of the annex**, each opening its own
+   `KmerPartition`/`PartitionCache` and calling `scan_family_pairs`/
+   `scan_layer_families` independently — nothing computed in one stage is
+   reused by another. `base_pair_tally` is explicitly documented
+   (`distance.rs:138-143`) as a second full pass over the same
+   traversal `raw_snp_distance` already did, needed only because
+   `raw_snp_distance` doesn't keep the resolved bases, only aggregate
+   counts. `cardinality_tally` and `snp_pseudo_alignment` are each a
+   third and fourth independent full pass. Per the module's own earlier
+   profiling note (`family_scan.rs:26-28`, cited already above), this
+   traversal is page-fault/mmap-bound, not compute-bound — the ~10-12%
+   CPU efficiency ("contention" status) observed on these three slow
+   stages is consistent with I/O stalls scaled by repeated full scans,
+   not lock contention (there are no `Mutex`/`RwLock` anywhere in
+   `siblings/*.rs`; shared writes use per-slot `AtomicU8::fetch_or`).
+2. **Worse: `raw_snp_distance` and `cardinality_tally` don't honor
+   `--subsample` at all** — both call `scan_layer_families` with
+   `Selection::All` hardcoded, and `args.subsample` isn't even threaded
+   into their function signatures (`mod.rs:212`, `mod.rs:226`). Only
+   `snp_pseudo_alignment(args.subsample)` builds a real reservoir-sampled
+   `Selection::Some(set)` (via `compute_selections`, `alignment.rs:89`).
+   So today, `--subsample 500000` only bounds the pseudo-alignment step —
+   the SNP-distance matrix, the Sankoff base-pair calibration, and the
+   cardinality histogram are always computed over the **full, unsampled**
+   index regardless of the flag. This is not merely "different subsamples
+   per stage" (which would already be a problem worth fixing) — it's that
+   three of the four stages never subsample, which explains most of the
+   ~10x runtime gap against `snp_pseudo_alignment` on its own.
+
+**Decided requirement**: all four stages consume **one shared selection**,
+computed once, not each stage either scanning everything or drawing its
+own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair
+calibration, the cardinality histogram, and the pseudo-alignment all
+describe the same set of families — the calibration and the alignment it
+calibrates are now guaranteed to agree on which sites exist.
+
+**Correction to the "single pass" framing above**: `base_pair_tally`/
+`cardinality_tally` both need `raw_snp_distance`'s *complete* aggregate
+SNP/shared counts before they can derive `included[i,j]` (the
+`ratio_ceiling` filter) — a genuine sequential dependency (`included`
+can't be known until every pair's aggregate count is final), not an
+artifact of the old code's structure. So the fix is **two** passes over
+the shared selection, not one: pass A computes the aggregate counts (and
+derives `included`); pass B fuses `base_pair_tally` + `cardinality_tally`
++ the pseudo-alignment (mutually independent once `included` is known)
+into a single scan. Still a 4→2 reduction, and — per the clarification
+that settled this — pass A itself now runs over the *same shared
+selection* pass B uses (not the full unsampled index): "les ratios, on
+les fait sur les sites sélectionnés, c'est tout, les autres sites
+n'existent pas" — once a selection is chosen, both passes are bounded by
+it, so on a real `--subsample`/`--entropy` run pass A is cheap too, not
+just pass B.
+
+**Entropy-biased selection's own resolution to the "forward-looking
+complication"** (entropy must be known before selection, but selection
+happens during the same scan that would resolve it): see "Entropy-biased
+selection" below — resolved via a persisted per-layer entropy annex, not
+by restructuring the scan into an inline pre-pass.
+
+## Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15)
+
+Refines the "forward-looking complication" above with a concrete
+mechanism. Instead of a hard `[low, high]` entropy band (or any other
+exact cutoff) deciding which non-monomorphic minorant families are
+eligible, selection is weighted by an **unnormalized Gaussian kernel**
+centered on a target entropy: `w(entropy) = exp(-(entropy - μ)² / (2σ²))`
+— deliberately not the normalized Gaussian density (which would peak
+below 1 and complicate the "probability" reading) — this kernel form
+equals 1 exactly at `entropy = μ` and decays smoothly to 0 away from it,
+so it reads directly as an acceptance weight: the further a family's
+entropy from the target, the less likely it is picked, with no hard
+in/out boundary — a few "bad" sites can still get in, by design. `μ`
+(default ~1.0) and `σ` (default ~0.5) are meant to be user-tunable.
+
+**Mechanism: joint probability, not a weighted reservoir**. Not
+Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex
+proposal, superseded before implementation) — instead, a single
+independent accept/reject draw per qualifying candidate: draw
+`u ~ Uniform(0,1)`, accept iff `u < p₀ · w(entropy)`. `p₀` is a single
+**index-wide** rate, `N / total_count` (`total_count` = the sum of
+`non_monomorphic_counts` across every layer, `N` = `--subsample`'s
+target), applied identically at every layer — this alone already gives
+each layer its proportional share (the same effect the old uniform
+reservoir's explicit per-layer `n_layer = N · count_layer / total_count`
+computation achieved, but without needing to compute it: applying one
+rate uniformly is mathematically the same as proportioning per layer).
+`p₀ = 1.0` when there is no `--subsample` at all — `--entropy` alone is a
+pure soft entropy filter over the whole index, no size target. Properties:
+(1) **strictly generalizes the existing uniform sampler** — with `σ` large
+enough that `w ≈ 1` everywhere, this reduces to the old uniform `N/total_count`
+draw; (2) yields "approximately N", not exactly N — expected accepted
+count is `N · mean(w)`, always `≤ N` — an intentional relaxation, matching
+"sous-échantillonnage à environ n" rather than the old reservoir's exact-N
+guarantee; (3) one streaming pass, one random draw per candidate, no
+reservoir state.
+
+**Resolving "entropy must be known before selection, but selection
+happens during the same resolving scan"**: solved with a **persisted,
+per-layer entropy annex** (`obikphylo/src/siblings/entropy_annex.rs`,
+`EntropyAnnex`/`EntropyAnnexBuilder`), not by restructuring the scan.
+Mirrors `SiblingAnnex`'s mmap-backed, read-only-after-build convention,
+but indexed by `family_idx` (every minorant of the layer, monomorphic
+included — the same numbering `Selection`/`scan_layer_families` already
+use), one `f32` entropy15 value per entry, `-1.0` sentinel for monomorphic/
+not-yet-computed. First use of `--entropy`/`--entropy-sd` on an index
+pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`: a full,
+unsampled `Selection::All` scan, resolving every non-monomorphic
+minorant's `genome_mask` once to compute and persist its entropy) — every
+later run (any `μ`/`σ`, any command) reads the file positionally, no
+re-scan, restoring the usual `Selection::Some` "skip resolving excluded
+families" speedup that a naive "weigh during the resolving scan" design
+would have permanently forfeited.
+
+**Resolved**: the existing hard "non-monomorphic minorant" eligibility
+filter stays a hard gate upstream of the Gaussian weighting — only
+qualifying families ever get a stored entropy value or a weighted draw.
+
+**CLI, implemented**: two `phylo` options, `--entropy <μ>` and
+`--entropy-sd <σ>` (`obikmer/src/cmd/phylo/args.rs`). The entropic filter
+activates as soon as *either* is given (`mod.rs`, computed once into an
+`Option<EntropyBias>` threaded through `--snp`/`--family-overlap`/
+`--shannon`/the fused sankoff pipeline below). If active but one or both
+are unset, defaults are `μ = 1.0`, `σ = 0.5`.
+
+## Two entropy definitions kept side by side, for comparison (2026-08-15)
+
+`--shannon`'s CSV carries both `entropy15` (`family_entropy` — the settled
+15-non-empty-state definition, see above) and `entropy4` (`family_entropy_4`
+— plain nucleotide reduction), computed from the same already-resolved
+`genome_mask`, not from the marginal approximation above. A genome carrying
+several bases at once contributes to *each* base's count (counted once per
+base present, not fractionally split, not folded into one combined state)
+— a genome polymorphic for the family is present at more than one base by
+construction, so it is expected to count more than once; the denominator is
+the total base-occurrence count, not the genome count (the two coincide
+only when no genome carries more than one base). Kept side by side
+specifically to measure, on real data, how much the two diverge — not yet
+analyzed.
+
+## `PersistentSparseBitMatrix` — implemented and measured (2026-08-15)
+
+A row-major (k-mer-major), deduplicated sparse alternative to
+`obicompactvec::PersistentBitMatrix`, motivated by the same sparsity that
+drove `--subsample`/`--shannon` above, but pursued as a foundational
+storage-layer change rather than an index-level workaround. Full design
+history, rationale, and rejected alternatives (external Elias-Fano crates,
+`cacheline-ef`, a single unsplit `dict_id` array) are in the dedicated
+implementation plan (`vivid-mapping-tiger.md` at the time of writing — the
+content below is the durable summary, not a pointer to a session-scoped
+file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi,
+"Compact Data Structures for Collections of Sets" (2025,
+`biblio/Alanko et al. - Compact Data Structures for Collections of
+Sets.pdf`) — this design implements only their exact-duplicate special
+case (a plain dedup dictionary), not their full subset-containment
+hierarchy.
+
+**Design**: four on-disk components, each mmap-backed, built once per
+layer (matching how the rest of the build pipeline already works — never
+the whole multi-billion-row index at once): an `is_multi` rank-capable
+flag per row (singleton vs. multi-genome), a fixed-bit-width array for
+singleton rows (genome index directly, `ceil(log2(n_cols))` bits), a
+separate fixed-bit-width array for multi-genome rows (`dict_id`,
+`ceil(log2(n_distinct_multi_sets))` bits — kept apart from the singleton
+array specifically because `n_distinct_multi_sets` can be large in
+absolute terms even when multi-genome rows are a small *fraction* of all
+rows, and a single shared array would force every row, singletons
+included, to pay the wider width), and a deduplicated dictionary of
+distinct multi-genome sets (Elias-Fano-encoded byte offsets + a
+varint-encoded values blob). New low-level primitives added to
+`obicompactvec` to build this: `PersistentFixedIntVec` (arbitrary,
+runtime-parameterized bit width, width 0 included — needed once a real
+bug surfaced, see below), `PersistentRankSelectBitVec` (rank1/rank0/select1
+on top of the crate's existing `count_ones`, using
+`common_traits::SelectInWord`), `EliasFano` (composes the two). A new
+`BinaryMatrix` trait (`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`,
+`count_ones`) unifies dense and sparse at the one call site that needs
+both interchangeably (`obikphylo::siblings::cache::Mat`) — column-oriented
+methods (`col`, `col_view`, the `partial_*_dist_matrix` family) stay
+dense-only.
+
+**Two real bugs caught by tests, not by inspection**: (1) `EliasFano::open`
+re-derived its low-bits width from the persisted low-vector file's own
+width byte; the zero-width case was built with a dummy 1-bit placeholder
+(the builder rejected true width 0), so every reopened value silently
+doubled. Fixed by making `PersistentFixedIntVec` genuinely support width 0
+(no storage, `get` always 0) instead of working around the limitation in
+`EliasFano`. (2) An empty row (cardinality 0 — not expected on a real
+built index, but not guarded against either) was recorded as a singleton
+at genome 0, indistinguishable on read-back from a *real* singleton at
+genome 0. Fixed by routing cardinality-0 rows through the dictionary path
+(a genuine empty entry) instead of the singleton shortcut. Both caught by
+`obicompactvec`'s test suite (142 tests, including disk-reopen round-trips
+that drop every builder/mmap before reopening fresh), not by manual
+review — worth remembering next time a "this edge case can't happen in
+practice" shortcut is tempting.
+
+**Measured on real data** (`layer_1` of `phyloskims_sal_vac`'s
+`part_00018`, 30,246,774 rows, 91 genomes — `#[ignore]`d benchmarks in
+`obikphylo/src/siblings/tests.rs`):
+
+| | dense | sparse | ratio |
+|---|---|---|---|
+| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
+| build time / peak RSS | — | 4.26s / 628MB | (per-layer, in-memory construction — comfortable) |
+| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse **faster** (smaller structure, better cache fit) |
+| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** (the real `--shannon`/family-lookup shape) |
+| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** (no native column method — every read decodes a full row to keep one bit) |
+
+The row-access wins (both directions) weren't the design's stated goal —
+compactness was — but turn out real: dense's genome-major layout scatters
+a single row read across a much bigger file, which costs more than
+sparse's rank/select/varint decode once the file is this much smaller.
+The column-access cost is the flip side of the same layout choice, and is
+exactly what the next item below exists to fix.
+
+**Next, not yet planned**: rewrite `partial_jaccard_dist_matrix`/
+`partial_hamming_dist_matrix`/etc. (`obicompactvec/src/bitmatrix/pairwise.rs`)
+as a row-major co-occurrence accumulation (`O(Σ_rows k²)`, per-row
+increments into an `NxN` genome-pair counter — the known alternative to
+today's column-fold, plausibly cheaper on data this sparse, not just a
+fallback) so `obikindex`'s `--metric`/distance-matrix path can use the
+sparse type without the measured 86x column-access penalty. Needs its own
+design pass (in particular how it plugs into the `BitPartials`/
+`ColumnWeights` traits so both matrix types keep serving `--metric`)
+before implementation — not just "port the loop", a genuinely different
+algorithm.
+
+Also still deferred, unchanged from the implementation plan: full Alanko
+et al. subset-hierarchy compression (only the exact-duplicate special case
+is built), a sparse `PersistentCompactIntMatrix` (count matrices), and
+BRWT-style column-correlation exploitation.
+
+## Wired into `pack` and the sibling-annex build path (2026-08-15)
+
+`PersistentSparseBitMatrix` went from a validated but unused type to a
+real, selectable on-disk format:
+
+- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
+  (`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
+  `D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
+  `PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
+  `LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
+  already generic over any `D: LayerData`, so they needed no change.
+  Verified by `obilayeredmap`'s
+  `presence_layer_generic_over_sparse_matches_dense` test: build a dense
+  presence layer, convert it to sparse via `build_from_dense`, open both
+  as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
+  the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree.
+  (This test must stay at `k=4` with mutually non-colliding canonical
+  4-mers across its input sequences — `K`/`M` are process-wide
+  `AtomicUsize`s in test builds, not thread-local, so a test using a
+  different `k` races every other test in the same crate binary; a k=11
+  version of this test passed alone but failed under the full
+  `obilayeredmap` suite for exactly that reason before being fixed.)
+- **`obikphylo::siblings::cache::Mat`** gained a third variant,
+  `SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
+  and `Presence` — every method (`find_slot`, `index_batch`,
+  `iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches
+  to it identically to `Presence`, since both go through the same generic
+  `Layer<D>` code. `PartitionCache::build` picks the variant per layer by
+  checking for `presence/is_multi.prsb` (the sparse format's own marker
+  file, see the design section above) before falling back to the dense
+  open path.
+- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`):
+  `pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb`
+  first); packs to dense `matrix.pbmx` first if that hasn't happened yet
+  (the dense→sparse transpose needs random row access, which only the
+  packed/columnar dense forms give), then `build_from_dense`s the sparse
+  form into the same directory and deletes `matrix.pbmx` — old-format
+  files are removed only after the new format is fully written, mirroring
+  `pack_bit_matrix`'s own crash-safety convention.
+- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through
+  `KmerIndex::pack_matrices` (all other call sites — `select`, `merge`,
+  `finalize_indexed` — pass `false`, unchanged dense behaviour). Count
+  matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix`
+  — see "still deferred" above).
+- **End-to-end coverage**: `obikphylo::siblings::tests::
+  sibling_annex_works_after_pack_sparse` builds a two-genome index, packs
+  it `--sparse`, asserts `is_multi.prsb` exists, then runs
+  `build_sibling_annex` and checks the resulting `FamilyMask`s match the
+  dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
+  sparse format round-trips through the real build pipeline
+  (`PartitionCache` sparse-detection included), not just the
+  `obicompactvec`/`obilayeredmap` unit layers below it.
+
+Full workspace `cargo test` (all crates, unit + doc tests) green after
+this change.
+
+## Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15)
+
+Replaces the "four independent scans" problem above and implements
+"Entropy-biased selection" above, end to end:
+
+- **`SankoffBundleExt::sankoff_bundle`** (new,
+  `obikphylo/src/siblings/sankoff_bundle.rs`) — the `--sankoff`/`--tnt`/
+  `--phyg`/`--iqtree` block in `obikmer/src/cmd/phylo/mod.rs` now calls
+  this once instead of three separate `raw_snp_distance`/
+  `base_pair_tally`/`cardinality_tally` calls. One `PartitionCache`, one
+  shared (possibly subsampled/entropy-biased) selection computed once via
+  `compute_selections`, two scans over it: pass A (aggregate SNP/shared
+  counts, `--exclude-genome` zeroing, then `included[i,j]`), pass B
+  (`base_pair_tally` + `cardinality_tally` + the pseudo-alignment, fused
+  into one `scan_layer_families` call per layer, all three read off the
+  same resolved `genome_mask`). `snp_pseudo_alignment`/
+  `shannon_entropy_csv` (still used standalone by `--snp`/
+  `--family-overlap`/`--shannon`) both gained an `entropy_bias` parameter
+  too, so entropy-biased selection isn't sankoff-specific.
+- **Regression proof**: `sankoff_bundle_matches_old_separate_calls`
+  (`obikphylo/src/siblings/tests.rs`) asserts `sankoff_bundle`'s four
+  outputs are bit-identical to calling the old, separate
+  `raw_snp_distance`/`base_pair_tally`/`cardinality_tally`/
+  `snp_pseudo_alignment` on the same fixture with no subsample — the
+  fusion is a performance change, not a behavior change.
+- **`EntropyAnnex`/`EntropyAnnexBuilder`** (new,
+  `obikphylo/src/siblings/entropy_annex.rs`) and **`ensure_entropy_annexes`**
+  (`entropy.rs`) implement the persisted-entropy mechanism from
+  "Entropy-biased selection" above. `entropy_annex_builds_on_demand_and_biases_selection`
+  (`tests.rs`) proves, on a fixture with one known-entropy family: the
+  annex file doesn't exist before any entropy-biased call; `compute_selections`
+  builds it on first use; `μ` set to the family's exact entropy with
+  `p₀ = 1.0` selects it deterministically (`u < 1.0` always, for
+  `u ∈ [0,1)`); `μ` set far away with tiny `σ` deterministically excludes
+  it (`w` underflows to exactly `0.0`); the persisted value matches
+  `--shannon`'s own `family_entropy` computation to `1e-6`.
+- **`EntropyBias`** (`pub`, `obikphylo::siblings::EntropyBias { mu, sigma }`)
+  is the one new public type threading `--entropy`/`--entropy-sd` through
+  every `Option<EntropyBias>`-accepting method — resolved once in
+  `obikmer/src/cmd/phylo/mod.rs` from `args.entropy`/`args.entropy_sd`
+  (activation: either given; defaults `1.0`/`0.5` for whichever is unset).
+
+Full workspace `cargo test` green after this change (167 unit tests in
+`obicompactvec`+`obilayeredmap`+`obikphylo` alone, plus every other
+crate's suite, no regressions).
+
+**Still open, not part of this change** (per "Correction to the 'single
+pass' framing" above): `--raw-snp-distance`/`--raw-snp-counts` (the
+standalone diagnostic flags, not the `--sankoff` pipeline) still always
+scan the full unsampled index — never threaded `--subsample`/`--entropy`,
+out of scope here since the reported problem was specifically about the
+`--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these
+two standalone flags.

DevDocMD/css/extra.css

@@ -0,0 +1,35 @@
+/* docs/css/extra.css */
+
+/* Styles principaux pour le conteneur et le texte */
+.ps-root {
+  font-family: "Courier New", monospace;
+  font-size: 0.9em;
+  line-height: 1.5;
+}
+
+/* Styles pour les mots-clés */
+.ps-keyword {
+  font-weight: bold;
+  color: #d73a49; /* Une belle teinte de rouge */
+}
+
+/* --- CORRECTION DE L'INDENTATION --- */
+/* Cible tous les niveaux d'indentation et applique une marge gauche */
+[class*="ps-indent-"] {
+  display: inline-block;
+}
+.ps-indent-1 {
+  margin-left: 2em;
+}
+.ps-indent-2 {
+  margin-left: 4em;
+}
+.ps-indent-3 {
+  margin-left: 6em;
+}
+.ps-indent-4 {
+  margin-left: 8em;
+}
+.ps-indent-5 {
+  margin-left: 10em;
+}

DevDocMD/ecology-letters.csl

@@ -0,0 +1,230 @@
+<?xml version="1.0" encoding="utf-8"?>
+<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" demote-non-dropping-particle="sort-only" default-locale="en-US">
+  <info>
+    <title>Ecology Letters</title>
+    <id>http://www.zotero.org/styles/ecology-letters</id>
+    <link href="http://www.zotero.org/styles/ecology-letters" rel="self"/>
+    <link href="http://www.zotero.org/styles/apa" rel="template"/>
+    <link href="http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291461-0248/homepage/ForAuthors.html" rel="documentation"/>
+    <author>
+      <name>David Kaplan</name>
+      <email>david.kaplan@ird.fr</email>
+    </author>
+    <contributor>
+      <name>Sebastian Karcher</name>
+    </contributor>
+    <category citation-format="author-date"/>
+    <category field="biology"/>
+    <issn>1461-023X</issn>
+    <eissn>1461-0248</eissn>
+    <updated>2023-10-11T10:45:32+00:00</updated>
+    <rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
+  </info>
+  <macro name="container">
+    <choose>
+      <if type="chapter paper-conference" match="any">
+        <text term="in" text-case="capitalize-first" suffix=": "/>
+        <text variable="container-title" font-style="italic"/>
+        <text variable="collection-title" prefix=", "/>
+        <names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
+          <label form="short" suffix=" "/>
+          <name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
+        </names>
+      </if>
+      <else>
+        <group delimiter=", ">
+          <text variable="container-title" font-style="italic" form="short"/>
+          <text variable="collection-title"/>
+        </group>
+      </else>
+    </choose>
+  </macro>
+  <macro name="author">
+    <names variable="author">
+      <name name-as-sort-order="all" and="symbol" sort-separator=", " initialize-with="." delimiter=", " delimiter-precedes-last="never"/>
+      <label form="short" prefix=" (" suffix=")" text-case="capitalize-first"/>
+      <et-al font-style="italic"/>
+      <substitute>
+        <names variable="editor"/>
+        <names variable="translator"/>
+        <text macro="title"/>
+      </substitute>
+    </names>
+  </macro>
+  <macro name="author-short">
+    <names variable="author">
+      <name form="short" and="symbol" delimiter=", " initialize-with=". "/>
+      <et-al font-style="italic"/>
+      <substitute>
+        <names variable="editor"/>
+        <names variable="translator"/>
+        <choose>
+          <if type="bill book graphic legal_case legislation motion_picture report song" match="any">
+            <text variable="title" form="short" font-style="italic"/>
+          </if>
+          <else>
+            <text variable="title" form="short" quotes="true"/>
+          </else>
+        </choose>
+      </substitute>
+    </names>
+  </macro>
+  <macro name="access">
+    <choose>
+      <if type="webpage">
+        <group>
+          <text term="available at" text-case="capitalize-first" suffix=": "/>
+          <text variable="URL" suffix="."/>
+        </group>
+        <text value="Last accessed" prefix=" " suffix=" "/>
+        <date variable="accessed">
+          <date-part name="day" suffix=" "/>
+          <date-part name="month" suffix=" "/>
+          <date-part name="year"/>
+        </date>
+      </if>
+    </choose>
+  </macro>
+  <macro name="title">
+    <choose>
+      <if type="report" match="any">
+        <text variable="title" font-style="italic"/>
+        <group prefix=" (" suffix=")">
+          <text variable="genre"/>
+          <text variable="number" prefix=" No. "/>
+        </group>
+      </if>
+      <else-if type="bill book graphic legal_case legislation motion_picture report song speech" match="any">
+        <text variable="title" font-style="italic"/>
+      </else-if>
+      <else-if type="webpage">
+        <text variable="title" font-style="italic"/>
+      </else-if>
+      <else>
+        <text variable="title"/>
+      </else>
+    </choose>
+  </macro>
+  <macro name="publisher">
+    <choose>
+      <if type="report" match="any">
+        <group delimiter=", ">
+          <text variable="publisher"/>
+          <text variable="publisher-place"/>
+        </group>
+      </if>
+      <else>
+        <text variable="genre" suffix=". "/>
+        <group delimiter=", ">
+          <text variable="publisher"/>
+          <text variable="publisher-place"/>
+        </group>
+      </else>
+    </choose>
+  </macro>
+  <macro name="event">
+    <choose>
+      <if variable="event">
+        <text term="presented at" text-case="capitalize-first" suffix=" "/>
+        <text variable="event"/>
+      </if>
+    </choose>
+  </macro>
+  <macro name="issued">
+    <choose>
+      <if variable="issued">
+        <date variable="issued">
+          <date-part name="year"/>
+        </date>
+      </if>
+      <else-if variable="accessed">
+        <choose>
+          <if type="webpage">
+            <date variable="accessed">
+              <date-part name="year"/>
+            </date>
+          </if>
+          <else>
+            <text term="no date" form="short"/>
+          </else>
+        </choose>
+      </else-if>
+      <else>
+        <text term="no date" form="short"/>
+      </else>
+    </choose>
+  </macro>
+  <macro name="edition">
+    <choose>
+      <if is-numeric="edition">
+        <group delimiter=" ">
+          <number variable="edition" form="ordinal"/>
+          <text value="edn"/>
+        </group>
+      </if>
+      <else>
+        <text variable="edition" suffix="."/>
+      </else>
+    </choose>
+  </macro>
+  <macro name="locators">
+    <choose>
+      <if type="article-journal article-magazine article-newspaper" match="any">
+        <group prefix=", " delimiter=", ">
+          <group>
+            <text variable="volume"/>
+          </group>
+          <text variable="page"/>
+        </group>
+      </if>
+      <else-if type="bill book graphic legal_case legislation motion_picture report song thesis" match="any">
+        <group delimiter=". " prefix=". ">
+          <text macro="edition"/>
+          <text macro="event"/>
+          <text macro="publisher"/>
+        </group>
+      </else-if>
+      <else-if type="chapter paper-conference" match="any">
+        <group delimiter=", " prefix=". ">
+          <text macro="event"/>
+          <text macro="publisher"/>
+          <group>
+            <label variable="page" form="short" suffix=" "/>
+            <text variable="page"/>
+          </group>
+        </group>
+      </else-if>
+    </choose>
+  </macro>
+  <citation et-al-min="3" et-al-use-first="1" disambiguate-add-year-suffix="true" collapse="year-suffix" year-suffix-delimiter=", ">
+    <sort>
+      <key macro="author"/>
+      <key macro="issued"/>
+    </sort>
+    <layout prefix="(" suffix=")" delimiter="; ">
+      <group delimiter=" ">
+        <text macro="author-short"/>
+        <text macro="issued"/>
+      </group>
+    </layout>
+  </citation>
+  <bibliography et-al-min="7" et-al-use-first="6" entry-spacing="0" hanging-indent="true">
+    <sort>
+      <key macro="author"/>
+      <key macro="issued" sort="ascending"/>
+      <key macro="title"/>
+    </sort>
+    <layout>
+      <group suffix=".">
+        <text macro="author" suffix="."/>
+        <text macro="issued" prefix=" (" suffix="). "/>
+        <group delimiter=". ">
+          <text macro="title"/>
+          <text macro="container"/>
+        </group>
+        <text macro="locators"/>
+        <text macro="access" prefix=". "/>
+      </group>
+    </layout>
+  </bibliography>
+</style>

DevDocMD/implementation/chunkreader.md

@@ -0,0 +1,125 @@
+# Chunk reader — implementation
+
+`obiread` exposes two distinct sequence reading paths, each optimised for a different use case.
+
+## Two reading paths
+
+| Path | API | Output unit | Per-record identity | Use case |
+|------|-----|-------------|---------------------|----------|
+| **Record path** | `read_sequence_chunks` → `parse_chunk` | `SeqRecord` (id + raw sequence + normalised rope) | yes | `query` — must read complete records |
+| **Stream path** | `open_nuc_stream` | `NucPage` (flat normalised byte buffer) | no | `index`, `superkmer` — bulk throughput |
+
+The record path uses `Rope`-backed chunks and is described in detail below.
+The stream path (`NucStream` / `NucPage`) is described in the scatter section of [pipeline](pipeline.md).
+
+---
+
+## Record path: chunk reader
+
+The chunk reader reads FASTA or FASTQ files in fixed-size blocks and yields self-contained chunks, each ending on a complete sequence record boundary. `parse_chunk` then converts each chunk into a `Vec<SeqRecord>`, where each record carries its identifier, raw sequence bytes, and a normalised rope ready for superkmer building.
+
+This path is mandatory for `query`, where superkmers must be tracked back to their originating sequence (id, kmer offset) for output annotation.
+
+## Output type: Rope
+
+Each chunk is a `Rope` — a segmented byte sequence: a `Vec` of blocks, where each block is a `Vec<Cell<u8>>`. The consumer iterates over the blocks via a forward or backward cursor.
+
+`Rope::split_off(pos)` splits at an absolute byte offset in O(log n) (binary search over block-start index). If `pos` falls inside a block, that block is split in two via `Vec::split_off` — no `memcpy` in the common case.
+
+## SeqChunkIter
+
+```rust
+pub struct SeqChunkIter<R: Read> { /* private */ }
+
+impl<R: Read> Iterator for SeqChunkIter<R> {
+    type Item = io::Result<Rope>;
+}
+
+pub fn fasta_chunks<R: Read>(source: R) -> SeqChunkIter<R>
+pub fn fastq_chunks<R: Read>(source: R) -> SeqChunkIter<R>
+```
+
+`next()` loop:
+
+```text
+1. read one block of block_size bytes → push onto Rope
+2. call splitter(rope) → Option<abs_offset>
+   if Some(pos):
+       tail = rope.split_off(pos)    ← O(log n), may split one block
+       chunk = mem::replace(&mut rope, tail)
+       return Some(Ok(chunk))
+3. if EOF and rope non-empty: return Some(Ok(rope)) as final chunk
+4. if EOF and rope empty: return None
+```
+
+The `Splitter` function signature is `fn(&Rope) -> Option<usize>`. It returns the absolute byte offset of the start of the last complete record, or `None` if no boundary was found in the accumulated rope (need more data).
+
+## Boundary detection — FASTA
+
+Backward scan with a 2-state machine. Searches (right to left) for `>` followed by `\n` or `\r` (i.e., a `>` that is preceded by a newline in forward order):
+
+```mermaid
+stateDiagram-v2
+    direction LR
+    [*]      --> Scanning
+    Scanning --> FoundGt  : '>'
+    FoundGt  --> Scanning : other
+    FoundGt  --> [*]      : '\\n' / '\\r' ✓
+```
+
+Returns the byte offset of the `>` that starts the last complete record. Returns `None` if only one `>` is found (cannot confirm there is a prior complete record).
+
+## Boundary detection — FASTQ
+
+FASTQ records have a rigid 4-line structure (`@header`, sequence, `+`, quality). The `@` character (ASCII 64, Phred score 31) can appear legitimately in quality lines, making any forward heuristic unreliable. The backward scanner verifies the full structural context before accepting a candidate `@`.
+
+7-state machine (states 0–6), scanning from **right to left**. Each time a `+` is found, its position is saved as `restart`; any state mismatch resets the scan to that position.
+
+```mermaid
+stateDiagram-v2
+    direction LR
+
+    [*]          --> Scanning
+
+    Scanning     --> FoundPlus    : '+' (save restart)
+    FoundPlus    --> AfterNlPlus  : '\\n' / '\\r'
+    FoundPlus    --> Scanning     : other → backtrack
+
+    AfterNlPlus  --> AfterNlPlus  : séparateur
+    AfterNlPlus  --> InSequence   : lettre / - / . / [ / ]
+    AfterNlPlus  --> Scanning     : other → backtrack
+
+    InSequence   --> AfterSequence : '\\n' / '\\r'
+    InSequence   --> InSequence    : lettre / - / . / [ / ]
+    InSequence   --> Scanning      : other → backtrack
+
+    AfterSequence --> AfterSequence : '\\n' / '\\r'
+    AfterSequence --> InHeader      : other
+
+    InHeader     --> FoundAt    : '@' (save cut)
+    InHeader     --> Scanning   : '\\n' / '\\r' → backtrack
+    InHeader     --> InHeader   : other
+
+    FoundAt      --> [*]       : '\\n' / '\\r' ✓
+    FoundAt      --> InHeader  : other
+```
+
+`restart` is updated each time a `+` is found. When any state fails its expected input, the scan jumps back to `restart` and continues from there — guaranteeing that a `@` in a quality line cannot be accepted as a record start, because the `\n+\n` structure immediately following it (going backward) will not be found.
+
+Returns the byte offset of the `@` that starts the last complete record.
+
+---
+
+## Future work — parallel gzip decompression in `xopen`
+
+`obiread::xopen` (`xopen.rs`) decompresses gzip via `niffler` → `flate2`, which is single-threaded (standard DEFLATE has no parallel-decodable structure). For large local gzip inputs this single-threaded decompression can become the throughput bottleneck feeding the `query`/`index`/`superkmer` pipelines, since chunk/page production for a given file is serialized ahead of the worker pool.
+
+Candidate: special-case local, on-disk, gzip-magic-detected paths in `open_raw`/`xopen` to use [`rapidgzip-rs`](https://github.com/alekseizarubin/rapidgzip-rs) (`ReaderBuilder::new().parallelism(n).open(path)`, implements `Read + Seek`) instead of `niffler`, keeping `niffler` for every other case: `stdin` (`-`), HTTP(S) sources, and all non-gzip formats (bzip2, xz, zstd — less used in practice here).
+
+Constraints identified so far (not yet validated against real data):
+- Branch point must move earlier than the current `decompress()` call in `open_raw` — rapidgzip's fast path needs the file **path**, not an already-opened generic `Read`, so the gzip/local-file detection has to happen before the generic `File::open` + `niffler::send::get_reader` path is taken.
+- `stdin` and HTTP sources are not seekable — they stay on `niffler` regardless; the gain only applies to local on-disk `.gz` files.
+- `rapidgzip-sys` vendors a native C++ engine: requires CMake ≥ 3.17, a C++17 compiler, and `nasm` on x86 targets — a real build-toolchain addition, not just a pure-Rust crate.
+- Low maturity of the Rust binding at review time (2 GitHub stars, ~15 commits, April 2026 latest release) — the underlying C++ engine is validated (HPDC 2023 paper), but the binding itself has limited production track record.
+
+Decision: parked for now. Before adopting, validate on real data: throughput vs. `niffler` on representative large `.gz` inputs, and byte-for-byte correctness of decompressed output.

DevDocMD/implementation/chunkreader.refs.md

@@ -0,0 +1,12 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/chunkreader.md
+
+## Code couvert
+
+- `obiread/src/chunk.rs` — SeqChunkIter, détection de frontières FASTA/FASTQ, state machines
+- `obikrope/src/lib.rs` — type Rope (Vec<Bytes>), opérations zero-copy
+
+## Notes
+
+Document stable (la stratégie de chunking rope ne devrait pas avoir changé).
+Vérifier que le split FASTA/FASTQ reste correct si de nouveaux formats ont été ajoutés.

DevDocMD/implementation/evidence_elimination.md

@@ -0,0 +1,181 @@
+# Approximate evidence: fingerprint-based index
+
+## Motivation
+
+`evidence.bin` maps each MPHF slot to the position of the k-mer that owns it,
+enabling zero-FP verification. On the bacterial BCT dataset (2048 partitions,
+k=31, ~33 M k-mers/partition) it accounts for 66 % of the lookup-layer footprint:
+
+| file | size/partition | fraction |
+|---|---|---|
+| evidence.bin | 132 MB | 66 % |
+| unitigs.bin | 58 MB | 29 % |
+| mphf.bin | 10 MB | 5 % |
+
+`evidence.bin` is a bijection from MPHF-space to unitig-position-space and
+costs at minimum ⌈log₂ N⌉ bits per slot — an information-theoretic floor with
+only ~22 % packing headroom. Compression is not a path to elimination.
+
+The approximate index replaces `evidence.bin` + `unitigs.bin.idx` with a
+`fingerprint.bin` file. The MPHF and `unitigs.bin` are kept unchanged. Set
+operations still require an exact index; the approximate index targets query
+workloads that can tolerate a bounded false-positive rate.
+
+---
+
+## The Findere model
+
+A B-bit fingerprint stored per MPHF slot provides the discrimination that
+`evidence.bin` would otherwise provide through full k-mer reconstruction.
+
+For a foreign k-mer query, the MPHF maps it to some slot `s`. The fingerprint
+stored at `s` belongs to the legitimate k-mer at that slot. The FP event is:
+
+```
+P(FP per k-mer) = 1 / 2^b
+```
+
+The Findere trick reduces the indexed k-mer size. When the user specifies k_user
+and z, the index physically stores k-mers of size `s = k_user − z + 1`. At query
+time, the same s-mer size is used. After collecting per-position s-mer results
+over the full query sequence, a sliding window of size z aggregates z consecutive
+s-mer hits into one confirmed k_user-mer hit, reducing the per-window FP rate:
+
+```
+P(FP per k_user-mer) = 1 / 2^(b·z)
+```
+
+`IndexConfig::kmer_size` stores `s = k_user − z + 1`, not k_user. Both indexing
+and querying use this stored size via `set_k(idx.kmer_size())`.
+
+Parameters b and z are stored in `layer_meta.json` (`EvidenceKind::Approx { b, z }`).
+
+---
+
+## `FingerprintVec` on disk
+
+`fingerprint.bin` layout:
+
+```
+magic:   b"FPVF"  (4 bytes)
+b:       u8       (bits per slot, 1..=64)
+padding: [0u8; 3]
+n:       u64 LE   (number of slots)
+data:    packed bits, ceil(n·b/8) bytes, Lsb0 order
+```
+
+`FingerprintVec` is memory-mapped. The match check against a query k-mer:
+
+```rust
+fn matches(&self, slot: usize, fingerprint: u64) -> bool {
+    self.get(slot) == (fingerprint & self.mask)
+}
+```
+
+`build_approx_evidence` iterates `unitigs.bin` sequentially, writes
+`kmer.seq_hash()` into the slot assigned by the MPHF, then saves `fingerprint.bin`
+and `layer_meta.json`. No `.idx` file is produced; random access into
+`unitigs.bin` is not needed.
+
+At build time, `find_approx` in `MphfLayer`:
+
+```rust
+let slot = self.mphf.index(&kmer.raw());
+if fingerprint.matches(slot, kmer.seq_hash()) { Some(slot) } else { None }
+```
+
+---
+
+## `EvidenceKind` and metadata
+
+`layer_meta.json` records which evidence bundle is present:
+
+```rust
+pub enum EvidenceKind {
+    Exact,
+    Approx { b: u8, z: u8 },
+}
+```
+
+`MphfLayer::open` reads this tag and dispatches `find` to `find_exact` or
+`find_approx` transparently. `find_exact` panics on an approximate layer;
+`find_approx` panics on an exact layer — mode mixing is a programming error.
+
+---
+
+## Parameter resolution (`resolve_approx_params`)
+
+The identity `b·z = ⌈−log₂(fp)⌉` lets any two of (b, z, fp) derive the third.
+`resolve_approx_params` implements a 2-of-3 rule with conservative ceiling
+rounding:
+
+| given | derived |
+|---|---|
+| b, z | fp = 1/2^(b·z) |
+| z, fp | b = ⌈−log₂(fp) / z⌉ |
+| b, fp | z = ⌈−log₂(fp) / b⌉ |
+| z only | b = 8 (default), fp derived |
+| b only | z = 1 (default), fp derived |
+| fp only | b = 8 (default), z derived |
+| none | b = 8, z = 1, fp = 1/256 |
+
+When all three are given, b and z are authoritative and fp is recomputed.
+
+---
+
+## CLI flags
+
+Both `index` and `reindex` accept the same flags:
+
+| flag | type | meaning |
+|---|---|---|
+| `--approx` | bool | enable fingerprint evidence |
+| `--evidence-bits` (`b`) | u8 | fingerprint bits per slot |
+| `-z` | u8 | Findere z parameter |
+| `--fp` | f64 | target FP rate per z-window |
+| `--block-size` | usize | unitig block size for exact `.idx`; ignored in approx mode |
+
+`--approx` must be set explicitly; the other three flags are optional and
+resolved by the 2-of-3 rule. Omitting all three produces b=8, z=1.
+
+---
+
+## `reindex` command
+
+`reindex` converts an existing index between exact and approximate evidence
+in-place across all partitions and layers, running partitions in parallel via
+Rayon.
+
+Conversion to approximate (`--approx`):
+
+- Builds `fingerprint.bin` from `unitigs.bin` + `mphf.bin`.
+- Removes `evidence.bin` and `unitigs.bin.idx`.
+- Updates `layer_meta.json` with `EvidenceKind::Approx { b, z }`.
+
+Conversion to exact (default, no `--approx`):
+
+- Builds `evidence.bin` + `unitigs.bin.idx` from `unitigs.bin` + `mphf.bin`.
+- Removes `fingerprint.bin`.
+- Updates `layer_meta.json` with `EvidenceKind::Exact`.
+
+The root `index.meta` is updated with the new evidence kind on success.
+`mphf.bin` and `unitigs.bin` are never modified.
+
+---
+
+## `estimate` command
+
+`estimate` is a dry-run that resolves and prints (b, z, fp) without touching
+any index. It accepts the same `--evidence-bits`, `-z`, and `--fp` flags and
+additionally accepts `-k` to display the effective indexed k-mer length:
+
+```
+k (user):              31
+k (indexed, s=k-z+1):  27
+z:                     5
+evidence bits (b):     8
+FP per s-mer:          3.906e-3  (1/2^8)
+FP per k-mer window:   9.537e-7  (1/2^(8·5))
+```
+
+Useful for choosing parameters before committing to an index build.

DevDocMD/implementation/evidence_elimination.refs.md

@@ -0,0 +1,22 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/evidence_elimination.md
+
+## Code couvert
+
+- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
+- `obilayeredmap/src/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
+- `obilayeredmap/src/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
+- `obikindex/src/reindex.rs` — KmerIndex::reindex(), conversion exact↔approx en place
+- `obikmer/src/cmd/reindex.rs` — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size
+- `obikmer/src/cmd/index.rs` — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp
+- `obikmer/src/cmd/estimate.rs` — commande estimate (dry-run des paramètres)
+
+## Notes
+
+Ce document était à l'origine une discussion de design (4 approches). L'implémentation
+a maintenant convergé vers l'approche fingerprint (Findere-style).
+FORT RISQUE DE DÉRIVE — le contenu est probablement un mélange de design et d'implémentation :
+- Le modèle FP = 1/2^(b·z) et les règles de résolution (2-of-3 parmi b, z, fp) sont implémentés
+- La commande `reindex` permet la conversion a posteriori exact↔approx
+- La commande `estimate` fait le dry-run des paramètres
+Cette page doit être réécrite pour documenter l'implémentation Findere réelle plutôt que les alternatives abandonnées.

DevDocMD/implementation/filtering.md

@@ -0,0 +1,320 @@
+# Kmer filtering and ingroup/outgroup predicates
+
+The `filter`, `dump`, and `unitig` commands share the same filtering system,
+implemented as a shared `FilterArgs` clap argument group embedded in each command
+via `#[command(flatten)]`. Filters select k-mers based on per-genome quorum
+counts, optionally restricted to **ingroup** and **outgroup** genome sets derived
+from genome metadata. All rules described here apply identically to all three commands.
+
+`filter` additionally accepts `--min-total-count` / `--max-total-count` filters
+that operate on the sum of counts across all genomes.
+
+## Predicate syntax
+
+Each `--ingroup` and `--outgroup` flag takes a predicate of the form:
+
+```
+key OP value1|value2|…
+```
+
+| Operator | Meaning |
+|----------|---------|
+| `*` or `all` | wildcard — every genome matches unconditionally |
+| `key=v1\|v2` | exact match — genome's `key` equals `v1` or `v2` |
+| `key!=v` | negation — genome's `key` equals none of the values |
+| `key~path` | path ancestry — genome's `key` is `path` or a descendant |
+| `key!~path` | not a descendant |
+
+Multiple values separated by `|` are always OR-ed within the predicate.
+
+### Path matching (`~` and `!~`)
+
+Metadata values can represent hierarchical concept paths such as
+`/Eukaryota/Viridiplantae/Streptophyta/Betulaceae/Betula/nana`.
+
+Stored taxonomy values always start with `/` (the root of the path).
+Query patterns do **not** need to start with `/` — a leading `/` is an optional
+start anchor, not a requirement.
+
+| Pattern form | Semantics |
+|---|---|
+| `A/B` | contiguous sub-path A then B, anywhere in the value |
+| `/A/B` | value starts with A then B |
+| `A/B$` | value ends with A then B |
+| `/A/B$` | value is exactly A then B |
+| `A@x/B` | A with class `x` followed by B with any class |
+
+- `taxon~/Betulaceae/Betula` matches any path that starts with `Betulaceae` then `Betula`.
+- `taxon~Betula` matches any path containing `Betula` as a segment, anywhere.
+
+### Missing metadata key → NA
+
+If a genome does not carry the queried metadata key, the predicate returns **NA**.
+NA propagates through the group evaluation logic (see below), and genomes that
+cannot be classified are **ignored** in all quorum counts.
+
+## Group semantics
+
+### Multiple predicates
+
+| Flag | Combination rule |
+|------|-----------------|
+| `--ingroup` (repeated) | **AND** — genome must satisfy all predicates |
+| `--outgroup` (repeated) | **OR** — genome satisfies any predicate |
+
+### Three-value logic
+
+Each predicate returns `true`, `false`, or `NA` (absent key).
+
+- AND: `false` absorbs everything; `NA` propagates unless already `false`.
+- OR: `true` absorbs everything; `NA` propagates unless already `true`.
+
+### Classification and priority
+
+For each genome:
+
+1. Evaluate `AND(ingroup predicates)` → `in_result`
+2. Evaluate `OR(outgroup predicates)` → `out_result`
+3. If `in_result = true` → **Ingroup** (ingroup wins over outgroup)
+4. Else if `out_result = true` → **Outgroup**
+5. Otherwise → **Uncategorized** (ignored in all quorum counts)
+
+### Implicit groups
+
+| `--ingroup` | `--outgroup` | Effective behaviour |
+|-------------|--------------|---------------------|
+| not set | not set | all genomes form the ingroup |
+| set | not set | only ingroup quorum flags apply |
+| not set | set | only outgroup quorum flags apply |
+| set | set | both constraints apply simultaneously |
+
+## Quorum flags
+
+| Flag | Applies to | Meaning |
+|------|-----------|---------|
+| `--min-count N` | ingroup | k-mer present in at least N ingroup genomes (N may be negative, see below) |
+| `--max-count N` | ingroup | k-mer present in at most N ingroup genomes (N may be negative, see below) |
+| `--min-frac F` | ingroup | k-mer present in at least fraction F of ingroup genomes |
+| `--max-frac F` | ingroup | k-mer present in at most fraction F of ingroup genomes |
+| `--min-outgroup-count N` | outgroup | k-mer present in at least N outgroup genomes (N may be negative, see below) |
+| `--max-outgroup-count N` | outgroup | k-mer present in at most N outgroup genomes (N may be negative, see below) |
+| `--min-outgroup-frac F` | outgroup | k-mer present in at least fraction F of outgroup genomes |
+| `--max-outgroup-frac F` | outgroup | k-mer present in at most fraction F of outgroup genomes |
+| `--min-total-count N` | all genomes | sum of per-genome counts ≥ N (`filter` only) |
+| `--max-total-count N` | all genomes | sum of per-genome counts ≤ N (`filter` only) |
+| `--presence-threshold N` | all | per-genome count > N to be considered "present" (default 0) |
+
+### Negative counts — offset from group size
+
+The four integer count flags (`--min-count`, `--max-count`, `--min-outgroup-count`,
+`--max-outgroup-count`) accept **negative** values, interpreted as an offset counted
+down from the group size `n`, resolved at run time once `n` is known:
+
+| Value | Effective threshold |
+|-------|---------------------|
+| `N ≥ 0` | literal absolute count `N` |
+| `-x` (x > 0) | `max(1, n − x)` — "all but x" |
+
+`-1` literally means *all but one*, `-2` *all but two*, and so on. This expresses
+a quorum relative to the group size that a plain fraction cannot state exactly
+(e.g. "present in every genome except at most one" is `n−1`, which is `0.9` for
+`n = 10` but `0.857…` for `n = 7`).
+
+The threshold is **floored at 1**, never 0: the negative form always keeps
+constraining the group. Without the floor, `--min-count -1` on a singleton
+ingroup (`n = 1`) would resolve to `0` ("at least 0") and silently drop the
+constraint; the floor makes it `1` ("present in that one genome") instead.
+
+To express a count of `0` (e.g. "absent from the ingroup"), use the literal `0`,
+not a negative — `0` and `-0` are indistinguishable, so the offset form starts at
+`-1`.
+
+> **Edge case** — on an *empty* group (`n = 0`, e.g. a predicate matching no
+> genome), a negative count still resolves to `1`, an impossible constraint that
+> rejects every k-mer. This is consistent with an empty group letting nothing
+> through, but differs from the "no constraint" behaviour of the fraction flags.
+
+**Conditional defaults** — the defaults for `--min-frac` and `--max-outgroup-count` depend on two conditions:
+whether the corresponding group was declared, **and** whether any quorum flag for that group was explicitly set.
+
+> **Rule**: declaring a group activates the smart default **only if no quorum flag for that group is explicitly set**.
+> As soon as any quorum flag for a group is present on the command line, all defaults for that group revert to no-op values.
+
+| `--ingroup` | Any ingroup quorum flag? | `--min-frac` default |
+|-------------|--------------------------|----------------------|
+| not set | — | 0.0 (no-op) |
+| set | no | **1.0** — all ingroup genomes must carry the k-mer |
+| set | yes | 0.0 — user controls quorum explicitly |
+
+| `--outgroup` | Any outgroup quorum flag? | `--max-outgroup-count` default |
+|--------------|---------------------------|-------------------------------|
+| not set | — | outgroup size (no-op) |
+| set | no | **0** — no outgroup genome may carry the k-mer |
+| set | yes | outgroup size — user controls quorum explicitly |
+
+"Any ingroup quorum flag" means any of: `--min-count`, `--max-count`, `--min-frac`, `--max-frac`.  
+"Any outgroup quorum flag" means any of: `--min-outgroup-count`, `--max-outgroup-count`, `--min-outgroup-frac`, `--max-outgroup-frac`.
+
+**Why this rule?** Setting any quorum flag signals explicit intent — the defaults are there to help when the user omits quorum entirely, not to interfere with deliberate constraints. Mixing implicit and explicit quorum on the same group would risk silent incoherence (e.g. `--max-count 0` with an implicit `--min-frac 1.0`).
+
+All other bounds default to 0 / group size / 0.0 / 1.0 regardless of whether groups are declared.
+
+### Validation
+
+After resolving defaults, the following are checked and cause an immediate error:
+
+| Condition | Error |
+|-----------|-------|
+| `--min-count > --max-count` | incoherent bounds |
+| `--min-frac > --max-frac` | incoherent bounds |
+| `--min-outgroup-count > --max-outgroup-count` | incoherent bounds |
+| `--min-outgroup-frac > --max-outgroup-frac` | incoherent bounds |
+| any fraction outside `[0.0, 1.0]` | invalid value |
+
+The check applies to the **effective** values (after defaults are resolved), so an explicit `--max-frac 0.5` with an implicit `--min-frac 1.0` would have been caught — but the rule above prevents that situation from arising in the first place.
+
+Fractions are computed over the size of the classified group, not over total
+genome count. An empty group (no genome classified as ingroup/outgroup) never
+triggers a filter failure.
+
+### Conservative rounding of fraction thresholds
+
+When a fraction threshold `F` is applied to a group of size `N`, the effective
+integer threshold is determined by the direction of the bound:
+
+| Bound | Effective count | Rounding | Rationale |
+|-------|----------------|----------|-----------|
+| `--min-frac F` | k-mer in ≥ ⌈F·N⌉ genomes | **ceil** | stricter — a kmer present in exactly ⌊F·N⌋ genomes does not meet the fraction |
+| `--max-frac F` | k-mer in ≤ ⌊F·N⌋ genomes | **floor** | stricter — a kmer present in ⌈F·N⌉ genomes already exceeds the fraction |
+
+The same rule applies symmetrically to `--min-outgroup-frac` (ceil) and
+`--max-outgroup-frac` (floor). The outgroup direction is not inverted: the
+conservative rounding depends only on whether the bound is a minimum or a
+maximum, not on which group it applies to.
+
+**Example** — `--min-frac 0.5` with an ingroup of 3 genomes:
+`⌈0.5 × 3⌉ = ⌈1.5⌉ = 2` → at least 2 of 3 ingroup genomes must carry the k-mer.
+
+**Implementation note** — the filter evaluates `n / denom < min_frac` directly
+(integer `n`, float comparison) rather than pre-computing `⌈F·N⌉`. This is
+mathematically equivalent for integer counts: `n / N < F` ↔ `n < F·N` ↔
+`n ≤ ⌈F·N⌉ − 1` ↔ `n < ⌈F·N⌉`. No explicit rounding is needed.
+
+## Examples
+
+Keep k-mers specific to *Betula nana* — present in at least 2 *B. nana* genomes
+and absent from every other genome in the index:
+
+```sh
+obikmer filter src --output dst \
+  --ingroup  "species=Betula_nana" \
+  --outgroup "*" \
+  --min-count 2 \
+  --max-outgroup-count 0
+```
+
+Keep k-mers found in at least 2 *Betula nana* genomes and absent from all
+other *Betula*:
+
+```sh
+obikmer filter src --output dst \
+  --ingroup  "species=Betula_nana" \
+  --outgroup "genus=Betula" \
+  --min-count 2 \
+  --max-outgroup-count 0
+```
+
+Use taxonomic paths — keep k-mers present in ≥ 50 % of the *Betula* clade
+and in fewer than 10 % of everything outside *Betulaceae*:
+
+```sh
+obikmer filter src --output dst \
+  --ingroup  "taxon~/Betulaceae/Betula" \
+  --outgroup "taxon!~/Betulaceae" \
+  --min-frac 0.5 \
+  --max-outgroup-frac 0.1
+```
+
+Multiple outgroup predicates (OR): exclude k-mers present in *Alnus* or *Carpinus*:
+
+```sh
+obikmer filter src --output dst \
+  --ingroup  "genus=Betula" \
+  --outgroup "genus=Alnus" \
+  --outgroup "genus=Carpinus" \
+  --max-outgroup-count 0
+```
+
+Noise-tolerant core — keep k-mers present in *all but one* ingroup genome
+(`-1` = `n−1`) and absent from *all but one* of the outgroup:
+
+```sh
+obikmer filter src --output dst \
+  --ingroup  "genus=Betula" \
+  --outgroup "*" \
+  --min-count -1 \
+  --max-outgroup-count -1
+```
+
+To dump only k-mers specific to *Betula nana*:
+
+```sh
+obikmer dump myindex \
+  --ingroup  "species=Betula_nana" \
+  --outgroup "*" \
+  --min-count 1 \
+  --max-outgroup-count 0
+```
+
+To enumerate unitigs of the *Betula*-specific subgraph:
+
+```sh
+obikmer unitig myindex \
+  --ingroup  "genus=Betula" \
+  --outgroup "*" \
+  --min-count 2 \
+  --max-outgroup-count 0
+```
+
+## Command-specific options
+
+### `dump --head N`
+
+Stops output after the first N k-mers that pass all active filters.
+Iteration terminates immediately — subsequent partitions and layers are not scanned.
+Useful for quick inspection of large indexes without loading the entire dataset.
+
+```sh
+obikmer dump myindex --head 100
+obikmer dump myindex --head 20 --ingroup "species=Betula_nana" --min-count 1
+```
+
+### `phylo --presence-threshold N`
+
+When computing Jaccard distance on a **count index**, a k-mer is considered present in a genome if its count is ≥ N (default 1).
+This option is independent of the `--presence-threshold` used in filtering.
+
+```sh
+# Jaccard treating kmers with count ≥ 2 as present
+obikmer phylo myindex --metric jaccard --presence-threshold 2
+```
+
+This parameter has no effect on presence/absence indexes (where values are already 0/1) or on metrics other than Jaccard.
+
+## Implementation
+
+- **`obikpartitionner::filter::GroupQuorumFilter`** — implements `KmerFilter`
+  using pre-computed ingroup and outgroup index vectors. The heavy logic
+  (predicate parsing, three-value evaluation, genome classification) happens
+  once before any iteration; each k-mer row evaluation is a simple index
+  lookup and counter.
+
+- **`obikmer::cmd::predicate::FilterArgs`** — shared `clap` argument group
+  embedded via `#[command(flatten)]` in `FilterArgs`, `DumpArgs`, and
+  `UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
+  list.
+
+- **`obikpartitionner::KmerPartition::iter_partition_kmers`** — accepts
+  `filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
+  the callback. `filter`, `dump`, and `unitig` all go through this single
+  entry point.

DevDocMD/implementation/kmer.md

@@ -0,0 +1,91 @@
+# Kmer — implementation
+
+## Types and layout
+
+`KmerOf<L>` is a `#[repr(transparent)]` newtype over `u64` parameterized by a `KmerLength` marker:
+
+```rust
+#[repr(transparent)]
+pub struct KmerOf<L: KmerLength>(u64, PhantomData<L>);
+```
+
+Three marker types implement `KmerLength`:
+
+| Marker | `len()` source | Used for |
+|--------|---------------|---------|
+| `KLen` | `params::k()` | k-mers |
+| `MLen` | `params::m()` | minimizers |
+| `ConstLen<N>` | const generic `N` | tests |
+
+Public aliases:
+
+```rust
+pub type Kmer     = KmerOf<KLen>;         // k-mer, global k
+pub type Minimizer = CanonicalKmerOf<MLen>; // canonical m-mer, global m
+```
+
+Nucleotides are packed 2 bits each, **left-aligned**, MSB-first. Nucleotide 0 occupies bits 63–62; nucleotide i occupies bits 63−2i and 62−2i. The low 64−2·len bits are always zero. The length is **not stored** — every operation reads it from `L::len()`.
+
+| 63–62 | 61–60 | … | 63−2(k−1)−1 to 63−2(k−1) | 63−2k down to 0 |
+|-------|-------|---|--------------------------|-----------------|
+| nt 0  | nt 1  | … | nt k−1                   | zero padding    |
+
+## Global parameters
+
+`params::set_k(k)` / `params::k()` and `params::set_m(m)` / `params::m()` are backed by `OnceLock<usize>` in production (write-once, panic on conflict) and by `thread_local! { Cell<usize> }` in test builds (per-thread, freely writable). `params::init(k, m)` sets both in one call.
+
+## Encoding
+
+`KmerOf::<L>::from_ascii(ascii)` encodes the first `L::len()` bytes using the shared `ENC` table (see [SuperKmer — ASCII encoding](superkmer.md#ascii-encoding-and-decoding)):
+
+```rust
+for i in 0..k {
+    val = (val << 2) | encode_base(ascii[i]) as u64;
+}
+KmerOf(val << (64 - 2 * k), PhantomData)
+```
+
+Zero allocation — result lives on the stack.
+
+## Decoding
+
+`write_ascii(writer)` writes k ASCII characters to any `W: Write` using the shared `DEC4` table: one lookup per 4 nucleotides, one partial lookup for the remainder. No allocation in the hot path.
+
+`to_ascii()` is a convenience wrapper that allocates and returns a `Vec<u8>`; intended for tests and display only.
+
+## Reverse complement
+
+Computed as pure arithmetic — no lookup table, no memory access:
+
+```rust
+let x = !self.0;                                                               // complement
+let x = x.swap_bytes();                                                        // reverse bytes
+let x = ((x >> 4) & 0x0F0F0F0F0F0F0F0F) | ((x & 0x0F0F0F0F0F0F0F0F) << 4); // swap nibbles
+let x = ((x >> 2) & 0x3333333333333333) | ((x & 0x3333333333333333) << 2);   // swap 2-bit groups
+KmerOf(x << (64 - 2 * k), PhantomData)
+```
+
+After complementing, bytes are reversed (`swap_bytes`), then nibbles, then 2-bit groups — restoring 2-bit nucleotides to their correct positions in reverse order. A final left-shift realigns to MSB. Zero allocation — result lives on the stack.
+
+## Canonical form and `CanonicalKmerOf`
+
+`canonical()` returns a `CanonicalKmerOf<L>` — a distinct newtype that carries the same `u64` layout but enforces the invariant that the stored value equals `min(kmer, revcomp)`:
+
+```rust
+pub fn canonical(&self) -> CanonicalKmerOf<L> {
+    let rc = self.revcomp();
+    CanonicalKmerOf(if self.0 <= rc.0 { self.0 } else { rc.0 }, PhantomData)
+}
+```
+
+Lexicographic minimum of forward and reverse-complement, comparing the raw `u64` values directly (left-aligned encoding makes this equivalent to nucleotide-wise comparison). Zero allocation — result lives on the stack.
+
+`CanonicalKmerOf::from_raw_unchecked(raw)` is the only other public constructor, for trusted paths such as deserialisation.
+
+## Sliding window helpers
+
+`push_right(nuc)` / `push_left(nuc)` shift the window by one base in O(1). `is_overlapping(other)` checks whether the last k−1 nucleotides of `self` equal the first k−1 of `other`.
+
+## Hashing
+
+`hash_kmer(raw: u64) -> u64` computes `mix64(raw ^ 0x9e3779b97f4a7c15)`, the seeded splitmix64 finalizer. `CanonicalKmerOf::seq_hash()` delegates to `hash_kmer`.

DevDocMD/implementation/kmer.refs.md

@@ -0,0 +1,13 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/kmer.md
+
+## Code couvert
+
+- `obikseq/src/kmer.rs` — layout mémoire (repr(transparent) u64), encodage/décodage, revcomp, forme canonique
+- `obikseq/src/params.rs` — k global (set_k / k())
+
+## Notes
+
+Document d'implémentation stable. L'algorithme de revcomp bit-à-bit est décrit —
+vérifier qu'il correspond à `revcomp_raw` dans `obiskio/src/unitig_index.rs` (copie locale)
+et à l'implémentation dans `obikseq/src/kmer.rs`.

DevDocMD/implementation/merge.md

@@ -0,0 +1,196 @@
+# Merge command
+
+## Purpose
+
+`obikmer merge` combines multiple existing kmer indexes into a single index. The result contains all kmers from all sources, with per-genome presence/absence or count data for every genome across every layer.
+
+---
+
+## Modes
+
+```rust
+pub enum MergeMode { Presence, Count }
+```
+
+Default mode is `Presence`. `Count` mode requires **all** source indexes to have `with_counts=true`; mixing count and non-count sources is rejected at validation.
+
+| Mode | Column type | Constraint |
+|---|---|---|
+| `Presence` | `PersistentBitMatrix` (one bit per genome per slot) | none |
+| `Count` | `PersistentCompactIntMatrix` (one u32 per genome per slot) | all sources `with_counts=true` |
+
+---
+
+## Input / output constraints
+
+All source indexes must satisfy:
+
+- `IndexState::Indexed` (fully built — `index.done` sentinel present)
+- Same `kmer_size`, `minimizer_size`, `n_partitions`
+- Same evidence kind: all `Exact`, or all `Approx` with identical `(b, z)` parameters
+- If `Count` mode: all sources must have `with_counts=true`
+
+`--force`: if the output directory already exists, it is deleted before the merge begins.
+
+---
+
+## Evidence compatibility
+
+`validate_evidence_compat(sources)` is called before any I/O. It compares each source's `EvidenceKind` against `sources[0]`:
+
+- All `Exact` → accepted, output uses `Exact`
+- All `Approx { b, z }` with same `(b, z)` → accepted, output uses those parameters
+- Any other combination → `OKIError::IncompatibleEvidence`, with a message directing the user to run `reindex` first
+
+Mixed exact/approx is a hard error, not a silent conversion.
+
+```rust
+fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
+```
+
+---
+
+## Genome label deduplication
+
+`compute_labels(sources, rename_duplicates)` assigns final genome labels across all sources before any file is written. The first occurrence of a label keeps the original name. Subsequent occurrences receive `.1`, `.2`, … suffixes when `rename_duplicates` is true, or trigger `OKIError::DuplicateGenomeLabel` otherwise.
+
+---
+
+## Algorithm
+
+### 1. Validation
+
+Check all sources against the constraints above. Abort on any mismatch.
+
+### 2. Bootstrap output from first source
+
+Recursive file copy of `sources[0]` → `output`. Immediately after the copy:
+
+- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
+- In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
+- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
+
+This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
+
+### 3. For each subsequent source (parallel across partitions)
+
+`KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)` is called for each partition index `i`. `block_bits` is taken from `dst.meta.config.block_bits`.
+
+Each entry in `sources` is `(&KmerPartition, n_genomes)` where `n_genomes` is the column count that source contributes (> 1 when the source is itself a merged index).
+
+**First merge, Presence mode**: when `n_dst_genomes == 1`, `Layer::<()>::init_presence_matrix` is called on every existing destination layer before any source column is appended. This creates `presence/col_000000.pbiv` set all-true (genome 0 is present in every slot).
+
+**Pass 1 — classify kmers**
+
+Iterate all kmers from all source partitions (via `UnitigFileReader` + canonical kmer iteration). For each kmer, probe the destination `LayeredMap<()>`:
+
+- **Hit**: kmer already in the destination; record for Pass 2.
+- **Miss**: push kmer into a `GraphDeBruijn` accumulator.
+
+**New layer construction**
+
+If the accumulator is non-empty, compute de Bruijn unitigs and call `Layer::<()>::build(&new_layer_dir, block_bits)`. All kmers absent from the destination — across **all** sources — accumulate into a **single** graph, producing one new layer per merge operation (not one per source).
+
+**Pass 2 — fill column builders**
+
+For each source and each of its layers, re-iterate unitigs and look up stored values via `SrcLayerData::lookup(kmer, src_n)`:
+
+- `SrcLayerData::SetMembership` — no data directory exists; every kmer returns `vec![1; n_genomes]`
+- `SrcLayerData::Presence` — reads `PersistentBitMatrix` from `presence/`
+- `SrcLayerData::Count` — reads `PersistentCompactIntMatrix` from `counts/`
+
+Hits are routed to `exist_builders[dst_layer][src_col]`; misses are routed to `new_src_builders[src_col]`.
+
+**Column prepending for new layers**
+
+Before source columns are written to the new layer, `n_dst_genomes` absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.
+
+**Close and update metadata**
+
+Close all builders; update `presence/meta.json` or `counts/meta.json` with `{"n": N, "n_cols": n_dst_genomes + n_src_total}`; increment `PartitionMeta::n_layers` if a new layer was added.
+
+### 4. Update index metadata
+
+`index.meta` was already written during bootstrap with the complete genome list and evidence kind. No further update is needed after the partition loop.
+
+---
+
+## `append_genome_column`
+
+Defined on two concrete specialisations of `Layer<D>`:
+
+```rust
+impl Layer<PersistentCompactIntMatrix> {
+    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> u32) -> OLMResult<()>
+}
+
+impl Layer<PersistentBitMatrix> {
+    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> bool) -> OLMResult<()>
+}
+```
+
+Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written directly via `PersistentBitVecBuilder` / `PersistentCompactIntVecBuilder` rather than through these helpers, but the invariant they enforce is the same.
+
+---
+
+## Column count invariant
+
+After any merge, **every layer in every partition has exactly `n_genomes` columns**, where `n_genomes` is the total genome count in the index at that point.
+
+Maintained by three mechanisms:
+
+1. **Existing layers**: `n_src_total` columns appended (one per source genome).
+2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
+3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
+
+The invariant is a precondition of `LayeredStore` aggregation traits: `col_weights()` and all partial distance methods assume every inner store has the same column count.
+
+---
+
+## Error variants relevant to merge
+
+| Variant | Condition |
+|---|---|
+| `OKIError::NotIndexed(path)` | Source not in `Indexed` state |
+| `OKIError::IncompatibleConfig` | Mismatched `kmer_size`, `minimizer_size`, or `n_partitions` |
+| `OKIError::MismatchedMode` | Count mode but a source has `with_counts=false` |
+| `OKIError::IncompatibleEvidence(msg)` | Mixed exact/approx or different approx `(b, z)` |
+| `OKIError::DuplicateGenomeLabel(label)` | Duplicate label and `rename_duplicates=false` |
+
+---
+
+## On-disk impact
+
+After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest):
+
+```
+partitions/
+  part_00000/
+    index/
+      meta.json              ← n_layers updated if new layer added
+      layer_0/
+        mphf.bin             ← unchanged
+        unitigs.bin          ← unchanged
+        evidence.bin         ← unchanged
+        presence/            ← created on first merge (Presence mode)
+          meta.json            {"n": N, "n_cols": G}
+          col_000000.pbiv      ← all-true (genome 0 … G0-1)
+          col_000001.pbiv      ← next source
+          ...
+        counts/              ← extended (Count mode)
+          meta.json            {"n": N, "n_cols": G}
+          col_000000.pciv      ← genome 0 counts (from original build)
+          col_000001.pciv      ← next source
+          ...
+      layer_N/               ← new layer (if new kmers found)
+        mphf.bin
+        unitigs.bin
+        evidence.bin
+        presence/ or counts/
+          meta.json            {"n": N1, "n_cols": G}
+          col_000000.pbiv      ← all-false (absent for existing genomes)
+          ...
+spectrums/
+  <label>.json               ← one file per genome, rebuilt from all sources
+index.meta                   ← complete genome list + evidence kind written at bootstrap
+```

DevDocMD/implementation/merge.refs.md

@@ -0,0 +1,19 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/merge.md
+
+## Code couvert
+
+- `obikindex/src/merge.rs` — `KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
+- `obikpartitionner/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
+- `obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
+- `obilayeredmap/src/layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
+- `obicompactvec/src/intmatrix.rs` — `append_column` pour PersistentCompactIntMatrix
+- `obicompactvec/src/bitmatrix.rs` — `append_column` pour PersistentBitMatrix
+
+## Notes
+
+FORT RISQUE DE DÉRIVE. Changements récents :
+- Ajout de la validation de compatibilité d'évidence : merge exact+approx → erreur (OKIError::IncompatibleEvidence)
+- `merge_partition` reçoit maintenant `block_bits: u8`
+- La commande `reindex` a été ajoutée comme outil de conversion exact↔approx avant merge
+Vérifier que la doc décrit la politique de merge mixed-evidence et le recours à `reindex`.

DevDocMD/implementation/merge_parallelism.md

@@ -0,0 +1,207 @@
+# Merge parallelism and memory pressure
+
+## Problem observed
+
+Running `obikmer merge` over 109 indexes (108 sources + 1 bootstrap) on a 192-core machine
+produces a fatal OOM during the `merge_partitions` stage:
+
+```
+memory allocation of 9126805520 bytes failed
+```
+
+A single allocation of ~8.5 GB fails. This is not an aggregate; it is one `malloc` call
+from hashbrown during a HashMap resize.
+
+---
+
+## Root cause
+
+### The merge pipeline per partition
+
+```
+source unitigs.bin
+  → iter_indexed_canonical_kmers()
+  → GraphDeBruijn::push()       ← HashSet<u64> + 1 byte flags, all in RAM
+  → compute_degrees_and_mark_starts()
+  → try_for_each_unitig()
+  → unitigs.bin (new layer)
+  → Layer::build() → MPHF + evidence
+```
+
+`GraphDeBruijn` is a `FastHashMap<CanonicalKmer, AtomicU8>` — a `HashSet<u64>` with
+one flag byte per node. Neighbor lookup is implicit: 4 probes into the same map.
+No edges are stored. The full kmer set of one partition must reside in RAM
+simultaneously to compute degrees and mark unitig starts.
+
+The matrix builders that follow (pass 2) are mmapped files — they do **not** consume
+significant RAM. The pressure is entirely in pass 1.
+
+### Unbounded Rayon parallelism
+
+With 192 cores, Rayon ran up to 192 partitions concurrently. Each partition built its
+own `GraphDeBruijn` accumulating all kmers absent from the destination. Peak memory =
+192 × peak_partition_hashset.
+
+### The 8.5 GB single allocation
+
+hashbrown allocates the entire backing array in one call when rehashing.
+At load factor 7/8: `capacity × (sizeof(K,V) + 1 control byte)`.
+For `(u64, AtomicU8)` with alignment: ~16 bytes per slot.
+
+```
+9 127 MB / 16 bytes ≈ 570 M slots → ~380 M new kmers in one partition
+```
+
+Plausible for the largest partition of 108 Salix/Betula sources (~450 Mbp each).
+
+---
+
+## Partition size distribution
+
+`obikmer utils --partition-stats` measures the sum of `unitigs.bin` file sizes
+per partition across all source indexes (pure `stat()` syscalls, negligible cost).
+
+Observed on a 9-genome pilot (256 partitions):
+
+| Stat | Value |
+|---|---|
+| min    | 30.5 MB |
+| max    | 232.1 MB |
+| mean   | 40.1 MB |
+| median | 37.2 MB |
+| p95    | 47.1 MB |
+| max/median ratio | 6.23× |
+
+The distribution is **bimodal with a heavy tail**:
+- 238/256 partitions in a narrow 30–50 MB band
+- 4 structurally extreme partitions (3–6× the median): 221, 233, 135, 191
+
+These correspond to minimizers over-represented in repetitive regions shared across
+all sources. They are extreme in every run on this dataset.
+
+With 109 sources, outlier partitions do not scale linearly: only kmers **absent from
+the destination** enter the GraphDeBruijn, and inter-source overlap is high for closely
+related species. Partition 221 is the likely trigger for the 8.5 GB crash.
+
+---
+
+## Solution: LFD scheduling + memory budget semaphore
+
+### Principle
+
+Pre-sort partitions by **decreasing estimated size** (First Fit Decreasing — FFD),
+then schedule them through a **continuous memory budget semaphore**. Each worker
+acquires an estimated cost before starting and releases it on completion.
+
+Large partitions run first when the full budget is available; small partitions fill
+the gaps. No hard outlier threshold is needed.
+
+### `MemoryBudget` (`obisys`)
+
+```rust
+pub struct MemoryBudget { … }
+
+impl MemoryBudget {
+    pub fn new(total: u64) -> Self;
+    pub fn acquire(&self, cost: u64);  // blocks until budget available
+    pub fn release(&self, cost: u64);
+    pub fn peak_active(&self) -> usize;
+}
+```
+
+Non-deadlock guarantee: when `active == 0`, acquire always succeeds regardless of cost.
+Without this, a partition whose estimated cost exceeds the total budget would block forever.
+
+### Adaptive expansion factor
+
+The expansion factor converts raw `unitigs.bin` bytes into an estimated GraphDeBruijn
+RAM footprint. hashbrown stores each kmer as `(u64, AtomicU8)` ≈ 16 bytes/kmer at 7/8
+load factor; unitig files encode ≈ 2 bits/base. The ratio depends on average unitig
+length (short unitigs: ~2×; long unitigs: up to ~50×).
+
+**Phase 1 — sequential pilot (worst partition)**
+
+The largest partition runs alone first. Its actual `g.len()` seeds the expansion factor
+before any parallel job starts. `FALLBACK_EXPANSION = 4×` is used only for empty partitions.
+
+```rust
+let worst_g_len = dst_partition.merge_partition(worst_id, …)?;
+//                              ↑ now returns SKResult<usize> (was SKResult<()>)
+
+let seed_expansion = worst_g_len as u64 * 16 * 1000 / worst_bytes;
+let max_expansion = AtomicU64::new(seed_expansion);
+```
+
+**Phase 2 — parallel with adaptive updates**
+
+```rust
+order[1..].into_par_iter().for_each(|&i| {
+    let cost = partition_sizes[i] * max_expansion.load(Relaxed) / 1000;
+    budget.acquire(cost);
+    let g_len = dst_partition.merge_partition(i, …)?;
+    budget.release(cost);          // releases estimated cost, not actual
+
+    let actual = g_len as u64 * 16 * 1000 / partition_sizes[i];
+    max_expansion.fetch_max(actual, Relaxed);  // always pessimistic (max)
+});
+```
+
+`budget.release(cost)` uses the estimated cost, not the actual one. The budget tracks
+reservations, not physical RAM; each partition pays what it promised at acquisition.
+
+**On the safety margin**
+
+There is no separate multiplier `k`. It is redundant with `budget_fraction`: both
+reduce effective concurrency by the same amount. A single parameter is easier to
+calibrate. `budget_fraction = 0.5` (default) reserves half of available RAM for the
+OS, MPHF build, pass 2, and estimation error.
+
+`--budget-fraction` is exposed as a CLI flag — the only escape hatch for pathological
+cases (extreme repetitive content, unusually long unitigs) that still cause OOM.
+
+### RAM source
+
+`obisys::available_memory_bytes()` — wraps `sysinfo::System::available_memory()`,
+falls back to `total / 2` on macOS when the memory compressor returns 0.
+
+---
+
+## Diagnostic report
+
+After the parallel phase, `merge_partition` emits a structured report via `tracing::info!`:
+
+```
+─── merge_partitions memory report ───
+  available RAM : 512.0 GB   budget 50% = 256.0 GB
+  expansion factor — seed: 4.2×  final max: 6.1×  (mean: 1.8×  median: 1.6×)
+  peak concurrent workers: 42
+  expansion factor distribution (256 partitions with data):
+     0.50× –  1.25× │██████████████████████████████ 148
+     1.25× –  2.00× │████████████████████████        82
+     …
+     5.50× –  6.25× │█                                2
+  top partitions by actual expansion factor:
+    partition  221 : 6.10×  (232.1 MB unitigs → 48M kmers, reserved at 4.20×)
+    partition  135 : 5.82×  (127.3 MB unitigs → 24M kmers, reserved at 4.20×)
+    …
+──────────────────────────────────────
+```
+
+Fields useful for diagnosis:
+
+| Field | Interpretation |
+|---|---|
+| `seed` vs `final max` expansion | gap indicates partitions with higher expansion than the worst-by-size |
+| `reserved at X×` | the factor used at acquisition; if much lower than actual, the budget was under-reserved for that partition |
+| `peak concurrent workers` | effective parallelism achieved under the budget constraint |
+| `mean` / `median` expansion | typical dataset characteristic; stable across runs on the same data |
+
+---
+
+## Parameters
+
+| Parameter | Default | CLI flag | Notes |
+|---|---|---|---|
+| `fallback_expansion` | 4× | — | seed for empty partitions only |
+| `budget_fraction` | 0.5 | `--budget-fraction` | reduce if OOM persists |
+| RAM source | `obisys::available_memory_bytes()` | — | falls back to `total/2` on macOS |

DevDocMD/implementation/mphf.md

@@ -0,0 +1,177 @@
+# MPHF selection — two-phase indexing architecture
+
+## Why two phases are needed
+
+Kmer indexing per partition proceeds in two phases. The separation is necessary because the exact number of surviving unique kmers is not known until after counting and filtering low-abundance kmers.
+
+### Phase 1 — provisional MPHF + kmer spectrum
+
+Implemented in `obikpartitionner::KmerPartition::count_kmer()` → `count_partition()`.
+
+1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
+2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
+3. **Create `counts1.bin`**: `PersistentCompactIntVec` with f0 slots, zero-initialised.
+4. **Accumulation pass**: re-read the dereplicated superkmer file; for each kmer in each superkmer, compute `slot = mphf.index(kmer.raw())` and increment `counts1[slot]` by the superkmer's COUNT.
+5. **Build kmer frequency spectrum** from `counts1`: histogram `{count → n_kmers}`, totals f0 (distinct kmers) and f1 (total abundance). Written to `kmer_spectrum_raw.json` per partition, then merged globally.
+
+Files produced per partition:
+
+```
+part_XXXXX/
+  mphf1.bin               — ptr_hash provisional MPHF (discarded after phase 2)
+  counts1.bin             — PersistentCompactIntVec, f0 × u32 kmer counts
+  kmer_spectrum_raw.json  — local frequency spectrum
+```
+
+### Phase 2 — definitive MPHF
+
+After filtering (applying a min-count threshold derived from the spectrum) and building the local De Bruijn graph + unitigs (see [Construction pipeline](pipeline.md)), the exact filtered kmer set is available via `unitigs.bin`.
+
+`MphfLayer::build(dir, block_bits, mode: &IndexMode, fill_slot)` is called on the unitig directory:
+
+1. **Pass 1** (parallel): a `CanonicalKmerIter` — clonable via `Arc<Mmap>`, no file reopening — is passed directly to `new_from_par_iter` via `par_bridge()`. No `.idx` is read or created at this stage; parallelism is at partition/layer level, not within a single MPHF. Produces `mphf.bin`.
+2. **Pass 2** (sequential): iterate with `iter_indexed_canonical_kmers`; fill evidence files; call `fill_slot(slot, kmer)` callback per kmer. For Exact/Hybrid, `.idx` is written at the end of this pass — never earlier.
+
+`mphf1.bin` and `counts1.bin` are no longer needed after phase 2 and can be deleted.
+
+---
+
+## MPHF candidates
+
+**boomphf** (BBHash algorithm, maintained by 10X Genomics):
+
+- ~3.7 bits/key; mature crate, used in production bioinformatics (Pufferfish, Piscem)
+- Supports streaming construction (no exact count needed)
+- Drawback: largest space footprint; streaming advantage is irrelevant at phase 2 since the exact count is available
+
+**ptr_hash** (PtrHash algorithm, Groot Koerkamp, SEA 2025):
+
+- ~2.4 bits/key; fastest queries (≥2.1× over alternatives, 8–12 ns/key for u64) and fastest construction (≥3.1×)
+- Requires exact key count at construction — available at both phases after pass 1
+- Published February 2025; accepted given performance profile and the fact that each MPHF is independently rebuildable from its unitig file
+
+**FMPH/FMPHGO** (`ph` crate, Beling, ACM JEA 2023):
+
+- ~2.1 bits/key — most compact; good query speed; deterministic construction
+- `GOFunction` (group-oriented variant) was the original phase-1 choice; eliminated when the external sort made the exact count available at phase 1 as well
+
+## MPHF choice per phase
+
+**Both phases**: **ptr_hash**, same type alias and construction parameters. The external sort (phase 1) and the unitig index (phase 2) both provide the exact key count before MPHF construction, so ptr_hash's requirement is satisfied in both cases. Using a single MPHF implementation removes the `ph` crate dependency.
+
+boomphf: eliminated — largest space overhead, streaming advantage no longer needed. FMPH/GOFunction: eliminated — exact count available, ptr_hash is faster at equivalent compactness.
+
+---
+
+## Space at scale
+
+For 1 024 partitions × 100 M kmers/partition (phase 2 index, after filtering):
+
+| MPHF     | bits/key | Total MPHF size |
+|----------|----------|-----------------|
+| boomphf  | 3.7      | ~47 GB          |
+| ptr_hash | 2.4      | ~31 GB          |
+| FMPH     | 2.1      | ~27 GB          |
+
+For a human genome at 30× coverage with 1 024 partitions, realistic partition sizes are 3–30 M unique kmers → 1–8 MB per phase-2 MPHF, well within RAM.
+
+---
+
+## ptr_hash configuration (phase 2)
+
+```rust
+type Mphf = PtrHash<
+    u64,                              // key: canonical kmer raw encoding
+    CubicEps,                         // bucket fn: 2.4 bits/key, λ=3.5, α=0.99
+    CachelineEfVec<Vec<CachelineEf>>, // remap: 11.6 bits/entry (Elias-Fano)
+    Xx64,                             // hasher: XXH3-64 with seed
+    Vec<u8>,                          // pilots
+>;
+```
+
+**Hasher — `Xx64`**: canonical kmer raw values are left-aligned u64 with structural zeros in low bits (42 zeros for k=11, 2 zeros for k=31). `FxHash` (single multiply) distributes these poorly; `Xx64` (XXH3-64, seeded) handles structured input correctly.
+
+**Bucket function — `CubicEps`**: λ=3.5, α=0.99. Balanced tradeoff: 2× slower construction than `Linear/λ=3.0`, 20% less space. `default_compact` (λ=4.0) saves a further 12.5% at 2× more construction time — not chosen.
+
+**Remap — `CachelineEfVec`**: Elias-Fano variant packing 44 sorted 40-bit values per 64-byte cacheline (11.6 bits/value vs 32 for `Vec<u32>`). One cacheline per query; space win dominates at billion-scale key counts.
+
+---
+
+## Multilayer index architecture
+
+### Layer structure
+
+Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
+
+```
+layer_i/
+  unitigs.bin      — packed 2-bit nucleotide sequences (kmer evidence source)
+  unitigs.bin.idx  — random-access block index (block_bits controls granularity)
+  mphf.bin         — ptr_hash phase-2 MPHF
+  evidence.bin     — n × (chunk_id: 25 bits | rank: 7 bits) per slot  [exact mode]
+  fingerprint.bin  — n × b-bit fingerprints per slot                  [approx mode]
+  [no layer_meta.json — mode stored once in partition-level meta.json]
+```
+
+Layers are **disjoint**: a canonical kmer belongs to exactly one layer. Layer 0 is built from dataset A. Adding dataset B:
+
+1. For each kmer in B: probe existing layers. If found, the kmer is already indexed.
+2. Collect kmers of B not present in any layer → set `B \ A`.
+3. Build layer 1 from `B \ A` (dereplicate → count → De Bruijn → unitigs → `MphfLayer::build`).
+
+### Evidence modes
+
+Three evidence modes are supported via `IndexMode`, stored once in `PartitionMeta` at partition root. There is no `layer_meta.json`.
+
+**Exact** (`IndexMode::Exact`): `evidence.bin` stores one `(chunk_id, rank)` pair per MPHF slot. Verification reconstructs the kmer and compares to the query. Zero false positives. `.idx` required at query time.
+
+**Approx** (`IndexMode::Approx { b, z }`): `fingerprint.bin` stores a b-bit hash per slot. False-positive rate 1/2^b per query; Findere z-parameter reduces window FP to ≈ 1/2^(b·z). No `.idx` written or needed.
+
+**Hybrid** (`IndexMode::Hybrid { b, z }`): both `fingerprint.bin` and `evidence.bin` + `.idx`. `find()` uses the fingerprint (O(1)); `find_strict()` uses exact evidence (O(1)).
+
+### Build functions
+
+```
+MphfLayer::build(dir, block_bits, mode: &IndexMode, fill_slot)
+    Pass 1: CanonicalKmerIter + par_bridge() → build mphf.bin  (no .idx used)
+    Pass 2: sequential iter → fill evidence files + call fill_slot
+            .idx written last for Exact/Hybrid (query-time only)
+
+MphfLayer::build_exact_evidence(dir, block_bits)
+    Post-hoc: builds evidence.bin + .idx from existing mphf.bin + unitigs.bin
+    Uses open_sequential(); no .idx required on entry
+
+MphfLayer::build_approx_evidence(dir, b, z)
+    Post-hoc: builds fingerprint.bin from existing mphf.bin + unitigs.bin
+    Uses open_sequential(); never writes .idx
+```
+
+There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
+
+In `obikpartitionner`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
+
+### Membership verification
+
+ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry:
+
+- **Exact**: decode `(chunk_id, rank)` from `evidence.bin`; reconstruct the kmer via `unitigs.verify_canonical_kmer`; compare to query.
+- **Approx**: compare `kmer.seq_hash()` to the b-bit fingerprint stored at the slot.
+
+A mismatch in either mode means the kmer is absent from this layer; probe the next layer.
+
+### Query algorithm
+
+```
+fn query(kmer) → Option<(layer_index, slot)>:
+    for (i, layer) in layers.iter().enumerate():
+        slot = layer.mphf.index(kmer)
+        if layer.evidence.matches(slot, kmer):   // exact or approx dispatch
+            return Some((i, slot))
+    return None
+```
+
+`MphfLayer::find` dispatches on `LayerEvidence` at O(1) — no panicking `find_exact`/`find_approx` methods. `find_strict` always performs an exact check: O(1) for Exact/Hybrid, O(n) sequential scan for Approx. Expected probe depth: 1 for kmers in layer 0. Each probe is a ptr_hash lookup (~10 ns) plus one evidence check.
+
+### Merging layers
+
+Two layer chains can be merged by re-indexing their union through the full pipeline. This is expensive (full rebuild) but produces an optimal single-layer index. Merge is a maintenance operation, not a query-path requirement.

DevDocMD/implementation/mphf.refs.md

@@ -0,0 +1,16 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/mphf.md
+
+## Code couvert
+
+- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
+- `obikpartitionner/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
+
+## Notes
+
+FORT RISQUE DE DÉRIVE. Changements récents :
+- `build_exact_evidence(dir, block_bits)` — `block_bits` maintenant paramétrisé (défaut 0)
+- `build_approx_evidence(dir, b, z)` — nouvelle fonction pour l'évidence fingerprint
+- `build_evidence(dir, kind, block_bits)` — dispatch selon EvidenceKind
+- Construction en 2 phases : pass 1 (Rayon parallèle) + pass 2 (callback `fill_slot`)
+Vérifier que la doc décrit correctement les deux nouvelles routes d'évidence et le paramètre `block_bits`.

DevDocMD/implementation/obicompactvec.md

@@ -0,0 +1,533 @@
+# obicompactvec — Complete Reference
+
+## Module structure
+
+```
+src/obicompactvec/src/
+  lib.rs            public re-exports
+  views.rs          BitSliceView<'a>, IntSliceView<'a> — zero-copy read views
+  traits.rs         ColumnWeights, CountPartials, BitPartials (matrix aggregation)
+  bitvec.rs         PersistentBitVec, PersistentBitVecBuilder, BitIter
+  reader.rs         PersistentCompactIntVec (read-only)
+  builder.rs        PersistentCompactIntVecBuilder (read-write)
+  tempintvec.rs     TempCompactIntVec, TempCompactIntVecBuilder (temp-file-backed)
+  tempbitvec.rs     TempBitVec, TempBitVecBuilder (temp-file-backed)
+  bitmatrix.rs      PersistentBitMatrix, PersistentBitMatrixBuilder
+  intmatrix.rs      PersistentCompactIntMatrix, PersistentCompactIntMatrixBuilder
+  colgroup.rs       ColGroup, MatrixGroupOps trait
+  format.rs         file format constants, encode/decode helpers
+  layer_meta.rs     LayerMeta (column metadata)
+  meta.rs           matrix metadata
+```
+
+```mermaid
+graph TD
+    views --> bitvec
+    views --> builder
+    views --> tempbitvec
+    views --> tempintvec
+    views --> bitmatrix
+    views --> intmatrix
+    format --> reader
+    format --> builder
+    reader --> intmatrix
+    reader --> tempintvec
+    builder --> intmatrix
+    builder --> tempintvec
+    bitvec --> tempbitvec
+    bitvec --> bitmatrix
+    tempintvec --> intmatrix
+    tempintvec --> bitmatrix
+    tempbitvec --> intmatrix
+    tempbitvec --> bitmatrix
+    colgroup --> intmatrix
+    colgroup --> bitmatrix
+    layer_meta --> bitmatrix
+    layer_meta --> intmatrix
+    meta --> bitmatrix
+    meta --> intmatrix
+```
+
+---
+
+## Compact int encoding
+
+All integer vectors use the same two-tier encoding regardless of storage backend.
+
+**Primary array** — one `u8` per slot:
+
+- Values **0–254** are stored directly. No overhead.
+- Value **255 is a sentinel**: the slot's actual value is ≥ 255 and lives in the overflow store.
+
+**Overflow store** — maps slot index to a `u32` value ≥ 255:
+
+- In `PersistentCompactIntVecBuilder`: a `HashMap<usize, u32>` in RAM.
+- In `PersistentCompactIntVec` (reader): a sorted `[(slot: u64, value: u32)]` array in the mmap, with a sparse L1-resident index for binary search.
+
+```mermaid
+flowchart LR
+    slot --> P["primary[slot]: u8"]
+    P -->|"< 255"| V["value = byte (0–254)"]
+    P -->|"= 255 sentinel"| OV["overflow store"]
+    OV -->|"Builder"| HM["HashMap&lt;usize, u32&gt;\nin RAM"]
+    OV -->|"PersistentCompactIntVec"| SA["sorted [(slot,value)] in mmap\n+ sparse L1 index"]
+```
+
+**Key property — sentinel 255 = +∞ on `u8`:**
+
+- `min(a, 255) = a` for all `a ≤ 254` → correct when only one side is overflow
+- `max(a, 255) = 255` → correct sentinel when either side is overflow
+- Only the **both-overflow** case requires reading actual values from the overflow store.
+
+In practice, k (overflow count) ≪ n (total slots). Observed genomic data: ~0.07% of kmer slots are in overflow.
+
+---
+
+## View types
+
+The previous trait hierarchy (`BitSlice`, `BitSliceMut`, `IntSlice`, `IntSliceMut`) has been replaced by two concrete zero-copy view structs with inherent methods. Views are **`Copy`** — passing them is free. All read operations live on these two types.
+
+### `BitSliceView<'a>`
+
+```rust
+#[derive(Clone, Copy)]
+pub struct BitSliceView<'a> { pub(crate) words: &'a [u64], pub(crate) n: usize }
+```
+
+Bit `i` is at `words[i >> 6]` bit `i & 63` (LSB-first). Padding bits in the last word are zero.
+
+| Method | Cost |
+|---|---|
+| `len()`, `is_empty()` | O(1) |
+| `get(slot)` | O(1) |
+| `count_ones()` | POPCNT per word, O(n/64) |
+| `count_zeros()` | `n − count_ones()`, O(n/64) |
+| `iter() -> BitSliceIter<'a>` | O(1) setup, O(n) iteration |
+| `partial_jaccard_dist(other: BitSliceView)` | `(a&b).popcount`, `(a\|b).popcount` per word, O(n/64) |
+| `jaccard_dist(other: BitSliceView)` | from partial, O(n/64) |
+| `hamming_dist(other: BitSliceView)` | `(a^b).popcount` per word, O(n/64) |
+
+`BitSliceIter<'a>`: word-level scan; one word per 64 iterations.
+
+### `IntSliceView<'a>`
+
+```rust
+#[derive(Clone, Copy)]
+pub struct IntSliceView<'a> {
+    pub(crate) primary:      &'a [u8],
+    pub(crate) overflow_raw: &'a [u8],   // sorted [(slot:u64, value:u32)] entries
+    pub(crate) n_overflow:   usize,
+    pub(crate) n:            usize,
+}
+```
+
+`overflow_raw` contains `n_overflow` entries of `OVERFLOW_ENTRY_SIZE` bytes each, sorted by slot. The sort invariant is established at `close()`/`freeze()` time.
+
+| Method | Cost |
+|---|---|
+| `len()`, `is_empty()` | O(1) |
+| `primary_bytes()` | O(1) |
+| `overflow_entries() -> impl Iterator<(usize,u32)>` | O(n_overflow) iteration |
+| `get(slot)` | O(1) primary; binary search O(log k) for overflow slots |
+| `iter() -> IntSliceViewIter<'a>` | merge scan, O(n + k) |
+| `sum()` | byte scan + overflow, O(n + k) |
+| `count_nonzero()` | byte scan, O(n) |
+| Distance methods (`bray_dist`, `euclidean_dist`, `jaccard_dist`, …) | O(n + k) |
+
+`IntSliceViewIter<'a>`: merge scan using `overflow_pos` index. Requires sorted overflow — guaranteed by the construction lifecycle.
+
+**Builder `view()` vs reader `view()`:** `PersistentCompactIntVecBuilder` stores overflow as an unsorted `HashMap`, not raw bytes. Its `view()` returns an `IntSliceView` with `overflow_raw = &[]` and `n_overflow = 0`. This is intentional — the view is primarily useful after `freeze()`. During building, callers that need overflow use `overflow_entries()` directly.
+
+---
+
+## Concrete types
+
+```mermaid
+classDiagram
+    class BitSliceView {
+        +words: &[u64]
+        +n: usize
+        +get(slot) bool
+        +count_ones() u64
+        +iter() BitSliceIter
+        +jaccard_dist/hamming_dist(other: BitSliceView)
+    }
+    class IntSliceView {
+        +primary: &[u8]
+        +overflow_raw: &[u8]
+        +n_overflow: usize
+        +n: usize
+        +get(slot) u32
+        +iter() IntSliceViewIter
+        +overflow_entries() Iterator
+        +bray_dist/euclidean_dist/…(other: IntSliceView)
+    }
+    class PersistentBitVec {
+        -mmap: Mmap
+        -n: usize
+        +view() BitSliceView
+        +get(slot) bool
+        +count_ones/zeros() u64
+        +iter() BitIter
+        +partial_jaccard_dist(&Self) (u64,u64)
+        +jaccard_dist/hamming_dist(&Self) …
+    }
+    class PersistentBitVecBuilder {
+        -mmap: MmapMut
+        -n: usize
+        +view() BitSliceView
+        +set(slot, bool)
+        +or/and/xor/not(BitSliceView)
+        +copy_from(BitSliceView)
+        +close() / finish() → PersistentBitVec
+    }
+    class PersistentCompactIntVec {
+        -mmap: Mmap
+        -n: usize
+        -n_overflow: usize
+        -step: usize
+        -index: Vec~(usize,usize)~
+        +view() IntSliceView
+        +get(slot) u32
+        +iter() Iter
+        +sum/count_nonzero() u64
+        +bray_dist/euclidean_dist/… (&Self)
+    }
+    class PersistentCompactIntVecBuilder {
+        -mmap: MmapMut
+        -n: usize
+        -overflow: HashMap~usize,u32~
+        +view() IntSliceView
+        +set(slot, u32) / get(slot) u32
+        +inc / inc_present / inc_present_fast
+        +inc_predicate / inc_predicate_fast
+        +add/min/max/diff/mask_with(…View)
+        +primary_bytes/primary_bytes_mut()
+        +close() / finish() → PersistentCompactIntVec
+    }
+
+    PersistentBitVec --> BitSliceView : view()
+    PersistentBitVecBuilder --> BitSliceView : view()
+    PersistentCompactIntVec --> IntSliceView : view()
+    PersistentCompactIntVecBuilder --> IntSliceView : view() (primary only)
+    PersistentBitVecBuilder --> PersistentBitVec : close() then open()
+    PersistentCompactIntVecBuilder --> PersistentCompactIntVec : close() then open()
+```
+
+### `PersistentBitVec` / `PersistentBitVecBuilder`
+
+`PersistentBitVec` is the read-only type. `view()` returns a `BitSliceView<'_>` over the mmap word array. Direct inherent methods delegate to the view: `count_ones()`, `count_zeros()`, `partial_jaccard_dist(&Self)`, `jaccard_dist(&Self)`, `hamming_dist(&Self)`.
+
+`BitIter<'a>` — exported iterator for `PersistentBitVec::iter()`:
+
+```rust
+pub struct BitIter<'a> { pub(crate) words: &'a [u64], pub(crate) slot: usize, pub(crate) n: usize }
+```
+
+`PersistentBitVecBuilder` is the read-write type. Mutation operations accept `BitSliceView<'_>`:
+
+| Method | Cost |
+|---|---|
+| `set(slot, bool)` | O(1) |
+| `view() -> BitSliceView<'_>` | O(1) |
+| `or/and/xor(BitSliceView)` | word-level, O(n/64), SIMD-friendly |
+| `not()` | `w ^= u64::MAX` per word, re-masks last word | O(n/64) |
+| `copy_from(BitSliceView)` | `copy_from_slice` | O(n/64) |
+
+### `PersistentCompactIntVec` / `PersistentCompactIntVecBuilder`
+
+`PersistentCompactIntVec` is the read-only type. `view()` returns an `IntSliceView<'_>` over the mmap primary and overflow arrays. Inherent `iter()` is a merge scan (`Iter` struct). Inherent `sum()` and `count_nonzero()` use fast byte-scan helpers.
+
+`PersistentCompactIntVecBuilder` is the read-write type. Mutation methods on the builder fall into two categories:
+
+**Point mutations:**
+
+| Method | Note |
+|---|---|
+| `set(slot, u32)` | writes primary[slot] or 255+overflow |
+| `get(slot) -> u32` | reads primary byte or HashMap |
+| `inc(slot)` | `get` + `set`, O(1) |
+
+**Bulk computation methods** — accept view arguments:
+
+| Method | Semantics | Overflow |
+|---|---|---|
+| `inc_present(BitSliceView)` | `+= 1` at each 1-bit | via `inc`, safe for any group size |
+| `inc_present_fast(BitSliceView)` | same, raw u8 `+= 1` | `debug_assert` no 255 reached |
+| `inc_predicate(IntSliceView, pred)` | `+= 1` where `pred(col[s])` | two-pass, safe |
+| `inc_predicate_fast(IntSliceView, pred)` | same, raw u8 | `debug_assert` no 255 reached |
+| `add(IntSliceView)` | `self[s] += other[s]` | primary fast path + overflow fallback |
+| `min(IntSliceView)` | byte min + both-overflow fixup | see algorithm below |
+| `max(IntSliceView)` | pre-pass + byte max | see algorithm below |
+| `diff(IntSliceView)` | saturating sub | self<255 hot path |
+| `mask_with(BitSliceView)` | zeros slots where mask bit = 0 | O(n_zeros) |
+
+**`inc_present_fast` / `inc_predicate_fast` invariant:** caller guarantees no counter reaches 255 during the operation (group size < 255 for `inc_present_fast`, or chunk size < 255 for `inc_predicate_fast`). Violation is caught by `debug_assert` in dev builds.
+
+**`min` algorithm:**
+
+Exploits 255 = +∞: byte-level min is correct unless both sides are overflow.
+
+```
+snapshot self_ov: Vec<(slot,val)>
+snapshot other_ov: HashMap<slot,val>
+clear_overflow()
+Pass 1 — byte min, SIMD-vectorizable, O(n)
+Pass 2 — both-overflow fixup, O(k_self):
+  for (slot, self_val) in self_ov:
+    if slot ∈ other_ov: set(slot, min(self_val, other_ov[slot]))
+```
+
+**`max` algorithm:**
+
+Cannot do byte max first — `max(255, b<255)=255` overwrites self's original overflow value. Pre-pass reads self's value at other's overflow slots before the byte pass.
+
+```
+Pre-pass O(k_other): for (slot, other_val) in other.overflow_entries():
+  set(slot, max(self.get(slot), other_val))
+Pass 1 — byte max, SIMD-vectorizable, O(n)
+```
+
+---
+
+## Matrix types
+
+Four matrix types, two encodings × two formats:
+
+| | Columnar format | Packed format |
+|---|---|---|
+| **Bit** | `PersistentBitMatrix` (Columnar variant) | `PersistentBitMatrix` (Packed variant) |
+| **Int** | `PersistentCompactIntMatrix` (Columnar variant) | `PersistentCompactIntMatrix` (Packed variant) |
+
+Both matrix types are enums (`Columnar` / `Packed` / `Implicit` for bit) behind a transparent API. `col_view(c)` returns the appropriate view directly:
+
+```rust
+// PersistentBitMatrix
+pub fn col_view(&self, c: usize) -> BitSliceView<'_>
+
+// PersistentCompactIntMatrix
+pub fn col_view(&self, c: usize) -> IntSliceView<'_>
+```
+
+No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view struct immediately usable with any view method or bulk builder method.
+
+`pack_compact_int_matrix` and `pack_bit_matrix` convert columnar → packed format.
+
+---
+
+## Aggregation traits (matrix level)
+
+### ColumnWeights
+
+```rust
+trait ColumnWeights: Send + Sync {
+    fn col_weights(&self) -> Array1<u64>;         // sum per column
+    fn partial_kmer_counts(&self) -> Array1<u64>; // default = col_weights()
+}
+```
+
+`partial_kmer_counts` is overridden for count matrices to return `count_nonzero` per column (distinct kmers) rather than total count.
+
+### CountPartials
+
+Abstract required methods: `partial_bray`, `partial_euclidean`, `partial_threshold_jaccard`, `partial_relfreq_bray`, `partial_relfreq_euclidean`, `partial_hellinger`.
+
+**Additivity rule:** self-contained partials (`partial_bray`, `partial_euclidean`, `partial_threshold_jaccard`) can be element-wise summed across all `(partition, layer)` pairs. Normalised partials (`partial_relfreq_*`, `partial_hellinger`) require the **global** `col_weights` (accumulated across all layers and all partitions) as parameter.
+
+**`partial_threshold_jaccard` returns `(inter, union)`** because `union[i,j]` depends on both columns simultaneously.
+
+Provided finalisations:
+
+| Finalisation | Formula |
+|---|---|
+| `bray_dist_matrix()` | `1 − 2·partial_bray[i,j] / (w[i] + w[j])` |
+| `euclidean_dist_matrix()` | `√partial_euclidean[i,j]` |
+| `threshold_jaccard_dist_matrix(t)` | `1 − inter[i,j] / union[i,j]` |
+| `relfreq_bray_dist_matrix()` | `1 − partial_relfreq_bray[i,j]` |
+| `relfreq_euclidean_dist_matrix()` | `√partial_relfreq_euclidean[i,j]` |
+| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
+| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
+| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
+
+### BitPartials
+
+Required: `partial_jaccard() -> (Array2<u64>, Array2<u64>)`, `partial_hamming() -> Array2<u64>`. Both additive across layers and partitions.
+
+Provided finalisations also include `jaccard_dist_matrix()`, `hamming_dist_matrix()`, and `mash_dist_matrix(k)`.
+
+### Mash distance
+
+`mash_dist_matrix`/`threshold_mash_dist_matrix` add no new additive primitive: both are a pointwise transform of the existing Jaccard distance matrix, per the Mash mutation-rate estimator [@Mash-distances-doc; @Fan2015-mash-formula]:
+
+```
+D = -1/k · ln(2J / (1+J)),  J = 1 - d_jaccard
+```
+
+`J ≤ 0` (i.e. `d_jaccard ≥ 1`, no shared k-mers) maps to `D = 1` (maximal distance) rather than the `ln` singularity at `J = 0`.
+
+---
+
+## Temp-file-backed types
+
+**All inter-function results use temp-file-backed types** so the OS can page them out under memory pressure. This matters in practice: processing dozens of layers × hundreds of partitions in parallel would otherwise accumulate gigabytes of live anonymous memory.
+
+### Lifecycle
+
+```
+TempCompactIntVecBuilder::new(n)   →  writable mmap in TempDir
+     ↓  (inc_present_fast / inc_predicate_fast / add / mask_with / …)
+ .freeze()                          →  TempCompactIntVec  (read-only mmap + TempDir)
+     ↓  (optional)
+ .make_persistent(path)             →  PersistentCompactIntVec  (permanent file)
+```
+
+Same pattern for `TempBitVecBuilder` → `TempBitVec` → `PersistentBitVec`.
+
+**Drop order**: `TempCompactIntVec { vec: PersistentCompactIntVec, _temp: TempDir }` — Rust drops fields in declaration order. `vec` (mmap) released before `_temp` (directory deleted). No explicit `drop()` needed.
+
+### TempCompactIntVec / TempCompactIntVecBuilder
+
+```rust
+pub struct TempCompactIntVec {
+    vec:   PersistentCompactIntVec,
+    _temp: TempDir,        // dropped after vec
+}
+
+pub(crate) struct TempCompactIntVecBuilder {
+    builder: PersistentCompactIntVecBuilder,
+    temp:    TempDir,
+}
+```
+
+`TempCompactIntVec`: read access via `get(slot)`, `sum()`, `iter()`, `view() -> IntSliceView<'_>`.
+
+`TempCompactIntVecBuilder`: full delegation to inner `PersistentCompactIntVecBuilder` — all bulk computation methods (`inc_present_fast`, `inc_predicate_fast`, `add`, `min`, `max`, `diff`, `mask_with`) are exposed as `pub(crate)`.
+
+### TempBitVec / TempBitVecBuilder
+
+```rust
+pub struct TempBitVec {
+    vec:   PersistentBitVec,
+    _temp: TempDir,
+}
+
+pub(crate) struct TempBitVecBuilder {
+    builder: PersistentBitVecBuilder,
+    temp:    TempDir,
+}
+```
+
+`TempBitVec`: read access via `get(slot)`, `count_ones()`, `view() -> BitSliceView<'_>`, `iter()`.
+
+`TempBitVecBuilder`: exposes `set(slot, bool)`, `or(BitSliceView)`, and:
+
+```rust
+pub(crate) fn or_where(&mut self, col: IntSliceView<'_>, pred: impl Fn(u32) -> bool)
+```
+
+`or_where` — two passes, no intermediate allocation:
+
+```
+Pass 1 — primary bytes, O(n):
+  for slot in 0..n:
+    b = col.primary_bytes()[slot]
+    if b < 255 AND pred(b as u32): self.set(slot, true)
+
+Pass 2 — overflow, O(k):
+  for (slot, val) in col.overflow_entries():
+    if pred(val): self.set(slot, true)
+```
+
+---
+
+## Filter / Select API
+
+### ColGroup
+
+```rust
+pub struct ColGroup { pub name: String, pub indices: Vec<usize> }
+```
+
+Defined **once at the index level** from column metadata. Valid in all matrices of all layers and partitions — column structure is identical across the entire hierarchy; only rows (kmer slots) are partitioned.
+
+### Composition axis
+
+- **Across partitions**: kmer space is partitioned → partial results **concatenated** (disjoint kmer ranges).
+- **Across layers**: same kmer space, different counts → partial results **aggregated** (add, OR, etc.).
+
+### MatrixGroupOps
+
+Five required primitives + two default methods derived from them. All return temp-file-backed types.
+
+```rust
+pub trait MatrixGroupOps {
+    // required
+    fn partial_group_presence_count(&self, g: &ColGroup, threshold: u32)
+        -> io::Result<TempCompactIntVec>;
+    fn partial_group_sum(&self, g: &ColGroup)
+        -> io::Result<TempCompactIntVec>;
+    fn partial_group_any(&self, g: &ColGroup, threshold: u32)
+        -> io::Result<TempBitVec>;
+    fn partial_group_min(&self, g: &ColGroup)
+        -> io::Result<TempCompactIntVec>;
+    fn partial_group_max(&self, g: &ColGroup)
+        -> io::Result<TempCompactIntVec>;
+
+    // defaults derived from partial_group_presence_count
+    fn partial_group_all(&self, g: &ColGroup, threshold: u32)
+        -> io::Result<TempBitVec>;   // slot=1 iff count == g.indices.len()
+    fn partial_group_none(&self, g: &ColGroup, threshold: u32)
+        -> io::Result<TempBitVec>;   // slot=1 iff count == 0
+}
+```
+
+Implemented for both `PersistentCompactIntMatrix` and `PersistentBitMatrix`.
+
+For **bit matrices**: values are 0/1, so `partial_group_sum` = `partial_group_presence_count(g, 1)`; `partial_group_min` is AND (set first column then mask-with remaining); `partial_group_max` is OR via `partial_group_any` + `inc_present`.
+
+**`partial_group_presence_count` — chunking for large groups:**
+
+When `g.indices.len() < 255`: per-slot counts stay within `u8` range. Use `inc_present_fast` (bit) or `inc_predicate_fast(col_view(c), |v| v >= threshold)` (int) — raw u8 increment, no overflow entry written.
+
+When `g.indices.len() ≥ 255`: process in chunks of 254 columns, accumulate via `.add(chunk_frozen.view())`.
+
+**`partial_group_min` (int matrix)**: copy first column via `.add(col_view(first))` (start from 0 ⇒ copy), then `.min(col_view(c))` for remaining.
+
+**`partial_group_max` (int matrix)**: `.max(col_view(c))` for all columns (start from 0 ⇒ first column acts as copy).
+
+**`partial_group_any`** uses `or_where` on `TempBitVecBuilder` (two-pass: primary bytes then overflow entries).
+
+**`partial_group_all` / `partial_group_none`** (default): call `partial_group_presence_count`, then iterate slots to produce the bit result. O(n) extra pass, not chunked.
+
+### add_col_from — matrix builder integration
+
+Both matrix builders accept temp-file results directly:
+
+```rust
+// PersistentBitMatrixBuilder
+fn add_col_from(&mut self, src: &TempBitVec)         -> io::Result<()>
+fn add_col_from_int(&mut self, src: &TempCompactIntVec) -> io::Result<()>  // nonzero → 1
+
+// PersistentCompactIntMatrixBuilder
+fn add_col_from(&mut self, src: &TempCompactIntVec)  -> io::Result<()>
+fn add_col_from_bit(&mut self, src: &TempBitVec)     -> io::Result<()>  // bit → 0/1 u32
+```
+
+`add_col_from` copies the temp file to the matrix directory and increments `n_cols`; `close()` writes `meta.json` with the final column count. No separate `write_meta` step needed.
+
+### mask_with
+
+Direct method on `PersistentCompactIntVecBuilder` (and delegation via `TempCompactIntVecBuilder`). Zeros every slot where the corresponding mask bit is 0. Iterates only zero bits — O(n_zeros), O(1) when mask is all-ones.
+
+```
+for (w_idx, word) in mask.words():
+  if word == u64::MAX: continue   // skip all-ones words
+  zeros = !word
+  while zeros != 0:
+    bit = trailing_zeros(zeros)
+    s = w_idx * 64 + bit
+    if primary[s] != 0: set(s, 0)   // clears overflow entry too
+    zeros &= zeros − 1
+```
+
+Terminal operation for Filter (retain only selected kmer slots in a count vector) and Select (positional selection without MPHF).

DevDocMD/implementation/obilayeredmap.md

@@ -0,0 +1,396 @@
+# obilayeredmap — layered kmer index crate
+
+## Purpose
+
+`obilayeredmap` implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
+
+---
+
+## Three usage modes
+
+The MPHF + evidence infrastructure is the same for all modes. The **payload** varies.
+
+| Mode | Description | Payload type | Storage |
+|---|---|---|---|
+| 1. Set | membership test only | `()` | — |
+| 2. Count | occurrences per kmer per sample | `PersistentCompactIntMatrix` | `counts/` directory |
+| 3. Presence/absence | which genomes contain each kmer | `PersistentBitMatrix` | `presence/` directory |
+
+Both `PersistentCompactIntMatrix` and `PersistentBitMatrix` come from the `obicompactvec` crate.
+
+---
+
+## Index mode (homogeneity invariant)
+
+A partitioned index is homogeneous: every layer within a partition shares the same mode. The mode is determined once at `LayeredMap::open()` from `PartitionMeta.mode` and passed to each `Layer::open()` — no per-layer file is read.
+
+```rust
+#[derive(Serialize, Deserialize, Default)]
+#[serde(tag = "type", rename_all = "snake_case")]
+pub enum IndexMode {
+    #[default]
+    Exact,
+    Approx { b: u8, z: u8 },
+    Hybrid { b: u8, z: u8 },
+}
+```
+
+`IndexMode` is stored once in `PartitionMeta` (`meta.json` at partition root). There is no `layer_meta.json`.
+
+- **Exact**: writes `evidence.bin` + `unitigs.bin.idx`. Zero false positives.
+- **Approx**: writes `fingerprint.bin` only. FP rate per kmer = 1/2^b; with Findere z-parameter, z consecutive kmers must all match → effective window FP ≈ 1/2^(b·z). No `.idx` written or required.
+- **Hybrid**: writes both `fingerprint.bin` and `evidence.bin` + `.idx`. `find()` uses the fingerprint (fast, O(1)); `find_strict()` uses exact evidence.
+
+---
+
+## MphfLayer — autonomous kmer → slot mapping
+
+`MphfLayer` encapsulates the MPHF and evidence store for one layer. It is independent of any payload.
+
+```rust
+pub struct MphfLayer {
+    mphf: Mphf,
+    ev:   LayerEvidence,   // loaded at open() time
+    n:    usize,
+}
+```
+
+`LayerEvidence` is an internal enum, not public:
+
+```rust
+enum LayerEvidence {
+    Exact  { evidence: Evidence, unitigs: UnitigFileReader },
+    Approx { fingerprint: FingerprintVec, unitigs_path: PathBuf },
+    Hybrid { evidence: Evidence, unitigs: UnitigFileReader, fingerprint: FingerprintVec },
+}
+```
+
+`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read.
+
+### Query API
+
+Two public query methods, both returning `Option<usize>` (slot index):
+
+```rust
+pub fn find(&self, kmer: CanonicalKmer) -> Option<usize>
+pub fn find_strict(&self, kmer: CanonicalKmer) -> Option<usize>
+```
+
+- `find`: O(1) auto-dispatch. Exact/Hybrid → exact evidence check. Approx/Hybrid → fingerprint comparison.
+- `find_strict`: always exact. Exact/Hybrid → O(1) evidence check. Approx → O(n) sequential scan (no `.idx`).
+
+There are no `find_exact`/`find_approx` methods; panicking dispatch is eliminated.
+
+### Build surface
+
+```rust
+// Full MPHF + evidence build (two-pass)
+pub(crate) fn build(dir, block_bits, mode: &IndexMode, fill_slot) -> OLMResult<usize>
+
+// Evidence-only post-hoc builds (MPHF already present)
+pub fn build_exact_evidence(dir, block_bits) -> OLMResult<usize>
+pub fn build_approx_evidence(dir, b, z)      -> OLMResult<usize>
+```
+
+`MphfLayer::build` runs two passes over `unitigs.bin`:
+
+1. **Pass 1** (parallel via rayon): a `CanonicalKmerIter` (clonable, `Arc<Mmap>`, no file reopening) is passed to `new_from_par_iter` via `par_bridge()`. Produces `mphf.bin`. No `.idx` is read or created at this stage.
+2. **Pass 2** (sequential): fill evidence files; call `fill_slot(slot, kmer)` per kmer. `.idx` is written last for Exact/Hybrid modes (query-time only).
+
+There is no `build_evidence` dispatch wrapper — callers invoke `build_exact_evidence` or `build_approx_evidence` directly.
+
+For empty layers (n = 0), all build variants return `Ok(0)` immediately after creating empty output files.
+
+---
+
+## Layer\<D: LayerData\> — MPHF + payload
+
+`Layer<D>` pairs an `MphfLayer` with one payload store.
+
+```rust
+pub trait LayerData: Sized {
+    type Item;
+    fn open(layer_dir: &Path) -> OLMResult<Self>;
+    fn read(&self, slot: usize) -> Self::Item;
+}
+
+pub struct Layer<D: LayerData = ()> {
+    mphf: MphfLayer,
+    data: D,
+}
+
+pub struct Hit<T = ()> {
+    pub slot: usize,
+    pub data: T,
+}
+```
+
+`LayerData` covers the **read path only** (`open` + `read`). Build signatures differ between modes and are not part of the trait.
+
+| Type | `Item` | Description |
+|---|---|---|
+| `()` | `()` | mode 1 — membership only |
+| `PersistentCompactIntMatrix` | `Box<[u32]>` | mode 2 — count matrix (one u32 per column per slot) |
+| `PersistentBitMatrix` | `Box<[bool]>` | mode 3 — presence matrix (one bit per genome per slot) |
+
+### Build signatures
+
+```rust
+// mode 1
+impl Layer<()> {
+    pub fn build(out_dir: &Path, block_bits: u8, mode: &IndexMode) -> OLMResult<usize>
+}
+
+// mode 2
+impl Layer<PersistentCompactIntMatrix> {
+    pub fn build(out_dir: &Path, block_bits: u8, mode: &IndexMode,
+                 count_of: impl Fn(CanonicalKmer) -> u32) -> OLMResult<usize>
+    pub fn build_from_map(out_dir: &Path, block_bits: u8, mode: &IndexMode,
+                          counts: &HashMap<CanonicalKmer, u32>) -> OLMResult<usize>
+}
+
+// mode 3
+impl Layer<PersistentBitMatrix> {
+    pub fn build_presence(out_dir: &Path, block_bits: u8, mode: &IndexMode,
+                          n_genomes: usize,
+                          present_in: impl Fn(CanonicalKmer, usize) -> bool) -> OLMResult<usize>
+}
+```
+
+All build impls delegate to `MphfLayer::build` via a mode-specific `fill_slot` callback. The `mode` parameter is forwarded directly — no `LayerMeta` is written.
+
+Evidence-only post-hoc builds are accessible directly on `Layer<D>`:
+
+```rust
+impl<D: LayerData> Layer<D> {
+    pub fn build_exact_evidence(layer_dir: &Path, block_bits: u8) -> OLMResult<usize>
+    pub fn build_approx_evidence(layer_dir: &Path, b: u8, z: u8)  -> OLMResult<usize>
+}
+```
+
+There is no `build_evidence` dispatch wrapper.
+
+---
+
+## FingerprintVec and FingerprintVecWriter
+
+Approximate evidence is stored as a packed b-bit array, one fingerprint per MPHF slot.
+
+```
+fingerprint.bin format:
+  magic:   b"FPVF"  (4 bytes)
+  b:       u8       (bits per fingerprint, 1..=64)
+  padding: [0u8; 3]
+  n:       u64 LE   (number of slots)
+  data:    packed bits, ceil(n*b/8) bytes, Lsb0 order
+```
+
+```rust
+impl FingerprintVec {
+    pub fn open(path: &Path) -> OLMResult<Self>
+    pub fn get(&self, slot: usize) -> u64
+    pub fn matches(&self, slot: usize, fingerprint: u64) -> bool
+    pub fn n(&self) -> usize
+    pub fn b(&self) -> u8
+}
+```
+
+`matches(slot, hash)` extracts the b-bit fingerprint stored at `slot` and compares it to the low b bits of `hash`. It is the core operation of `find_approx`.
+
+---
+
+## LayeredMap\<D\> — collection of layers
+
+`LayeredMap<D>` wraps `Vec<Layer<D>>` for a single partition directory.
+
+```rust
+pub struct LayeredMap<D: LayerData = ()> {
+    root:   PathBuf,
+    meta:   PartitionMeta,
+    layers: Vec<Layer<D>>,
+}
+```
+
+`PartitionMeta` (`meta.json` at the partition root) stores `n_layers`.
+
+### Common methods
+
+```rust
+pub fn open(root: &Path)              -> OLMResult<Self>
+pub fn create(root: &Path, mode: IndexMode) -> OLMResult<Self>
+pub fn n_layers(&self)                -> usize
+pub fn layer(&self, i: usize)         -> &Layer<D>
+pub fn mode(&self)                    -> &IndexMode
+pub fn query(&self, kmer: CanonicalKmer) -> Option<(usize, Hit<D::Item>)>
+pub fn next_layer_writer(&self)       -> OLMResult<UnitigFileWriter>
+```
+
+`open` reads `PartitionMeta` once, extracts `mode`, and passes it to every `Layer::open` — no per-layer file is read. `create` stores the given mode in `PartitionMeta`.
+
+`query` probes layers in order and returns `(layer_index, Hit)` on the first match. Expected probe depth: 1 for kmers in layer 0.
+
+### push_layer
+
+`push_layer` builds the next layer from a `unitigs.bin` already written via `next_layer_writer`, using `DEFAULT_BLOCK_BITS`:
+
+```rust
+// mode 1
+impl LayeredMap<()> {
+    pub fn push_layer(&mut self) -> OLMResult<usize>
+}
+
+// mode 2
+impl LayeredMap<PersistentCompactIntMatrix> {
+    pub fn push_layer(&mut self, count_of: impl Fn(CanonicalKmer) -> u32) -> OLMResult<usize>
+    pub fn push_layer_from_map(&mut self, counts: &HashMap<CanonicalKmer, u32>) -> OLMResult<usize>
+}
+```
+
+Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers build directly via `Layer<PersistentBitMatrix>::build_presence`.
+
+---
+
+## LayeredStore\<S\> and aggregation traits
+
+`LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls:
+
+```rust
+pub struct LayeredStore<S>(pub Vec<S>);
+
+impl<S: ColumnWeights> ColumnWeights for LayeredStore<S> { … }  // Σ col_weights across inner stores
+impl<S: CountPartials> CountPartials  for LayeredStore<S> { … }  // element-wise Σ partials
+impl<S: BitPartials>   BitPartials    for LayeredStore<S> { … }  // element-wise Σ partials
+```
+
+Because blanket impls compose, `LayeredStore<LayeredStore<S>>` automatically inherits all three traits when `S` does — providing the partitioned level without a separate type.
+
+**Leaf implementors** (in `obicompactvec`):
+
+| Type | Traits |
+|---|---|
+| `PersistentCompactIntMatrix` | `ColumnWeights` (via `sum()`) + `CountPartials` |
+| `PersistentBitMatrix` | `ColumnWeights` (via `count_ones()`) + `BitPartials` |
+
+See [Kmer index architecture](../architecture/index_architecture.md) for the full trait API and the two-pass normalised-metric pattern.
+
+---
+
+## On-disk structure
+
+```
+partition_root/                    ← LayeredMap (one partition)
+  meta.json                        — {"n_layers": N, "mode": {"type": "exact"|"approx"|"hybrid", ...}}
+  layer_0/                         ← Layer
+    mphf.bin                       — ptr_hash MPHF (epserde format)
+    unitigs.bin                    — packed 2-bit nucleotide sequences
+    unitigs.bin.idx                — UIDX index (Exact/Hybrid only; query-time, never built during MPHF construction)
+    evidence.bin                   — [u32; n], LE  (Exact/Hybrid only)
+    fingerprint.bin                — packed b-bit array  (Approx/Hybrid only)
+    counts/                        [mode 2] PersistentCompactIntMatrix
+      meta.json
+      col_000000.pciv
+    presence/                      [mode 3] PersistentBitMatrix
+      meta.json
+      col_000000.pbiv  …
+  layer_1/
+    …
+```
+
+There is no `layer_meta.json`. The mode is stored once in `PartitionMeta` and is valid for all layers. `unitigs.bin.idx` is built at the end of `build_exact_evidence` — never during MPHF construction — and is consumed at query time only.
+
+---
+
+## Evidence encoding (exact)
+
+`evidence.bin` is a flat `[u32; n]` array with no header. Each u32 encodes one slot:
+
+```
+bits [31:7] = chunk_id (25 bits) — index of the unitig chunk
+bits [6:0]  = rank     (7 bits)  — kmer index within the chunk (0-based)
+```
+
+`chunk_id = raw >> 7`, `rank = raw & 0x7F`. Reconstructing the kmer: read k nucleotides at position `rank` within unitig `chunk_id` (requires `unitigs.bin.idx` for random access).
+
+For k=31, m=11, the observed maximum is ~46 kmers per chunk — well within the 127-kmer u7 capacity.
+
+---
+
+## ptr_hash configuration
+
+```rust
+type Mphf = PtrHash<
+    u64,                              // key type: canonical kmer raw encoding
+    CubicEps,                         // bucket fn: 2.4 bits/key, λ=3.5, α=0.99
+    CachelineEfVec<Vec<CachelineEf>>, // remap: Elias-Fano
+    Xx64,                             // hasher: XXH3-64 with seed
+    Vec<u8>,                          // pilots
+>;
+```
+
+`Xx64` is chosen over `FxHash` because canonical kmer raw values are left-aligned u64 with structural zeros in the low bits (42 zeros for k=11, 2 zeros for k=31), which single-multiply hashes distribute poorly.
+
+`CubicEps` with `PtrHashParams::<CubicEps>::default()` (λ=3.5): 2× slower construction than `Linear/λ=3.0`, ~20% less space.
+
+---
+
+## Column append and merge support
+
+These methods extend existing layers with new genome columns without touching the MPHF.
+
+### Layer-level genome column append
+
+```rust
+impl Layer<PersistentBitMatrix> {
+    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> bool) -> OLMResult<()>
+}
+
+impl Layer<PersistentCompactIntMatrix> {
+    pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> u32) -> OLMResult<()>
+}
+```
+
+Both delegate to the corresponding `PersistentBitMatrix::append_column` / `PersistentCompactIntMatrix::append_column`. They write a new column file (`col_NNNNNN.pbiv` / `col_NNNNNN.pciv`) and update `meta.json` to increment `n_cols`. `value_of` is called once per slot (0..n).
+
+### Presence matrix initialisation
+
+```rust
+impl Layer<()> {
+    pub fn init_presence_matrix(layer_dir: &Path, n_kmers: usize) -> OLMResult<()>
+}
+```
+
+Called on the first merge of a Presence-mode index. Creates `presence/` with `meta.json {"n": n_kmers, "n_cols": 1}` and `col_000000.pbiv` set entirely to `true`. This retroactively records genome 0 (the original source) as present in every slot, satisfying the column-count invariant before any new-source column is appended.
+
+### Why the MPHF is never rebuilt
+
+The MPHF, evidence, and unitigs are built once from the kmer set of a layer and are immutable for the lifetime of that layer. Adding a genome column does not change the kmer set — it only appends a new data column indexed by the same slot numbers. The only disk writes are one new `.pciv`/`.pbiv` file and a single `meta.json` update.
+
+---
+
+## Add-layer algorithm
+
+When adding dataset B to an existing index:
+
+1. For each partition, probe existing layers for kmers of B routed to that partition.
+2. Collect kmers absent from all layers → `B \ index`.
+3. Write `B \ index` to a new `unitigs.bin` via `next_layer_writer()`.
+4. Call `Layer<D>::build` (or `build_presence`) on the new layer directory.
+5. Call `push_layer` (or `append_layer`) to register the new layer in `meta.json`.
+
+Each partition's new layer is built independently; the operation is fully parallel across partitions.
+
+---
+
+## Dependencies
+
+| crate | role |
+|---|---|
+| `ptr_hash 1.1` | MPHF per layer |
+| `cacheline-ef 1.1` | compact remap inside ptr_hash |
+| `epserde 0.8` | zero-copy MPHF serialisation |
+| `memmap2 0.9` | mmap of evidence and fingerprint files |
+| `bitvec` | packed b-bit fingerprint storage |
+| `obiskio` | unitig file writer/reader + `.idx` build |
+| `obicompactvec` | payload types + aggregation traits |
+| `rayon 1` | parallel MPHF construction pass |
+| `serde / serde_json` | `PartitionMeta` serialisation |

DevDocMD/implementation/obilayeredmap.refs.md

@@ -0,0 +1,22 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/obilayeredmap.md
+
+## Code couvert
+
+- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
+- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
+- `obilayeredmap/src/map.rs` — LayeredMap<D>, push_layer(), query()
+- `obilayeredmap/src/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
+- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
+- `obilayeredmap/src/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
+
+## Notes
+
+FORT RISQUE DE DÉRIVE. C'est le fichier le plus affecté par les changements récents :
+- EvidenceKind (Exact / Approx) est désormais un concept de premier plan — toute la sémantique de query en dépend
+- `LayerEvidence` enum interne à `MphfLayer` : dispatch transparent find() → find_exact() ou find_approx()
+- `fingerprint.rs` : module entièrement nouveau (FingerprintVec + FingerprintVecWriter)
+- `build_evidence()` / `build_exact_evidence()` / `build_approx_evidence()` sont nouveaux
+- `block_bits` dans les fonctions build : O(1) garanti avec le chemin chaud explicit pour block_bits=0
+- Séparation open() (accès aléatoire, requiert .idx) vs open_sequential() (itération seule)
+Pratiquement toute cette page est à réécrire.

DevDocMD/implementation/obipipeline.md

@@ -0,0 +1,179 @@
+# obipipeline — parallel pipeline library
+
+`obipipeline` is a generic, multi-threaded data pipeline crate. It connects a **source**, a chain of **stages**, and a **sink** via crossbeam channels, running each stage with a shared worker pool and a biased scheduler.
+
+## Core types
+
+| Type alias | Rust type | Role |
+|---|---|---|
+| `SourceFn<D>` | `Box<dyn FnMut() -> Result<D, PipelineError> + Send>` | Called repeatedly; `FnMut` because it holds iterator state |
+| `SharedFn<D>` | `Arc<dyn Fn(D) -> Result<D, PipelineError> + Send + Sync>` | 1→1 transform shared across workers via `Arc::clone` |
+| `SharedFlatFn<D>` | `Arc<dyn Fn(D, &Sender<Result<D, _>>, &Sender<isize>) + Send + Sync>` | 1→N transform; pushes items into channel, sends delta |
+| `SinkFn<D>` | `Box<dyn Fn(D) -> Result<(), PipelineError> + Send>` | Final consumer; returns `Result` so errors propagate back |
+
+Stages come in two variants:
+
+```rust
+pub enum Stage<D> {
+    Transform(SharedFn<D>),   // 1→1
+    Flat(SharedFlatFn<D>),    // 1→N
+}
+```
+
+`Pipeline<D>` holds one `SourceFn`, a `Vec<Stage>`, and one `SinkFn`.  
+`WorkerPool<D>` wraps a `Pipeline` with `n_workers` and channel `capacity`.
+
+## WorkerPool
+
+```rust
+WorkerPool::new(pipeline: Pipeline<D>, n_workers: usize, capacity: usize) -> Self
+WorkerPool::run(self)
+```
+
+| Parameter | Role |
+|---|---|
+| `n_workers` | Number of parallel worker threads. Each worker is generic — it executes whichever transform the scheduler assigns it. |
+| `capacity` | Bound on every crossbeam channel in the pipeline. Controls memory and back-pressure: a full channel blocks the sender until a slot frees. |
+
+`run` consumes `self` (all fields are moved into threads). It blocks the calling thread until the pipeline has fully drained — source exhausted and every in-flight item processed by the sink — then joins all threads before returning.
+
+## Data enum
+
+All pipeline stages communicate through a single user-defined enum:
+
+```rust
+enum MyData {
+    Unsigned(u64),
+    Number(f64),
+    Text(String),
+}
+```
+
+Each variant carries the concrete type for one stage's output. The macros pattern-match on this enum to route values between stages.
+
+## Macros
+
+Eight low-level macros build individual stages; one high-level macro (`make_pipeline!`) composes them.
+
+### Low-level
+
+```rust
+make_source!(Enum, iterator, OutputVariant)          // iterator yields T
+make_source_fallible!(Enum, iterator, OutputVariant) // iterator yields Result<T, E>
+
+make_transform!(Enum, func, InputVariant, OutputVariant)          // func: T -> U
+make_transform_fallible!(Enum, func, InputVariant, OutputVariant) // func: T -> Result<U, E>
+
+make_flat_transform!(Enum, func, InputVariant, OutputVariant)          // func: T -> impl IntoIterator<Item=U>
+make_flat_transform_fallible!(Enum, func, InputVariant, OutputVariant) // func: T -> Result<impl IntoIterator<Item=U>, E>
+
+make_sink!(Enum, func, InputVariant)           // func: T -> ()
+make_sink_fallible!(Enum, func, InputVariant)  // func: T -> Result<(), E>
+```
+
+Each macro wraps the closure in the correct smart pointer (`Box` for source/sink, `Arc` for transforms).
+
+### make_pipeline! DSL
+
+```
+make_pipeline! {
+    DataEnum,
+    source   iterator     => OutputVariant,   // or source?  for fallible
+    |   func: In => Out,                       // 1→1 non-fallible transform
+    |?  func: In => Out,                       // 1→1 fallible transform
+    ||  func: In => Out,                       // 1→N non-fallible flat transform
+    ||? func: In => Out,                       // 1→N fallible flat transform
+    sink     func         @ InputVariant,      // or sink?    for fallible
+}
+```
+
+`?` marks fallibility on source, individual transforms, or sink independently.  
+Implemented as a **TT muncher**: the internal rule `@build` recurses over transform tokens one at a time, accumulating them into a `vec![]`, then terminates on `sink`/`sink?`.
+
+### make_pipe! DSL
+
+`make_pipe!` builds a sourceless/sinkless `Pipe<D, In, Out>` — a reusable, composable stage sequence:
+
+```
+make_pipe! {
+    DataEnum : InType => OutType,
+    |   func: InVariant => OutVariant,
+    |?  func: InVariant => OutVariant,
+    ||  func: InVariant => OutVariant,
+    ||? func: InVariant => OutVariant,
+}
+```
+
+Two pipes compose with `.then(other)`. Apply to an iterator with `.apply(iter, n_workers, capacity)` to get a `PipeIter<Out>` — an iterator over the pipeline output, backed by a background `WorkerPool`. The scatter step in `obikmer` uses `make_pipe!` and `.apply()` rather than the full `make_pipeline!` / `WorkerPool` pattern.
+
+## Scheduler architecture
+
+```
+Source thread ──► [source_rx] ──► Scheduler ──► [worker_tx] ──► Workers (×N)
+                                      ▲                               │
+                  [stage_rxs] ────────┘◄──────────────────────────────┘
+                  [flat_delta_rx] ──► Scheduler  (in_flight adjustment)
+                                      │
+                              [sink_err_rx]  ← errors from sink (highest priority)
+                                      │
+                                   Sink thread
+```
+
+The scheduler is a single thread running a biased `Select` over all input channels. Priority order (highest first):
+
+```
+index 0       sink_err_rx          abort on sink error
+index 1       flat_delta_rx        adjust in_flight before dispatching
+index 2..=n+1 stage_rxs[n-1..0]   drain last stage first
+index n+2     source_rx            pull new data last
+```
+
+This back-pressure-friendly ordering ensures downstream stages are drained before new items enter the pipeline.
+
+**Workers** are generic: each receives a `WorkerTask` — either `Transform(data, stage_idx)` or `Flat(data, stage_idx)`. For `Transform`, the worker calls `f(data)` and sends the result to `stage_txs[stage_idx]`. For `Flat`, the worker calls `f(data, &push_tx, &delta_tx)`: the closure pushes N items into `push_tx` then sends `N-1` to `delta_tx`. The scheduler uses the delta to adjust `in_flight` without knowing N in advance.
+
+**Termination** uses an `in_flight: isize` counter and a `flat_workers_active: usize` counter:
+
+- `in_flight` incremented when an item is dispatched from source to workers
+- `in_flight` decremented when the item exits the last stage to the sink
+- `flat_workers_active` incremented when a `Flat` task is dispatched, decremented when the delta arrives
+- the loop exits only when `source_done && in_flight == 0 && flat_workers_active == 0`
+
+This guarantees all in-flight items complete (including all N outputs of a flat stage) before `join()`.
+
+## Error handling
+
+`PipelineError` has four variants:
+
+| Variant | Meaning |
+|---|---|
+| `EndOfStream` | Source exhausted (normal termination, not sent downstream) |
+| `TypeMismatch` | Wrong enum variant arrived at a stage |
+| `StepKindMismatch` | Internal routing error |
+| `StepError(Box<dyn Error + Send + Sync>)` | Error from user code (wrapped by `make_*_fallible!`) |
+
+Sink errors flow back to the scheduler via a dedicated `Receiver<PipelineError>` registered at index 0 of the Select — the pipeline stops immediately on the first sink error.
+
+## Example
+
+```rust
+enum PipelineData { Unsigned(u64), Number(f64), Text(String) }
+
+fn to_f64(x: u64) -> f64 { x as f64 }
+fn format_num(n: f64) -> String { format!("{}", n) }
+fn reverse(s: String) -> String { s.chars().rev().collect() }
+fn hash(s: String) -> u64 { /* djb2 */ }
+fn print_hash(h: u64) -> Result<(), std::io::Error> { println!("{}", h); Ok(()) }
+
+let pipeline = make_pipeline! {
+    PipelineData,
+    source  1u64..=10 => Unsigned,
+    | to_f64:     Unsigned => Number,
+    | format_num: Number   => Text,
+    | reverse:    Text     => Text,
+    | hash:       Text     => Unsigned,
+    sink?   print_hash     @ Unsigned,
+};
+
+WorkerPool::new(pipeline, 4, 64).run();
+```

DevDocMD/implementation/obipipeline.refs.md

@@ -0,0 +1,13 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/obipipeline.md
+
+## Code couvert
+
+- `obipipeline/src/lib.rs` — WorkerPool, Pipeline, macro make_pipeline!
+- `obipipeline/src/scheduler.rs` — Scheduler avec Select biaisé sur les entrées de canaux
+
+## Notes
+
+Document stable (librairie générique, peu de risque de dérive).
+Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartitionner`
+ou s'il a été remplacé par Rayon dans certains chemins.

DevDocMD/implementation/obitaxonomy.md

@@ -0,0 +1,143 @@
+# `obitaxonomy` — taxonomy concept paths
+
+`obitaxonomy` is a dependency-free crate that defines a typed representation
+of hierarchical concept paths (taxonomic or otherwise) stored in genome metadata.
+
+---
+
+## Concept path syntax
+
+A concept path is stored as a metadata value with the prefix `taxonomy:/`:
+
+```
+taxonomy:/enterobacteriaceae@family/Escherichia@genus/Escherichia coli@species
+```
+
+Structure:
+
+- The `taxonomy:/` prefix is the type discriminator. Any metadata value starting
+  with it is parsed as a `TaxPath`; all others remain plain strings.
+- The remainder is one or more `/`-separated segments.
+- Each segment is `name` or `name@rank`, where `rank` is a label for the
+  taxonomic level (e.g. `family`, `genus`, `species`).
+- Rank annotations are **optional per segment** and can be mixed freely.
+- Spaces are allowed in both names and ranks.
+
+### Reserved character
+
+`@` is reserved throughout the taxonomy system and may **not** appear in:
+
+| Context | Constraint |
+|---------|------------|
+| Segment name | forbidden |
+| Rank/class label | forbidden |
+| Metadata key names | forbidden (used as `key@rank` in predicate syntax) |
+
+`@` is freely allowed in plain-text metadata values (non-taxonomy).
+
+### Parse errors
+
+| Condition | Error |
+|-----------|-------|
+| Value does not start with `taxonomy:/` | `MissingPrefix` |
+| No segments after the prefix | `EmptyPath` |
+| Segment with empty name (consecutive `/`) | `EmptySegmentName` |
+| Segment with trailing `@` and no rank (`name@`) | `EmptyRankName` |
+| Segment with more than one `@` | `AmbiguousRank` |
+
+---
+
+## Public API
+
+### `TaxSegment`
+
+A single node: a name and an optional rank.
+
+```rust
+seg.name()            // &str
+seg.rank()            // Option<&str>
+seg.to_string()       // "name" or "name@rank"
+TaxSegment::parse(s)  // Result<TaxSegment, TaxError>
+```
+
+### `TaxPath`
+
+```rust
+TaxPath::parse(s)               // Result<TaxPath, TaxError>
+path.segments()                 // &[TaxSegment]
+path.depth()                    // usize — number of segments
+path.is_ancestor_of(&other)     // bool — prefix match by name, ranks ignored
+path.name_at_rank("genus")      // Option<&str>
+path.to_string()                // reconstructs "taxonomy:/…"
+```
+
+`is_ancestor_of` compares segment **names** only — rank annotations are
+informational and do not affect the ancestry relation.
+
+```rust
+let a: TaxPath = "taxonomy:/Enterobacteriaceae@family/Escherichia@genus".parse()?;
+let b: TaxPath = "taxonomy:/Enterobacteriaceae@family/Escherichia@genus/Escherichia coli@species".parse()?;
+
+assert!(a.is_ancestor_of(&b));   // true
+assert!(b.is_ancestor_of(&a));   // false
+assert!(a.is_ancestor_of(&a));   // true  (equal ⇒ ancestor)
+
+assert_eq!(b.name_at_rank("species"), Some("Escherichia coli"));
+assert_eq!(b.name_at_rank("genus"),   Some("Escherichia"));
+assert_eq!(b.name_at_rank("order"),   None);
+```
+
+---
+
+## Integration with `GenomeInfo`
+
+At index load time, every metadata value is inspected once:
+
+- Starts with `taxonomy:/` → parsed into `TaxPath`, stored in `genome.taxonomy`.
+- Otherwise → kept as-is in `genome.meta`.
+
+```rust
+struct GenomeInfo {
+    label:    String,
+    meta:     HashMap<String, String>,    // plain text metadata
+    taxonomy: HashMap<String, TaxPath>,   // parsed taxonomy metadata
+}
+```
+
+The raw string is not duplicated. `TaxPath::to_string()` reconstructs the
+original value losslessly for serialisation.
+
+---
+
+## Predicate operators (in `filter` / `select`)
+
+Path predicates use the `~` / `!~` operators. The **stored value** always starts
+with `/` (rooted path); the **query pattern** does not need to.
+
+### Path pattern syntax
+
+| Pattern | Semantics |
+|---------|-----------|
+| `A/B` | contiguous sub-path A then B, anywhere in the value |
+| `/A/B` | value starts with A then B (start-anchored) |
+| `A/B$` | value ends with A then B (end-anchored) |
+| `/A/B$` | value is exactly A then B (fully anchored) |
+| `A@x/B` | A with class `x` followed by B with any class |
+| `A@x/B@y` | A with class `x` followed by B with class `y` |
+
+A segment pattern without `@` matches the segment name regardless of its stored class.
+
+### Rank-aware queries
+
+```
+key@rank=value
+```
+
+| Predicate form | Semantics |
+|----------------|-----------|
+| `key@rank=value` | genome's `key` has `value` at rank `rank` |
+| `key@rank!=value` | does not |
+| `key@rank=v1\|v2` | value at `rank` is `v1` or `v2` |
+
+`~` combined with `@rank` on the key (e.g. `key@genus~pattern`) is not defined
+and is rejected at parse time.

DevDocMD/implementation/persistent_bit_vec.md

@@ -0,0 +1,270 @@
+# PersistentBitVec and PersistentBitMatrix
+
+## Purpose
+
+`PersistentBitVec` stores a dense bit vector (presence/absence per slot) backed by a single mmap'd file. It is the binary counterpart of `PersistentCompactIntVec` and shares the same lifecycle pattern (builder → close → reader). All bulk operations work on u64 words rather than bytes, giving 8× fewer iterations and enabling the compiler to emit POPCNT and SIMD instructions.
+
+Typical use: converting k-mer count vectors to presence/absence vectors (with optional threshold), then computing set-theoretic distances (Jaccard) or edit distances (Hamming) between samples.
+
+`PersistentBitMatrix` wraps multiple `PersistentBitVec` columns in a directory, exposing a column-major binary matrix with row-access API. A single-column bit matrix is a vector at the API level.
+
+---
+
+## PersistentBitVec — single-column file
+
+### File format
+
+Single `.pbiv` file.
+
+```
+offset 0:
+  magic:     [u8; 4]  = b"PBIV"
+  _pad:      [u8; 4]  = 0           alignment padding
+  n:         u64       number of bits
+
+offset 16:
+  data:      [u64; ⌈n/64⌉]          bit words, LSB-first, zero-padded
+```
+
+**Header is 16 bytes**, so data starts at an offset divisible by 8. Since `mmap` returns page-aligned memory (≥ 4096-byte aligned), the data slice is u64-aligned, enabling a zero-copy `&[u8] → &[u64]` reinterpretation.
+
+**Bit layout**: bit `i` is in `data[i >> 6]` at bit position `i & 63` (LSB-first). Bits `[n, ⌈n/64⌉×64)` are **always zero** (padding). This invariant is maintained by all write operations and must be restored by `not()` after flipping.
+
+**Total file size**: `16 + ⌈n/64⌉ × 8` bytes.
+
+### Lifecycle
+
+#### Builder (`PersistentBitVecBuilder`)
+
+```rust
+struct PersistentBitVecBuilder {
+    mmap: MmapMut,
+    n:    usize,
+}
+```
+
+The file and mmap are created immediately at construction. The header is written once at `new()` or copied from the source at `build_from*()`. `close()` is a single flush — there is no tail to append, unlike `PersistentCompactIntVec`.
+
+**`new(n: usize, path: &Path) -> io::Result<Self>`**
+
+Creates the file, writes the header, zero-extends to `16 + ⌈n/64⌉×8` bytes, mmaps immediately. All bits default to 0.
+
+**`build_from(source: &PersistentBitVec, path: &Path) -> io::Result<Self>`**
+
+OS-level file copy (no per-bit iteration), then mmap. Initialisation cost: O(file_size).
+
+**`build_from_counts(source: &PersistentCompactIntVec, threshold: u32, path: &Path) -> io::Result<Self>`**
+
+Creates a new file, iterates `source` with its merge-scan iterator (O(n)), and writes bits directly into u64 words:
+
+```rust
+// bit i = 1 iff source[i] >= threshold
+words[slot >> 6] |= 1u64 << (slot & 63);
+```
+
+Handles overflow values (≥ 255) transparently — the count iterator returns the true u32 value regardless.
+
+**`build_from_presence(source: &PersistentCompactIntVec, path: &Path) -> io::Result<Self>`**
+
+Shorthand for `build_from_counts(source, 1, path)`.
+
+**Bit-level access**
+
+```rust
+fn get(&self, slot: usize) -> bool
+fn set(&mut self, slot: usize, value: bool)
+```
+
+Byte-level mmap access: `mmap[16 + slot/8]`, bit `slot % 8`. O(1).
+
+**Word-level bulk operations**
+
+All operate on `⌈n/64⌉` u64 words. O(n/64) per call.
+
+```rust
+builder.and(&other);   // self[i] &= other[i]  for all i
+builder.or(&other);    // self[i] |= other[i]
+builder.xor(&other);   // self[i] ^= other[i]
+builder.not();         // self[i]  = !self[i], then re-zero padding bits
+```
+
+`and`/`or`/`xor` read `other`'s word slice directly (no allocation). `not()` flips all words then masks the last word's padding bits to restore the invariant.
+
+**`close(self) -> io::Result<()>`**
+
+Flushes the mmap. The header was written at construction and is never rewritten. O(1) in Rust code.
+
+#### Reader (`PersistentBitVec`)
+
+```rust
+struct PersistentBitVec {
+    mmap: Mmap,
+    n:    usize,
+    path: PathBuf,
+}
+```
+
+**`open(path: &Path) -> io::Result<Self>`**
+
+Mmaps the file, validates magic, reads `n` from bytes `[8..16]`. O(1).
+
+**`get(slot: usize) -> bool`**
+
+Byte-level read from `mmap[16 + slot/8]`. O(1).
+
+**`iter() -> BitIter<'_>`**
+
+Sequential scan, byte by byte, yielding `bool` values in slot order. Implements `ExactSizeIterator`. O(n).
+
+**Aggregates**
+
+```rust
+fn count_ones(&self)  -> u64   // popcount over all words; padding bits are 0
+fn count_zeros(&self) -> u64   // n - count_ones()
+```
+
+`count_ones` iterates `⌈n/64⌉` words and calls `u64::count_ones()` (maps to `POPCNT`). O(n/64).
+
+**Distance methods**
+
+Both operate word by word. O(n/64).
+
+| Method | Formula | Notes |
+|---|---|---|
+| `jaccard_dist(&other) -> f64` | `1 − \|A∩B\| / \|A∪B\|` | `(a&b).count_ones()`, `(a\|b).count_ones()` per word |
+| `hamming_dist(&other) -> u64` | number of differing bits | `(a^b).count_ones()` per word |
+
+Edge case (both all-zero → union = 0): `jaccard_dist` returns 0.0.
+
+### Implementation notes
+
+#### u64 word view
+
+The unsafe cast from `&[u8]` to `&[u64]` is sound because:
+
+1. `mmap` base is page-aligned (≥ 4096-byte boundary).
+2. Data offset = 16, and `16 % 8 == 0` → the data pointer is 8-byte aligned.
+3. Data length = `⌈n/64⌉ × 8` bytes — always a multiple of 8.
+
+This gives zero-copy word-level access with no intermediate allocation.
+
+#### Padding invariant
+
+Writing `not()` without masking the last word would corrupt `count_ones()`, `hamming_dist()`, and `jaccard_dist()`. The mask applied after flipping is `(1u64 << (n % 64)) - 1` (no-op if `n % 64 == 0`). All other operations (`and`, `or`, `xor`) preserve existing zero padding since they can only clear or preserve bits already set by `not()`.
+
+### Complexity
+
+| Operation | Time | Notes |
+|---|---|---|
+| `new` / `open` | O(1) | mmap setup + header parse |
+| `get` / `set` (builder or reader) | O(1) | byte-level mmap |
+| `iter()` | O(n) | byte-by-byte scan |
+| `count_ones` / `count_zeros` | O(n/64) | POPCNT per u64 word |
+| `and` / `or` / `xor` / `not` | O(n/64) | word-level bitwise ops |
+| `jaccard_dist` / `hamming_dist` | O(n/64) | word AND/OR/XOR + POPCNT |
+| `build_from` | O(file_size) | OS copy |
+| `build_from_counts` / `build_from_presence` | O(n) | count iter + word fill |
+| `close` | O(1) | flush only |
+
+---
+
+## PersistentBitMatrix — column-major directory
+
+### Design
+
+A directory containing `meta.json` and N column files `col_000000.pbiv`, `col_000001.pbiv`, …, each a `PersistentBitVec`. Used for presence/absence matrices: one column per genome, one bit per MPHF slot.
+
+```
+presence/
+  meta.json          {"n": <n_slots>, "n_cols": <G>}
+  col_000000.pbiv    genome 0
+  col_000001.pbiv    genome 1
+  ...
+```
+
+Column-major layout makes per-genome set operations (Jaccard, Hamming, AND/OR) cache-friendly — each genome is a contiguous file. Row access (which genomes contain a given kmer) requires one O(1) read per column.
+
+### Builder (`PersistentBitMatrixBuilder`)
+
+```rust
+struct PersistentBitMatrixBuilder {
+    dir:    PathBuf,
+    n:      usize,
+    n_cols: usize,
+}
+```
+
+**`new(n: usize, dir: &Path) -> io::Result<Self>`**
+
+Creates the directory (including parents).
+
+**`add_col(&mut self) -> io::Result<PersistentBitVecBuilder>`**
+
+Creates `col_NNNNNN.pbiv` for the next column and returns its builder. The caller fills the column and calls `builder.close()` before calling `add_col` again.
+
+**`close(self) -> io::Result<()>`**
+
+Writes `meta.json` with the final `n` and `n_cols`.
+
+### Reader (`PersistentBitMatrix`)
+
+```rust
+struct PersistentBitMatrix {
+    cols: Vec<PersistentBitVec>,
+    n:    usize,
+}
+```
+
+**`open(dir: &Path) -> io::Result<Self>`**
+
+Reads `meta.json`, opens all `col_NNNNNN.pbiv` files.
+
+**`row(slot: usize) -> Box<[bool]>`**
+
+Returns the presence vector: `[col_0[slot], col_1[slot], …, col_{G-1}[slot]]`. One byte read per column. O(G).
+
+**`col(c: usize) -> &PersistentBitVec`**
+
+Direct access to a single column for column-oriented operations.
+
+### LayerData implementation
+
+```rust
+impl LayerData for PersistentBitMatrix {
+    type Item = Box<[bool]>;
+    fn open(layer_dir: &Path) -> OLMResult<Self> { /* opens layer_dir/presence/ */ }
+    fn read(&self, slot: usize) -> Box<[bool]>   { self.row(slot) }
+}
+```
+
+---
+
+## Aggregation traits — `obicompactvec::traits`
+
+`PersistentBitMatrix` implements two aggregation traits used by `LayeredStore<S>` for cross-layer and cross-partition distance computations.
+
+### ColumnWeights
+
+```rust
+impl ColumnWeights for PersistentBitMatrix {
+    fn col_weights(&self) -> Array1<u64>   // = self.count_ones()
+}
+```
+
+`col_weights()[c]` = number of set bits in column `c` across all slots.
+
+### BitPartials
+
+```rust
+impl BitPartials for PersistentBitMatrix {
+    // Self-contained partials (additive across layers)
+    fn partial_jaccard(&self) -> (Array2<u64>, Array2<u64>)   // (inter, union)
+    fn partial_hamming(&self) -> Array2<u64>                   // differing bits
+
+    // Provided finalisations
+    fn jaccard_dist_matrix(&self) -> Array2<f64>
+    fn hamming_dist_matrix(&self) -> Array2<u64>
+}
+```
+
+`partial_jaccard` returns `(inter, union)` as a pair because `union` is not reconstructible from per-column `count_ones()` — it depends on both columns simultaneously. Both components are additively decomposable across `(partition, layer)` pairs; the final `jaccard_dist_matrix()` is computed from their element-wise sums.

DevDocMD/implementation/persistent_bit_vec.refs.md

@@ -0,0 +1,13 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/persistent_bit_vec.md
+
+## Code couvert
+
+- `obicompactvec/src/bitvec.rs` — PersistentBitVec, opérations mot u64, invariant de padding
+- `obicompactvec/src/bitmatrix.rs` — PersistentBitMatrix, wrapper colonne-major, append_column
+- `obicompactvec/src/bitmatrix.rs` — PersistentBitMatrixBuilder
+
+## Notes
+
+Document d'implémentation stable. Vérifier que `PersistentBitMatrixBuilder` et `append_column`
+sont couverts (utilisés dans `Layer::<PersistentBitMatrix>::build_presence` et `append_genome_column`).

DevDocMD/implementation/persistent_compact_int_vec.md

@@ -0,0 +1,308 @@
+# PersistentCompactIntVec and PersistentCompactIntMatrix
+
+## Purpose
+
+`PersistentCompactIntVec` stores a dense array of non-negative integers indexed by MPHF slot where the vast majority of values are small (0–254) and large values are rare. It is designed for mmap-compatible random and sequential access with minimal memory footprint and optimal cache behaviour.
+
+Motivation from observed count distributions in genomics data: 99.9% of k-mer counts fit in a u8; overflow (count ≥ 255) affects ~0.07% of distinct k-mers but can reach values above 10⁶ (chloroplast, ribosomal repeats).
+
+`PersistentCompactIntMatrix` wraps multiple `PersistentCompactIntVec` columns in a directory, exposing a column-major matrix with row-access API. A vector is a matrix with 1 column.
+
+---
+
+## PersistentCompactIntVec — single-column file
+
+### Design
+
+Two-tier structure:
+
+1. **Primary array** — `[u8; n]`, stored at offset 40 in the PCIV file and mmap'd. Values 0–254 are stored directly. Value **255 is a sentinel** meaning "look in overflow".
+2. **Overflow section** — sorted list of `(slot: u64, value: u32)` pairs for all slots where the true value ≥ 255, with a **sparse L1-fitting index** for fast lookup.
+
+```
+primary[slot] < 255  →  return primary[slot]
+primary[slot] == 255 →  binary search in overflow
+```
+
+### File format
+
+Single `.pciv` file. Write order: header placeholder → primary → overflow + index → header overwrite at offset 0.
+
+```
+offset 0:
+  magic:      [u8; 4]   = b"PCIV"
+  _pad:       [u8; 4]   = 0
+  n:          u64        number of slots
+  n_overflow: u64        number of overflow entries
+  n_index:    u64        number of sparse index entries
+  step:       u64        sparse index step (0 = no index)
+
+offset 40:
+  primary:    [u8; n]    one byte per slot, 255 = overflow sentinel
+
+offset 40 + n:
+  data:       [(slot: u64, value: u32); n_overflow]   12 bytes each, sorted by slot
+
+offset 40 + n + n_overflow × 12:
+  index:      [(slot: u64, pos: u64); n_index]         16 bytes each, sparse index
+```
+
+The index entries point into `data`: `index[i] = (slot of data[i×step], i×step)`.
+
+All integer fields are little-endian. Slot indices are stored as `u64` in the file; they are `usize` in Rust code.
+
+### Lifecycle
+
+#### Builder (`PersistentCompactIntVecBuilder`)
+
+Used during construction. The primary section is **mmap'd immediately** at construction time (both for `new` and `build_from`), so the file exists and is addressable from the start. The overflow is held in a `HashMap<usize, u32>` in RAM.
+
+```rust
+struct PersistentCompactIntVecBuilder {
+    path:     PathBuf,
+    mmap:     MmapMut,            // primary section live in the file from the start
+    n:        usize,
+    overflow: HashMap<usize, u32>, // values ≥ 255
+}
+```
+
+**`new(n: usize, path: &Path) -> io::Result<Self>`**
+
+Creates the file, pre-allocates `HEADER_SIZE + n` zero bytes, mmaps it. The primary is zero-initialised (all slots = 0). Returns immediately ready for `set` / `get`.
+
+**`build_from(source: &PersistentCompactIntVec, path: &Path) -> io::Result<Self>`**
+
+Copies the source PCIV file to `path` (OS-level copy — no per-slot iteration), mmaps the copy, then loads the overflow section into a `HashMap`. Initialisation cost: O(file copy) + O(n_overflow), not O(n).
+
+At `close()`, the primary section is **not rewritten**: it is already in the file via mmap. Only the overflow data, the sparse index, and the header are updated.
+
+**`set(slot: usize, value: u32)` / `get(slot: usize) -> u32`**
+
+Direct mmap byte access for the primary; HashMap for the overflow. Both O(1). Mutations can move a slot between tiers freely (downward mutation removes the HashMap entry; upward mutation adds it).
+
+**Element-wise operations — `min`, `max`, `add`, `diff`**
+
+Each takes a `&PersistentCompactIntVec` of equal length and updates `self` in place via `set`:
+
+```rust
+builder.min(&other);   // self[i] = min(self[i], other[i])
+builder.max(&other);   // self[i] = max(self[i], other[i])
+builder.add(&other);   // self[i] = self[i].checked_add(other[i])  (panics on u32 overflow)
+builder.diff(&other);  // self[i] = self[i].saturating_sub(other[i])
+```
+
+All iterate `other` with `other.iter()` (merge-scan, O(n_other)).
+
+**`close(self) -> io::Result<()>`**
+
+1. Flush and drop the mmap (primary changes are now on disk).
+2. Sort the overflow HashMap into `Vec<(usize, u32)>`.
+3. Truncate the file to `HEADER_SIZE + n` (removes old data+index if `build_from` was used).
+4. Append sorted overflow data, then sparse index.
+5. Seek to offset 0, overwrite the header with final values.
+
+#### Reader (`PersistentCompactIntVec`)
+
+Used at query time. The whole file is mmap'd; only the sparse index is copied into a `Vec` at open time (≤ 32 KB, L1-resident).
+
+```rust
+struct PersistentCompactIntVec {
+    mmap:           Mmap,
+    n:              usize,
+    n_overflow:     usize,
+    step:           usize,
+    index:          Vec<(usize, usize)>,  // (slot, pos) — L1-resident
+    primary_offset: usize,               // = 40 (HEADER_SIZE)
+    data_offset:    usize,               // = 40 + n
+    path:           PathBuf,
+}
+```
+
+**`open(path: &Path) -> io::Result<Self>`**
+
+Mmaps the file, parses the 40-byte header, copies the sparse index entries into a `Vec`. The primary and data sections stay mmap'd.
+
+**`get(slot: usize) -> u32` — random access**
+
+```
+primary[slot] < 255  →  return it directly
+
+step == 0:
+    binary_search(data[0..n_overflow], slot)
+
+step > 0:
+    i = upper_bound(index[..].slot, slot) − 1     // in L1-resident Vec
+    binary_search(data[index[i].pos .. index[i+1].pos], slot)
+```
+
+**`iter() -> Iter<'_>` — sequential scan, O(n)**
+
+Merge-scan: reads primary bytes in order; on sentinel 255, advances a sequential pointer into the sorted data section rather than doing a binary search. This gives O(n + n_overflow) with no random access into the data section.
+
+`Iter` implements `ExactSizeIterator`. `&PersistentCompactIntVec` implements `IntoIterator`.
+
+**Aggregate**
+
+```rust
+fn sum(&self) -> u64   // Σ self[i] as u64, via iter()
+```
+
+**Distance methods**
+
+All take `&other` of equal length, iterate both with `zip(self.iter(), other.iter())`, and return `f64`.
+
+| Method | Formula |
+|---|---|
+| `bray_dist` | `1 − 2·Σmin(aᵢ,bᵢ) / (Σaᵢ + Σbᵢ)` |
+| `relfreq_bray_dist` | Bray-Curtis on relative frequencies: `1 − Σmin(pᵢ,qᵢ)` where `pᵢ = aᵢ/Σa` |
+| `euclidean_dist` | `√Σ(aᵢ − bᵢ)²` |
+| `relfreq_euclidean_dist` | Euclidean on relative frequencies |
+| `hellinger_euclidean_dist` | `√Σ(√pᵢ − √qᵢ)²` — Euclidean on sqrt(relfreq) |
+| `hellinger_dist` | `hellinger_euclidean_dist / √2` — standard Hellinger distance ∈ [0, 1] |
+| `threshold_jaccard_dist(&other, threshold: u32)` | `1 − \|A∩B\| / \|A∪B\|` where presence iff count ≥ threshold |
+| `jaccard_dist` | `threshold_jaccard_dist(&other, 1)` |
+
+Edge cases (both vectors all-zero, or union empty for Jaccard): distance = 0.0.
+
+### Step computation
+
+Chosen at `close()` once `n_overflow` is known:
+
+```
+L1_INDEX_ENTRIES = 2048
+
+step = 0                                if n_overflow ≤ 2048
+step = ⌈n_overflow / 2048⌉             otherwise
+```
+
+### Complexity
+
+| Operation | Time | Notes |
+|---|---|---|
+| `set` / `get` (builder) | O(1) | mmap byte + HashMap |
+| `get` (reader, no overflow) | O(1) | single mmap byte |
+| `get` (reader, with index) | O(log step) | ≤ 2 memory regions |
+| `get` (reader, no index) | O(log n_overflow) | data fits in a few cache lines |
+| `iter()` full scan | O(n + n_overflow) | merge-scan, no binary search |
+| `sum`, distances | O(n) | via `iter()` / `zip(iter(), iter())` |
+| `min` / `max` / `add` / `diff` | O(n) | via `other.iter()` + builder `set` |
+| `close` | O(n_overflow log n_overflow) | sort + sequential write |
+| `open` | O(n_index) | index copy into Vec |
+| `build_from` | O(file_size) + O(n_overflow) | OS copy + HashMap load |
+
+---
+
+## PersistentCompactIntMatrix — column-major directory
+
+### Design
+
+A directory containing `meta.json` and N column files `col_000000.pciv`, `col_000001.pciv`, …, each a `PersistentCompactIntVec`. This is the type used by `LayerData` — a single-column matrix is functionally equivalent to a vector but shares the same interface as multi-column matrices.
+
+```
+counts/
+  meta.json          {"n": <n_slots>, "n_cols": <N>}
+  col_000000.pciv
+  col_000001.pciv
+  ...
+```
+
+### Builder (`PersistentCompactIntMatrixBuilder`)
+
+```rust
+struct PersistentCompactIntMatrixBuilder {
+    dir:    PathBuf,
+    n:      usize,
+    n_cols: usize,
+}
+```
+
+**`new(n: usize, dir: &Path) -> io::Result<Self>`**
+
+Creates the directory (including parents). Does not write `meta.json` yet.
+
+**`add_col(&mut self) -> io::Result<PersistentCompactIntVecBuilder>`**
+
+Creates `col_NNNNNN.pciv` for the next column and returns its builder. The caller fills the column and calls `builder.close()` before calling `add_col` again.
+
+**`close(self) -> io::Result<()>`**
+
+Writes `meta.json` with the final `n` and `n_cols`. Must be called after all column builders are closed.
+
+### Reader (`PersistentCompactIntMatrix`)
+
+```rust
+struct PersistentCompactIntMatrix {
+    cols: Vec<PersistentCompactIntVec>,
+    n:    usize,
+}
+```
+
+**`open(dir: &Path) -> io::Result<Self>`**
+
+Reads `meta.json`, opens all `col_NNNNNN.pciv` files.
+
+**`row(slot: usize) -> Box<[u32]>`**
+
+Returns the full row: `[col_0[slot], col_1[slot], …, col_{N-1}[slot]]`. One mmap access per column. O(N).
+
+**`col(c: usize) -> &PersistentCompactIntVec`**
+
+Direct access to a single column for column-oriented operations (distance computations, iteration).
+
+### LayerData implementation
+
+```rust
+impl LayerData for PersistentCompactIntMatrix {
+    type Item = Box<[u32]>;
+    fn open(layer_dir: &Path) -> OLMResult<Self> { /* opens layer_dir/counts/ */ }
+    fn read(&self, slot: usize) -> Box<[u32]>    { self.row(slot) }
+}
+```
+
+---
+
+## Aggregation traits — `obicompactvec::traits`
+
+`PersistentCompactIntMatrix` implements two aggregation traits used by `LayeredStore<S>` for cross-layer and cross-partition distance computations.
+
+### ColumnWeights
+
+```rust
+impl ColumnWeights for PersistentCompactIntMatrix {
+    fn col_weights(&self) -> Array1<u64>   // = self.sum()
+}
+```
+
+`col_weights()[c]` = sum of all values in column `c` across all slots.
+
+### CountPartials
+
+```rust
+impl CountPartials for PersistentCompactIntMatrix {
+    // Self-contained partials (additive across layers, no external parameter)
+    fn partial_bray(&self)                                      -> Array2<u64>
+    fn partial_euclidean(&self)                                 -> Array2<f64>
+    fn partial_threshold_jaccard(&self, threshold: u32)         -> (Array2<u64>, Array2<u64>)
+
+    // Normalised partials (require global col_weights across all layers/partitions)
+    fn partial_relfreq_bray(&self, global: &Array1<u64>)        -> Array2<f64>
+    fn partial_relfreq_euclidean(&self, global: &Array1<u64>)   -> Array2<f64>
+    fn partial_hellinger(&self, global: &Array1<u64>)           -> Array2<f64>
+
+    // Provided finalisations (default implementations on the trait)
+    fn bray_dist_matrix(&self)                                  -> Array2<f64>
+    fn euclidean_dist_matrix(&self)                             -> Array2<f64>
+    fn threshold_jaccard_dist_matrix(&self, threshold: u32)     -> Array2<f64>
+    fn relfreq_bray_dist_matrix(&self)                          -> Array2<f64>
+    fn relfreq_euclidean_dist_matrix(&self)                     -> Array2<f64>
+    fn hellinger_dist_matrix(&self)                             -> Array2<f64>
+}
+```
+
+**Self-contained partials** are additively decomposable: summing `partial_bray()` across all `(partition, layer)` pairs and finalising gives the same result as computing on the combined data.
+
+**Normalised partials** require the global column weights (sum across all layers and all partitions). The `global` parameter must reflect the complete index, not a per-layer sum. The provided `relfreq_bray_dist_matrix()` etc. call `col_weights()` first (pass 1) then the normalised partial (pass 2); when called on a `LayeredStore<LayeredStore<…>>` these two-pass calls cascade automatically through the blanket impls.
+
+**`partial_bray` returns `Array2<u64>`** (sum_min only, not a tuple). The denominator is always reconstructible as `col_weights()[i] + col_weights()[j]`.
+
+**`partial_threshold_jaccard` returns `(inter, union)`** as a pair because `union[i,j]` is not reconstructible from per-column statistics — it depends on both columns simultaneously.

DevDocMD/implementation/persistent_compact_int_vec.refs.md

@@ -0,0 +1,14 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/persistent_compact_int_vec.md
+
+## Code couvert
+
+- `obicompactvec/src/builder.rs` — PersistentCompactIntVecBuilder, cycle de vie
+- `obicompactvec/src/reader.rs` — PersistentCompactIntVec, accès aléatoire et séquentiel
+- `obicompactvec/src/intmatrix.rs` — PersistentCompactIntMatrix, wrapper colonne-major, append_column
+- `obicompactvec/src/format.rs` — format de fichier (magic PCIV, header, primary u8, overflow, index)
+
+## Notes
+
+Document d'implémentation stable. Vérifier que `append_column` (utilisé dans merge et reindex)
+est décrit. Vérifier que `PersistentCompactIntMatrixBuilder` est couvert (utilisé dans `layer.rs`).

DevDocMD/implementation/pipeline.md

@@ -0,0 +1,228 @@
+# Construction pipeline
+
+All phases after scatter are embarrassingly parallel across partitions.
+
+## Phase 0 — Parameter estimation
+
+The construction parameters p, n, and min_count depend on the kmer frequency spectrum of the dataset. Estimating this spectrum before construction avoids costly re-partitioning if p is badly chosen.
+
+Two approaches are supported:
+
+- **External estimation (preferred):** run [NT-CARD](https://github.com/bcgsc/ntCard) on the input files and pass its histogram output to `obikmer build`. NT-CARD produces a kmer frequency histogram in a single streaming pass using ntHash and a Flajolet-Martin-style estimator; obikmer reads this file and derives p, n, and min_count automatically.
+- **Internal estimation (future):** an `obikmer estimate` subcommand for users who prefer a single-tool workflow. The implementation would combine two components: (1) **ntHash**, a rolling hash that updates the kmer hash in O(1) per nucleotide by incrementally adding the incoming base and removing the outgoing one — Rust crates exist; (2) a **Flajolet-Martin-style streaming estimator** that maintains a small table of minimum hash values and infers the frequency histogram from their statistical distribution, as described in the NT-CARD paper [@Mohamadi2017-ok].
+
+The histogram gives:
+
+- **F0** (number of distinct kmers) → sets p (target ~10M kmers/partition → p = ⌈log₂(F0 / 10M)⌉)
+- **frequency distribution** → sets n (choose n so that fewer than 1% of kmers overflow)
+- **error valley** → suggests min_count (typically the local minimum between the error peak and the coverage peak)
+
+## Phase 1 — Scatter
+
+Single streaming pass over raw input files (FASTA/FASTQ, gzip). FASTQ quality scores are ignored.
+
+Input files are read via `open_nuc_stream`, which opens and decompresses the file, auto-detects the format (FASTA / FASTQ / GenBank), and yields a sequence of `NucPage` buffers. Each `NucPage` is a flat 64 KB buffer of normalised bytes (`ACGT` + `\x00` separators), carrying a k−1 byte overlap from the preceding page so that no k-mer is lost at page boundaries. Per-record identity (sequence id, raw bytes) is not preserved; this is intentional — the scatter phase only needs normalised bases to produce superkmers.
+
+For each read fragment within a page:
+
+1. **Ambiguous base filter**: cut at any non-ACGT base; discard fragments shorter than k.
+2. **Entropy filter**: scan each fragment with a sliding window of size k. When the kmer $K_i = S[i \mathinner{..} i+k-1]$ ended by nucleotide $S[j]$ (with $j = i+k-1$) has entropy below threshold $\theta$, emit the current segment and start a new one (see algorithm below). $K_i$ belongs to neither segment, and no valid kmer is lost.
+3. **Length filter**: discard any segment shorter than k produced by step 2.
+4. **Super-kmer extraction**: for each clean segment, slide a minimizer window and group consecutive kmers sharing the same canonical minimizer; canonise each super-kmer by lexicographic comparison with its reverse complement (early exit).
+5. **Partition routing**: `hash(canonical_minimizer) → PART` → append super-kmer to `partition/superkmers.bin.gz`.
+
+**Segmentation behavior:**
+
+When $K_i$ (ended by $S[j]$, $j = i+k-1$) fails the entropy threshold:
+
+- Current segment $S[\textit{seg_start} \mathinner{..} j-1]$ is emitted (last valid kmer = $K_{i-1}$)
+- New segment starts at $S[i+1]$ (first new kmer = $K_{i+1}$)
+- $K_i$ is excluded: current segment lacks $S[j]$, new segment lacks $S[i]$
+- Overlap = $S[i+1 \mathinner{..} j-1]$ = $k-2$ nucleotides
+
+!!! abstract "Algorithm — Entropy filter: sliding window segmentation"
+    ```text
+    procedure EntropyFilter(S, N, k, θ):
+        seg_start ← 0
+        window ← []
+        for j ← 0 to N−1:
+            window.push(S[j])
+            if |window| < k: continue
+            i ← j − k + 1
+            if entropy(window) ≤ θ:
+                emit S[seg_start .. j−1]
+                seg_start ← i + 1
+                window ← S[i+1 .. j]
+            else:
+                window.pop_front()
+        emit S[seg_start .. N−1]
+    ```
+
+Writes are sequential and append-only — IO-friendly. Gzip applied at write time. Data volume ≈ raw genome size (2 bits/nt compaction offsets header overhead).
+
+## Phase 2 — Dereplication
+
+Performed independently per partition. Identical super-kmers are consolidated and their COUNT accumulated — analogous to amplicon dereplication in metabarcoding. Uses external bucket sort to stay within RAM bounds:
+
+**Pass 1** (streaming): hash the nucleotide payload of each super-kmer, route to one of B bucket files:
+```
+hash(sequence) % B → bucket_i.bin
+```
+B ≈ 100 is tunable; RAM needed ≈ partition_size / B.
+
+**Pass 2**: for each bucket, load into an in-memory `HashMap<sequence, COUNT>`, dereplicate by summing COUNT values, write consolidated super-kmers.
+
+After dereplication: at Nx coverage the partition shrinks by ~x (errors aside). The COUNT field in each super-kmer header = number of times that exact super-kmer sequence was observed across all input reads.
+
+**Important:** super-kmer COUNT ≠ individual kmer count. A kmer can appear in multiple distinct super-kmers (same partition, different flanking context); its true count = sum of COUNT of all super-kmers containing it. A super-kmer with COUNT=1 may contain only high-abundance kmers, each appearing in many other super-kmers. Abundance filtering therefore cannot be applied at this phase.
+
+## Phase 3 — Per-kmer count aggregation and quorum filtering
+
+For each dereplicated super-kmer, enumerate its kmers and accumulate counts:
+
+```
+for each super-kmer (sequence, COUNT):
+    for each kmer in sequence:
+        kmer_counts[canonical(kmer)] += COUNT
+```
+
+Implemented as a three-step pipeline in `count_partition()`:
+
+1. **External sort** (`kmer_sort::sort_unique_kmers`): read dereplicated superkmers, extract canonical kmer raw `u64` values, sort in RAM-bounded chunks (adaptive: 40% of available RAM ÷ n_threads, min 1 M kmers/chunk), k-way merge with inline dedup → `sorted_unique.bin`. f0 is now known exactly.
+2. **Provisional MPHF** (ptr_hash): built from `sorted_unique.bin` via `new_from_par_iter(f0, ...)`. Stored to `mphf1.bin`; `sorted_unique.bin` deleted immediately.
+3. **Accumulation pass**: re-read dereplicated superkmers; for each kmer, `slot = mphf.index(kmer.raw())`, increment `counts1[slot]` by the superkmer COUNT. Stored in a `PersistentCompactIntVec` (`counts1.bin`).
+
+At the end of this phase, each distinct canonical kmer has its exact total count, and the frequency spectrum (`spectrums/{label}.json`) is written to the index root.
+
+No pre-filter on super-kmer COUNT is possible at phase 2: a super-kmer with COUNT=1 may contain only high-abundance kmers, each present in many other super-kmers across the partition.
+
+## Phase 4 — Super-kmer compaction
+
+The valid kmer set from phase 3 is used as a mask to rewrite the super-kmer files:
+
+```
+for each dereplicated super-kmer:
+    scan kmer by kmer
+    kmer not in valid set → break point (terminates current super-kmer)
+    kmer in valid set     → extend current super-kmer
+```
+
+Three cases per super-kmer:
+
+- **All kmers valid** → copied as-is
+- **No kmer valid** → discarded
+- **Mixed** → split into sub-super-kmers at invalid boundaries; each sub-super-kmer inherits the original COUNT
+
+After splitting, re-apply dereplication (bucket sort, phase 2 method) — splitting can produce new identical super-kmers. This re-dereplication is cheap: the volume is already greatly reduced.
+
+Output: a clean super-kmer file where every kmer passes quorum. This file feeds phase 5.
+
+## Phase 5 — Local de Bruijn graph and unitig construction
+
+Within each partition, build a **local de Bruijn graph** from the valid kmer set and compute its unitigs. All operations are local to the partition — no cross-partition communication.
+
+```
+valid kmers → HashSet<u64>
+
+for each kmer K:
+    out_degree = |{K[1:]+b | b ∈ {A,C,G,T}} ∩ HashSet|
+    in_degree  = |{b+K[:-1] | b ∈ {A,C,G,T}} ∩ HashSet|
+
+internal node ↔ in_degree=1 AND out_degree=1
+branching / dead-end → unitig start or end
+```
+
+Traverse non-branching paths to assemble unitigs. Kmers whose neighbours fall in other partitions appear as dead ends locally — they terminate the unitig. The result: **each kmer appears in exactly one unitig** within the partition.
+
+The partition size (controlled by p) must be calibrated so that the HashSet fits in RAM during this phase.
+
+Output: `unitigs.bin` — the permanent evidence structure for the partition. Each kmer in the partition appears at exactly one (unitig_id, offset) location.
+
+**Scope of local unitigs:** these are unitigs of the partition's local de Bruijn graph, not global unitigs. A kmer whose k-1 successor or predecessor falls in another partition appears as a dead end locally and terminates the unitig. This does not affect correctness of verification but means partition-local unitigs cannot be directly reused for global assembly.
+
+## Phase 6 — MPHF construction and index finalisation
+
+`build_index_layer` is called per partition (in parallel via `build_layers`) with the following parameters sourced from `IndexConfig`:
+
+- `block_bits` — from `IndexConfig::block_bits`; controls the `.idx` block size (2^block_bits unitig chunks per block) for exact evidence
+- `evidence` — `EvidenceKind::Exact` or `EvidenceKind::Approx { b, z }`; propagated unchanged from `IndexConfig::evidence`
+- `min_ab` / `max_ab` — abundance bounds applied before graph construction
+- `with_counts` — whether to store kmer counts alongside set membership
+
+**Abundance filtering:** when `min_ab > 1` or `max_ab.is_some()`, the provisional `mphf1.bin` and `counts1.bin` produced in phase 3 are memory-mapped. Each canonical kmer is accepted only if its count in `counts1` satisfies the bounds. If either file is absent, filtering is skipped (all kmers accepted).
+
+```
+for each kmer in dereplicated super-kmer:
+    ab = counts1[mphf1.index(kmer.raw())]
+    if ab < min_ab || ab > max_ab: skip
+    graph.push(kmer)
+```
+
+**Graph build and unitig write:**
+
+The surviving kmers are fed into `GraphDeBruijn`, which computes degrees and yields unitigs. Unitigs are written to `layer_0/unitigs.bin` via a `UnitigFileWriter`.
+
+**MPHF and evidence build:**
+
+`Layer::build` (membership-only) or `Layer::<PersistentCompactIntMatrix>::build` (with counts) is called next. Internally, `MphfLayer::build` performs two passes:
+
+1. **Pass 1 (parallel):** build `unitigs.bin.idx` (block size = 2^`block_bits`) then construct the MPHF from all canonical kmers in `unitigs.bin`; store to `mphf.bin`.
+2. **Pass 2 (sequential):** for each kmer in `unitigs.bin`, compute its slot and write `evidence.bin` (`chunk_id: 25 bits | rank: 7 bits` packed into a `u32`); also invoke the payload callback (`fill_slot`) to populate `counts/` if `with_counts`.
+
+After `Layer::build` completes, `layer_meta.json` records `EvidenceKind::Exact`.
+
+**Approximate evidence override:**
+
+If `evidence` is `EvidenceKind::Approx { b, z }`, `build_approx_evidence` is called immediately after `Layer::build`. It overwrites the exact evidence bundle with `fingerprint.bin` (b-bit hash per slot) and rewrites `layer_meta.json` with `EvidenceKind::Approx { b, z }`. No `.idx` file is needed at query time in this mode.
+
+```
+// Exact path  →  evidence.bin + unitigs.bin.idx + layer_meta.json(Exact)
+// Approx path →  fingerprint.bin + layer_meta.json(Approx{b,z})
+//               (evidence.bin left on disk but not used)
+```
+
+**Partition metadata:**
+
+After all layer files are written, `PartitionMeta { n_layers: 1 }` is serialised to `index/meta.json` inside the partition directory. This file is required by `LayeredMap::open` for subsequent merge operations.
+
+**File layout per partition after phase 6:**
+
+```
+part_XXXXX/
+  index/
+    meta.json                ← PartitionMeta { n_layers: 1 }
+    layer_0/
+      unitigs.bin            ← permanent evidence (all modes)
+      unitigs.bin.idx        ← block index (exact mode only)
+      mphf.bin               ← MPHF
+      evidence.bin           ← exact evidence (exact mode)
+      fingerprint.bin        ← b-bit fingerprints (approx mode)
+      layer_meta.json        ← EvidenceKind tag
+      counts/                ← PersistentCompactIntMatrix (with_counts only)
+```
+
+**Cleanup:** unless `--keep-intermediate` is set, `remove_build_artifacts` deletes `dereplicated.skmer.zst`, `mphf1.bin`, and `counts1.bin` after all partitions are indexed.
+
+See [obilayeredmap](obilayeredmap.md) and [MPHF selection](mphf.md) for data structure details.
+
+**Query path (exact evidence):**
+
+```
+query kmer q
+  → canonical_minimizer(q) → hash → PART → part_XXXXX/
+  → MPHF(q) → slot s
+  → evidence[s] = (chunk_id, rank)
+  → read k nucleotides at rank in unitigs[chunk_id] → compare with q
+  → match   : return payload[s]   ← exact hit
+  → no match: kmer absent         ← MPHF collision on absent kmer
+```
+
+**Query path (approximate evidence):**
+
+```
+query kmer q
+  → MPHF(q) → slot s
+  → fingerprint[s] matches seq_hash(q)?
+  → yes : probable hit (FP rate = 1/2^b per kmer, 1/2^(b·z) per z-window)
+  → no  : kmer absent
+```

DevDocMD/implementation/pipeline.refs.md

@@ -0,0 +1,19 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/pipeline.md
+
+## Code couvert
+
+- `obikpartitionner/src/partition.rs` — estimation des paramètres (phase 0)
+- `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
+- `obikpartitionner/src/filter.rs` — déduplication bucket-sort (phase 2)
+- `obikpartitionner/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
+- `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
+- `obikpartitionner/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
+- `obikindex/src/index.rs` — `build_layers()`, `dereplicate_and_count()`
+
+## Notes
+
+RISQUE DE DÉRIVE modéré. Vérifier :
+- Phase 6 : la doc mentionne-t-elle le filtre d'abondance (`min_ab`, `max_ab`) ?
+- Phase 6 : `block_bits` passé à `build_index_layer` depuis `IndexConfig`
+- Phase 6 : dispatch exact/approx selon `EvidenceKind` dans `build_index_layer`

DevDocMD/implementation/select.md

@@ -0,0 +1,234 @@
+# `select` — column projection and aggregation
+
+`select` transforms an index by operating on its **genome columns**: projecting a
+subset of columns, aggregating groups of genomes into synthetic columns, or both.
+It is the column-axis counterpart of `filter` (row-axis operations).
+
+Following relational algebra conventions:
+
+| Command  | Relational operation | Axis     |
+|----------|---------------------|----------|
+| `filter` | σ — selection       | rows (k-mers) |
+| `select` | π — projection      | columns (genomes) |
+
+The two commands compose naturally: run `filter` first to restrict the kmer set,
+then `select` to reshape the genome columns.
+
+`select` never changes the kmer set. The MPHF and `unitigs.bin` of each layer
+are preserved unchanged; only the data matrices are rewritten.
+
+---
+
+## Synopsis
+
+```sh
+obikmer select <input-index>
+        { --output <dir> | --in-place }
+        [--group    <name>:<pred>  ...]
+        [--group-op <name>:<op>    ...]
+        [--aggregate-by <key>          ]
+        [--aggregate-op <op>           ]
+        [--select   <col1,col2,...>    ]
+        [--presence-threshold <N>      ]
+```
+
+---
+
+## Output destination
+
+Exactly one of `--output` or `--in-place` must be specified.
+
+**`--output <dir>`** — writes a new index to `<dir>`. The source index is
+unchanged. The MPHF and unitig files are copied; only the data matrices are
+rewritten with the new column layout.
+
+**`--in-place`** — rewrites the data matrices of the source index directly.
+Removed or replaced columns are lost. The operation writes to temporary files
+first, then renames atomically, so an interrupted run leaves the index intact.
+
+---
+
+## Defining output columns
+
+### Named groups — `--group`
+
+```
+--group <name>:<pred>
+```
+
+Defines a named group of genomes using the same predicate syntax as `filter`.
+Repeatable; a genome can belong to several groups.
+
+```sh
+--group "pub:species=Betula_pubescens"
+--group "nan:species=Betula_nana"
+```
+
+### Per-group operator — `--group-op`
+
+```
+--group-op <name>:<op>
+```
+
+Assigns an aggregation operator to a named group. Optional; if absent, the
+default operator applies (see below).
+
+```sh
+--group-op "pub:any"
+--group-op "nan:all"
+```
+
+### Shorthand — `--aggregate-by` / `--aggregate-op`
+
+`--aggregate-by <key>` automatically creates one group per unique value of the
+metadata key `<key>`. Equivalent to one `--group <val>:<key>=<val>` per distinct
+value. `--aggregate-op <op>` sets the operator for all auto-generated groups.
+
+`--aggregate-by` and `--group` are mutually exclusive.
+
+### Column selection and ordering — `--select`
+
+```
+--select col1,col2,...
+```
+
+Lists the output columns in order. Each element is either a group name (defined
+by `--group` or generated by `--aggregate-by`) or a genome label from the source
+index (pass-through, no aggregation).
+
+**Default when `--select` is absent:**
+all defined groups in declaration order (for `--group`), or all generated groups
+in metadata-value order (for `--aggregate-by`). Individual genomes not in any
+group are excluded unless named explicitly.
+
+**When neither `--group` nor `--aggregate-by` is specified:**
+`--select` can still reference genome labels for pure column projection (no
+aggregation). If `--select` is also absent, all genomes are output unchanged
+(identity transform — useful combined with row filtering via a prior `filter`
+run).
+
+---
+
+## Aggregation operators
+
+| Operator | Input       | Output   | Semantics |
+|----------|-------------|----------|-----------|
+| `any`    | pres / count | presence | 1 if ≥ 1 genome in group carries the k-mer |
+| `all`    | pres / count | presence | 1 if every genome in group carries the k-mer |
+| `none`   | pres / count | presence | 1 if no genome in group carries the k-mer |
+| `sum`    | count        | count    | sum of counts across the group |
+| `min`    | count        | count    | minimum count |
+| `max`    | count        | count    | maximum count |
+
+**Default operator:**
+- Presence index: `any`
+- Count index: `sum`
+
+Logical operators (`any`/`all`/`none`) on a count index use
+`--presence-threshold N` (default 0): a genome "carries" the k-mer if its count
+is > N.
+
+**Output index type:**
+- If the source is a presence index, the output is always a presence index.
+- If the source is a count index and every output column uses a logical operator
+  or is a pass-through from a presence source, the output is a presence index.
+- Otherwise (at least one arithmetic operator on a count source), the output is
+  a count index.
+
+---
+
+## Behaviour for edge cases
+
+| Situation | Behaviour |
+|-----------|-----------|
+| Genome missing the metadata key in `--aggregate-by` | genome ignored (no `NA` group) |
+| Genome in multiple groups | contributes independently to each |
+| `--group-op` references undefined group | error |
+| `--select` element is neither group name nor genome label | error |
+| `--output` and `--in-place` both specified | error |
+| Neither `--output` nor `--in-place` | error |
+| Group with zero matching genomes | column is all-zeros (or all-ones for `none`) |
+
+---
+
+## Examples
+
+### Aggregate by metadata group, default operators
+
+```sh
+obikmer select myindex --output out --aggregate-by group
+# one column per unique value of "group"; presence→any, count→sum
+```
+
+### Named groups with different operators
+
+```sh
+obikmer select myindex --output out \
+  --group    "pub:species=Betula_pubescens" \
+  --group    "nan:species=Betula_nana" \
+  --group-op "pub:any" \
+  --group-op "nan:all" \
+  --select   "pub,nan"
+```
+
+### Mix aggregated group and individual genome
+
+```sh
+obikmer select myindex --output out \
+  --group  "A:group=A" \
+  --select "A,Betula_nana--IGA-24-39"
+```
+
+### Pure column projection (no aggregation)
+
+```sh
+obikmer select myindex --output out \
+  --select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
+```
+
+### In-place: keep only group A
+
+```sh
+obikmer select myindex --in-place --group "A:group=A" --select "A"
+```
+
+### Compose with filter
+
+```sh
+# Step 1: keep only B. nana-specific k-mers
+obikmer filter myindex --output filtered \
+  --ingroup "species=Betula_nana" --outgroup "*"
+
+# Step 2: aggregate genome columns by collection site
+obikmer select filtered --output final --aggregate-by site
+```
+
+---
+
+## Implementation notes
+
+`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
+(rayon), each writing its output independently; results require no synchronisation
+because every partition owns a distinct set of files.
+
+For each layer in each partition:
+
+1. The slot count `n` is read by opening the source data matrix.
+2. A new data matrix is built with M columns (M = number of output columns).
+3. For each slot `s` in `0..n`:
+   - `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
+   - For each output column `j`:
+     - `new_row[j] = aggregate(op, old_row[group_indices])`.
+     - Pass-through columns are represented as single-element groups with the
+       default operator (`any` for presence, `sum` for count) — same code path.
+   - The new row is written slot by slot into each column builder.
+4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
+   evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
+   or `counts/` subdirectory is rewritten.
+5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
+
+**`--in-place` write strategy:** new data is written to a temporary sibling
+directory (`presence_new/` or `counts_new/`); on success the old directory is
+removed and the temporary one is renamed into place. An interrupted run leaves
+at most one stale `*_new/` directory; the original data is intact until the
+rename step.

DevDocMD/implementation/storage.md

@@ -0,0 +1,136 @@
+# On-disk index layout
+
+## Directory tree
+
+```
+<index_root>/
+  index.meta                  ← JSON: IndexMeta
+  scatter.done                ← sentinel: scatter phase complete
+  count.done                  ← sentinel: dereplicate + count complete
+  index.done                  ← sentinel: MPHF index fully built
+  spectrums/
+    <label>.json              ← kmer frequency spectrum per genome
+  partitions/
+    part_00000/               ← one dir per partition (zero-padded 5 digits, 0..2^n_bits−1)
+      index/
+        meta.json             ← PartitionMeta { n_layers }
+        layer_0/
+          unitigs.bin         ← binary unitig sequences (2-bit packed)
+          unitigs.bin.idx     ← block-sampled offset index (exact evidence only)
+          mphf.bin            ← serialised PtrHash MPHF
+          layer_meta.json     ← LayerMeta { evidence: EvidenceKind }
+          evidence.bin        ← chunk_id:rank per MPHF slot (Exact only)
+          fingerprint.bin     ← b-bit fingerprints per MPHF slot (Approx only)
+          counts/             ← PersistentCompactIntMatrix (if with_counts=true)
+          presence/           ← PersistentBitMatrix (if presence mode, merge)
+        layer_1/              ← added by merge; same structure as layer_0
+        layer_2/ …
+    part_00001/ …
+```
+
+## State machine (sentinels)
+
+The sentinels are touched atomically at the end of each pipeline stage.
+A partial run (e.g. scatter interrupted) leaves no sentinel; the state is
+detected as the lowest sentinel present.
+
+| State | Sentinel present | Meaning |
+|---|---|---|
+| `Empty` | — | `index.meta` exists; scatter not started or interrupted |
+| `Scattered` | `scatter.done` | All super-kmers routed to partition files |
+| `Counted` | `count.done` | Partitions dereplicated; `spectrums/` written |
+| `Indexed` | `index.done` | All MPHF layers built; index ready for queries |
+
+## index.meta (IndexMeta)
+
+```json
+{
+  "version": 1,
+  "config": {
+    "kmer_size": 31,
+    "minimizer_size": 11,
+    "n_bits": 8,
+    "with_counts": false,
+    "evidence": "Exact",
+    "block_bits": 0
+  },
+  "genomes": [
+    { "label": "genome_A", "meta": { "species": "Homo sapiens" } }
+  ]
+}
+```
+
+`n_bits` determines the partition count: `2^n_bits` directories under `partitions/`.
+
+`evidence` is either the string `"Exact"` or `{"Approx": {"b": 8, "z": 1}}`.
+
+`block_bits` controls the `.idx` granularity: one offset entry every `2^block_bits`
+chunks. `block_bits=0` stores one entry per chunk (O(1) random access, largest `.idx`).
+
+`GenomeInfo.meta` is a free-form string→string map for categorical metadata (e.g.
+taxonomy, sample origin). It is optional; defaults to empty.
+
+## Layer files
+
+### unitigs.bin
+
+2-bit packed binary unitig sequences. Each record: 1 byte `seql_minus_k`
+(nucleotide length − k), followed by `ceil((seql_minus_k + k) / 4)` bytes of
+packed sequence. Long unitigs are transparently split into overlapping chunks
+(k−1 nucleotide overlap) so no k-mer crosses a chunk boundary.
+
+### unitigs.bin.idx (Exact only)
+
+Magic `UIX3`, little-endian header: `block_bits` (u32), `n_unitigs` (u32),
+`n_kmers` (u64), then `ceil(n_unitigs / 2^block_bits) + 1` byte-offset entries
+(u32 each, last entry is a sentinel past-end offset). Absent for Approx layers.
+
+### mphf.bin
+
+PtrHash MPHF serialised with epserde. Maps canonical kmer (u64, left-aligned
+2-bit) to a slot index in `[0, n_kmers)`.
+
+### layer_meta.json (LayerMeta)
+
+```json
+{ "evidence": { "type": "exact" } }
+```
+or
+```json
+{ "evidence": { "type": "approx", "b": 8, "z": 1 } }
+```
+
+### evidence.bin (Exact)
+
+One `(chunk_id: u32, rank: u8)` record per MPHF slot, packed. Used to verify
+that the kmer mapped to a slot is actually present: `unitigs.bin[chunk_id][rank]`
+is re-read and compared against the query.
+
+### fingerprint.bin (Approx)
+
+`b`-bit fingerprint per MPHF slot derived from the kmer's sequence hash.
+False-positive rate per query ≈ `1/2^b`. With Findere parameter `z ≥ 2`,
+`z` consecutive k-mers must all match, reducing the effective FP rate to
+approximately `W / 2^(b·z)` per read of length `L`
+(where `W = L − k − z + 2`).
+
+### counts/ (PersistentCompactIntMatrix)
+
+Present when `with_counts=true`. One column per genome; each row holds the
+per-genome k-mer count for the corresponding MPHF slot. Appended column-by-column
+during indexing and merge.
+
+### presence/ (PersistentBitMatrix)
+
+Present when the layer was built in presence/absence mode (merge path).
+One bit per genome per MPHF slot. Written during merge; never present on a
+freshly indexed single-genome layer.
+
+## meta.json (PartitionMeta)
+
+```json
+{ "n_layers": 2 }
+```
+
+Records how many `layer_N/` directories exist under `index/`. Incremented by
+each merge that adds a layer.

DevDocMD/implementation/storage.refs.md

@@ -0,0 +1,18 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/storage.md
+
+## Code couvert
+
+- `obikindex/src/meta.rs` — IndexMeta, IndexConfig (version, config, genomes)
+- `obikindex/src/index.rs` — layout sur disque : partitions/, index.meta
+- `obilayeredmap/src/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
+- `obiskio/src/unitig_index.rs` — fichiers unitigs.bin + unitigs.bin.idx
+
+## Notes
+
+FORT RISQUE DE DÉRIVE. Nombreux champs ajoutés :
+- `IndexConfig` : champs `evidence` (EvidenceKind) et `block_bits` ajoutés
+- Nouveau fichier `fingerprint.bin` pour l'évidence approximative
+- `LayerMeta` / `layer_meta.json` introduit pour stocker EvidenceKind par layer
+- Structure du répertoire layer : `evidence.bin` vs `fingerprint.bin` selon le mode
+Mettre à jour le schéma de layout sur disque en conséquence.

DevDocMD/implementation/superkmer.md

@@ -0,0 +1,171 @@
+# SuperKmer — implementation
+
+## Memory layout
+
+`SuperKmer` holds two separate fields:
+
+```rust
+pub struct SuperKmer {
+    pub(crate) count: u32,
+    pub(crate) inner: PackedSeq,
+}
+```
+
+`PackedSeq` stores a 2-bit packed DNA sequence as a heap-allocated `Box<[u8]>` plus a `tail: u8` field:
+
+| Field | Type | Role |
+|-------|------|------|
+| `tail` | `u8` | Number of valid nucleotides in the last byte: 0 encodes 4, 1–3 are identity |
+| `seq`  | `Box<[u8]>` | 2-bit packed bytes, nucleotide 0 at bits 7–6 of `seq[0]` |
+
+Nucleotide length is recovered without storing it explicitly:
+
+```text
+seql = (seq.len() - 1) * 4 + tail_count(tail)
+```
+
+There is no packed header word — `count` and the sequence live in separate fields.
+
+The on-disk binary format (produced by `write_to_binary`) is:
+
+```text
+[varint(count)] [u8: seql − k] [packed bytes…]
+```
+
+`seql − k` fits in a `u8` when `n_kmers = seql − k + 1 ≤ MAX_KMERS_PER_CHUNK (= 256)`. If a super-kmer exceeds 256 kmers, `write_to_binary` splits it into overlapping chunks (k−1 nucleotide overlap, same count per chunk), each a self-contained record readable by `read_from_binary`.
+
+The public accessors operate on the struct fields directly:
+
+```rust
+fn seql(&self)            -> usize { self.inner.seql() }
+fn count(&self)           -> u32   { self.count }
+fn increment(&mut self)            { self.count += 1; }
+fn add(&mut self, n: u32)          { self.count += n; }
+fn set_count(&mut self, n: u32)    { self.count = n; }
+```
+
+## ASCII encoding and decoding
+
+Two lookup tables handle ASCII ↔ 2-bit conversion:
+
+- **`ENC: [u8; 32]`** — indexed by `b & 0x1F` (lower 5 bits of the ASCII byte). Maps A/a→0, C/c→1, G/g→2, T/t and U/u→3; ambiguous bases and unknowns silently map to 0 (A). 32 entries, fits entirely in L1 cache. Upper- and lowercase are handled identically.
+- **`DEC4: [u32; 256]`** — maps a packed byte (4 nucleotides) to 4 ASCII characters packed as a big-endian `u32`. 1 KB total, fits in L1 cache. One lookup per output byte yields 4 decoded characters.
+
+Encoding 4 nucleotides into one byte:
+
+```rust
+byte = ENC[c0 & 0x1F] << 6 | ENC[c1 & 0x1F] << 4 | ENC[c2 & 0x1F] << 2 | ENC[c3 & 0x1F]
+```
+
+Decoding one byte into 4 ASCII characters:
+
+```rust
+DEC4[byte].to_be_bytes()   // [nuc0, nuc1, nuc2, nuc3] in ASCII
+```
+
+## Reverse complement
+
+The reverse complement is computed **in place** with zero allocation in two steps.
+
+**Step 1 — byte swap with `REVCOMP4`.**  A 256-byte lookup table `REVCOMP4` maps each byte (4 nucleotides) to its reverse complement. Bytes are swapped from the outside in, applying `REVCOMP4` to each:
+
+```rust
+const fn revcomp4(x: u8) -> u8 {
+    let x = !x;                                       // complement all bases
+    let x = (x >> 4) | (x << 4);                     // swap nibbles
+    let x = ((x >> 2) & 0x33) | ((x & 0x33) << 2);  // swap 2-bit groups
+    x
+}
+```
+
+`REVCOMP4` is 256 bytes (fits in L1 cache), computed at compile time. No endianness dependency — all operations are pure arithmetic on byte values.
+
+**Step 2 — realignment.**  After step 1, `padding = n × 8 − seql × 2` spurious bits (complements of the original padding A's) appear at the start of the array. They are flushed left using `BitSlice<u8, Msb0>::rotate_left(padding)` from the `bitvec` crate, which is SIMD-accelerated. The trailing `padding` bits are then zeroed:
+
+```rust
+let seql = self.seql();
+shift = n * 8 - seql * 2          // number of padding bits
+bits.rotate_left(shift)
+bits[len - shift..].fill(false)
+```
+
+`Msb0` ordering makes the bit layout hardware-independent.
+
+!!! abstract "Algorithm — Super-kmer canonisation"
+    ```text
+    procedure SuperKmerCanonical(seq, SEQL):
+        for i ← 0 to SEQL − 1:
+            fwd ← nucleotide(seq, i)
+            rev ← complement(nucleotide(seq, SEQL − 1 − i))
+            if fwd < rev: return seq                          -- forward is canonical
+            if fwd > rev: return SuperKmerRevcomp(seq, SEQL) -- revcomp is canonical
+        return seq                                            -- palindrome: either orientation valid
+    ```
+
+## Minimizer sliding window
+
+Super-kmers are built by `SuperKmerIter` (crate `obiskbuilder`), which tracks the current minimizer with a **monotonic deque** (`Ring<MmerItem, 32>`) inside `RollingStat`, a rolling-window entropy and minimizer tracker.
+
+Each deque entry stores:
+
+| Field      | Type  | Purpose                                      |
+|------------|-------|----------------------------------------------|
+| `position` | usize | 0-based start of this m-mer in the segment   |
+| `canonical`| u64   | right-aligned canonical m-mer value (lex-min of fwd and rc); used as partition key |
+| `hash`     | u64   | `hash_kmer(canonical << (64 − 2m))` — ordering key for random minimizer selection |
+
+The hash uses the seeded splitmix64 finalizer (`mix64(raw ^ 0x9e3779b97f4a7c15)`), the same function as `kmer::hash_kmer`.
+
+On each new nucleotide, once the window is full, the deque is updated:
+
+!!! abstract "Algorithm — minimizer deque update"
+    ```text
+    procedure UpdateMinimizer(deque, position, canonical, hash, k, received):
+        -- pop dominated entries from the back
+        while deque.back.hash ≥ hash:
+            deque.pop_back()
+        deque.push_back({position, canonical, hash})
+
+        -- evict expired entries from the front
+        while deque.front.position + k < received:
+            deque.pop_front()
+    ```
+
+The front of the deque is always the current minimizer. Because the deque is maintained in strictly increasing hash order, each entry is popped at most once — O(1) amortized per nucleotide.
+
+A super-kmer boundary is emitted when the minimizer changes: `current_minimizer != prev_minimizer`. `SuperKmerIter` also emits a boundary when:
+
+- entropy of the current k-mer falls at or below the threshold θ (cursor retreated by k−1)
+- super-kmer length reaches 256 nucleotides (cursor retreated by k)
+
+## Kmer extraction
+
+A k-mer is extracted from a super-kmer with `SuperKmer::kmer(i)`, which delegates to `PackedSeq::extract::<KLen>(i)` and returns a `Kmer` — a left-aligned `u64` newtype (see [Kmer implementation](kmer.md)):
+
+```rust
+pub fn kmer(&self, i: usize) -> Result<Kmer, KmerError>
+```
+
+The bit slice `seq[i*2 .. (i+k)*2]` (Msb0 order) is loaded as a `u64` via `bitvec::load_be`, then left-shifted to produce the canonical left-aligned layout. One call — no loop, no allocation.
+
+---
+
+!!! abstract "Algorithm — Super-kmer reverse complement"
+    ```text
+    procedure SuperKmerRevcomp(seq, SEQL):
+        seql  ← nucleotide length
+        n     ← ⌈seql / 4⌉                  -- number of bytes
+        shift ← n × 8 − seql × 2            -- padding bits to flush
+
+        -- step 1: swap bytes outside-in, applying REVCOMP4 to each (256-byte L1 table)
+        lo ← 0 ; hi ← n − 1
+        while lo < hi:
+            seq[lo], seq[hi] ← REVCOMP4[seq[hi]], REVCOMP4[seq[lo]]
+            lo ← lo + 1 ; hi ← hi − 1
+        if lo == hi: seq[lo] ← REVCOMP4[seq[lo]]
+
+        -- step 2: left-rotate entire bit array by shift, zero trailing bits (SIMD via bitvec)
+        if shift > 0:
+            bits.rotate_left(shift)
+            bits[n×8 − shift .. n×8].fill(0)
+    ```

DevDocMD/implementation/superkmer.refs.md

@@ -0,0 +1,13 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/superkmer.md
+
+## Code couvert
+
+- `obikseq/src/superkmer.rs` — layout mémoire SuperKmer (header 32 bits + séquence byte-alignée), encodage ASCII, revcomp, deque minimiseur
+- `obiskbuilder/src/lib.rs` — fenêtre glissante monotone pour le maintien du minimiseur
+
+## Notes
+
+Document d'implémentation détaillé. Vérifier que le layout header (longueur, orientation,
+position minimiseur) n'a pas changé. La doc mentionne un revcomp SIMD — vérifier si c'est
+toujours le cas ou si l'implémentation est scalaire.

DevDocMD/implementation/unitig_evidence.md

@@ -0,0 +1,170 @@
+# Unitig-based MPHF evidence encoding
+
+## Role of unitigs in the index
+
+The MPHF maps each canonical kmer to an integer slot but provides no inverse: a slot index alone cannot reconstruct the kmer. The **evidence file** supplies this inverse: for each MPHF slot it stores a pointer into the unitig sequence file, from which k nucleotides can be extracted.
+
+Unitigs are the natural compact representation: a run of L nucleotides encodes L − k + 1 consecutive canonical kmers. The entire kmer set of a partition is reconstructible from its unitig binary file.
+
+---
+
+## Binary file formats
+
+### `unitigs.bin` — sequence chunks
+
+A sequence of binary records. Each record:
+
+```
+[u8: seql − k]  [ceil(seql / 4) bytes: 2-bit packed nucleotides]
+```
+
+- `seql − k` (0–255): nucleotide length minus k, so `seql = byte[0] + k` and `n_kmers = byte[0] + 1`.
+- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
+- Byte count for packed sequence: `ceil(seql / 4)`.
+
+Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
+
+```
+chunk 1: nucleotides [0,   MAX_KMERS_PER_CHUNK + k − 2]   (256 kmers)
+chunk 2: nucleotides [256, end]                             (remaining kmers)
+overlap: k−1 nucleotides shared between the two chunks
+```
+
+### `unitigs.bin.idx` — block-sampled offset index
+
+```
+magic     : 4 bytes  = "UIX3"
+block_bits: u32 LE   — granularity parameter (0–31)
+n_unitigs : u32 LE   — total number of chunks in unitigs.bin
+n_kmers   : u64 LE   — total number of kmers across all chunks
+offsets   : [u32 LE] — byte offsets into unitigs.bin, one per 2^block_bits chunks + sentinel
+```
+
+One offset entry is stored every `2^block_bits` chunks; the array is sentinel-terminated (last entry = file size). `DEFAULT_BLOCK_BITS = 0` stores one offset per chunk (exact table, no scan).
+
+### `evidence.bin` — per-slot MPHF evidence
+
+A flat array of u32 values, one per MPHF slot, no header:
+
+```
+bits [31:7] = chunk_id  (25 bits)
+bits  [6:0] = rank      (7 bits, 0–127)
+```
+
+File size = `n_slots × 4` bytes. `chunk_id` is the 0-based index of the record in `unitigs.bin`; `rank` is the position of the canonical kmer within that chunk (counting only canonical kmers). Encoding: `raw = (chunk_id << 7) | (rank & 0x7F)`. Decoding: `chunk_id = raw >> 7`, `rank = raw & 0x7F`.
+
+---
+
+## Building and reading the index
+
+### `build_unitig_idx(path, block_bits)`
+
+Scans `unitigs.bin` sequentially: for each chunk at byte offset `offset`, if `chunk_count & mask == 0` (where `mask = (1 << block_bits) − 1`), appends `offset as u32` to `block_offsets`. After the scan, appends a sentinel (= total file size), then writes the `.idx` file. Called after the unitig file is fully written and closed.
+
+### `open()`, `open_sequential()`, `open_direct_access()`
+
+`UnitigFileReader` has three constructors:
+
+- `open(path)` — smart default: if `unitigs.bin.idx` exists, delegates to `open_direct_access`; otherwise delegates to `open_sequential`. Prefer this in call sites that don't require one specific mode.
+- `open_sequential(path)` — never reads `.idx`. Sequential iterators only; `chunk_start(i)` falls back to an O(i) mmap scan rather than panicking.
+- `open_direct_access(path)` — requires `.idx` to be present. Enables O(1) or O(2^block_bits) `chunk_start(i)`, used by `verify_canonical_kmer` at query time.
+
+`CanonicalKmerIter` — a clonable sequential iterator returned by `UnitigFileReader::iter_canonical_kmers()`. It holds an `Arc<Mmap>` so cloning resets the cursor to the start without reopening the file. This makes it usable with `par_bridge()` for parallel MPHF construction without random access.
+
+### `chunk_start(i)` — access modes
+
+When `.idx` is loaded (`open_direct_access`):
+
+- `block_bits = 0`: single array lookup, O(1).
+- `block_bits > 0`: lookup block, then scan ≤ 2^block_bits records, O(2^block_bits).
+
+When `.idx` is absent (`open_sequential`): `chunk_start(i)` performs an O(i) sequential mmap scan from offset 0. No panic — the function degrades gracefully. This degraded path is used by `find_strict()` on Approx layers (sequential scan of all canonical kmers).
+
+### Decoding a kmer from slot `s`
+
+```rust
+let (chunk_id, rank) = evidence.decode(s);      // u32 → (chunk_id: u32, rank: u8)
+let kmer = unitigs.raw_kmer(chunk_id, rank);    // 2-bit packed slice → left-aligned u64
+```
+
+Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extraction from `unitigs.bin` via the mmap. The retrieved sequence is already canonical (only canonical kmers are inserted into the De Bruijn graph).
+
+---
+
+## Field widths and capacity
+
+| field      | bits | range         | capacity check (*B. nana*, 256 partitions) |
+|------------|------|---------------|---------------------------------------------|
+| `seql − k` | 8    | 0–255         | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` |
+| `rank`     | 7    | 0–127         | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
+| `chunk_id` | 25   | 0–33 554 431  | avg U ≈ 275 k chunks/partition |
+
+The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total.
+
+---
+
+## Evidence bit-cost
+
+Strategy B (chunk_id + rank) is the implemented strategy. For *B. nana* (k=31, 256 partitions, P ≈ 10.4 M unique kmers/partition, U ≈ 275 k chunks/partition, m_u ≈ 37.9 kmers/chunk):
+
+| field      | theoretical cost        | value   |
+|------------|-------------------------|---------|
+| chunk_id   | ⌈log₂ U⌉               | 19 bits |
+| rank       | ⌈log₂ m_u⌉ (≈ fixed)   | 6 bits  |
+| **stored** | aligned u32             | **32 bits/slot** |
+
+The u32 layout is chosen for alignment and simplicity; no bit-addressing arithmetic is needed.
+
+Comparison with strategy A (global nucleotide offset): `⌈log₂(P · (1 + (k−1)/m_u))⌉ = 25 bits`. Strategy A is theoretically 2 bits cheaper; strategy B's advantage is **locality** (decoding touches one chunk's cache lines) and a bounded, constant-width rank field independent of partition size.
+
+---
+
+## Unitig decomposition non-determinism
+
+The unitig extraction from `GraphDeBruijn` is **not deterministic**: two runs on identical input can produce different unitig counts and sequences while covering exactly the same canonical kmer set.
+
+The hash map (`hashbrown::HashMap` with `Xxh3Builder`) has run-dependent iteration order. The `start_iter` first pass emits every node where `can_extend_left` is false — this includes true dead-ends and branch points (nodes with ≥2 left neighbours). When a branch point is encountered before its upstream neighbours, it claims the downstream chain and those upstream neighbours later produce length-k degenerate unitigs. When upstream neighbours appear first, they extend through the branch point.
+
+**Example** — fork topology (k = 31):
+
+```
+A → B ← C
+    ↓
+    D
+```
+
+B has two left neighbours, so `can_extend_left = false`. Two valid tilings:
+
+| iteration order | unitigs | count |
+|---|---|---|
+| A first | ABD, C | 2 |
+| B first | BD, A, C | 3 |
+
+Both cover the same 4 canonical kmers. Pure cycles are unaffected: all cycle nodes have both extensions present, so none are emitted in the first pass; each cycle produces exactly one unitig regardless of entry point (only the cut point varies).
+
+This non-determinism is benign for MPHF construction: the MPHF is built from the kmer set, which is identical across tilings.
+
+---
+
+## Partition-size tradeoff
+
+Measured on *B. nana* (k=31, m=11), summing across all partitions:
+
+| N partitions | m_u  |
+|---|---|
+| 1            | 41.89 |
+| 16           | 38.19 |
+| 256          | 37.90 |
+| 1 024        | 37.89 |
+
+`m_u` is set by De Bruijn graph topology (heterozygosity, repeats, sequencing errors), not partition count. The variation from 1 to 1024 partitions is under 10%; within 16–1024 it is under 1%. Unitigs provide ~3.1× nucleotide compaction over super-kmers at 256 partitions.
+
+Evidence cost decreases by 1 bit/kmer with each doubling of partition count (via `log₂ U = log₂(P/m_u)`). The sequence storage term `2 · (1 + (k−1)/m_u) ≈ 3.6 bits/kmer` is approximately constant.
+
+---
+
+## Alternative: fingerprint evidence
+
+`evidence.bin` can be replaced by `fingerprint.bin` at index build time (`--approx`) or after the fact (`reindex --approx`). The fingerprint stores b bits per MPHF slot (the low b bits of `kmer.seq_hash()`); verification becomes a single bitfield comparison instead of a unitig dereference. False-positive rate per k-mer query: 1/2^b. With the Findere z parameter, z consecutive k-mers must all match, reducing the effective window FP rate to 1/2^(b·z) while skipping z−1 of every z k-mers. No `.idx` file is written or read in approx mode.
+
+See [Approximate evidence (Findere fingerprint)](evidence_elimination.md) for the full design and CLI parameters.

DevDocMD/implementation/unitig_evidence.refs.md

@@ -0,0 +1,18 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: implementation/unitig_evidence.md
+
+## Code couvert
+
+- `obiskio/src/unitig_index.rs` — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()
+- `obilayeredmap/src/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
+- `obidebruinj/src/debruijn.rs` — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK
+
+## Notes
+
+FORT RISQUE DE DÉRIVE. Changements récents :
+- `DEFAULT_BLOCK_BITS` est passé de 6 à 0 (accès O(1) par défaut)
+- `block_bits` est maintenant un paramètre runtime de `build_unitig_idx()` et `UnitigFileWriter`
+- `chunk_start()` a un chemin chaud explicite pour block_bits=0 (accès tableau direct, 0 scan)
+- `open()` vs `open_sequential()` : distinction nouvelle, importante pour la compréhension du coût
+- `iter_unitigs()` ajouté comme alias public de `iter_chunks_sequential()`
+Mettre à jour la description du format .idx et le modèle de coût d'accès aléatoire.

DevDocMD/index.md

@@ -0,0 +1,63 @@
+# obikmer
+
+`obikmer` is a Rust tool for manipulation, counting, indexing, and set operations on DNA sequences represented as kmer sets.
+
+## Subcommands
+
+| Subcommand  | Purpose |
+|-------------|---------|
+| `superkmer` | Extract super-kmers from a sequence file and write to stdout |
+| `index`     | Build a complete genome index (scatter → dereplicate → count → layered MPHF) |
+| `merge`     | Merge multiple built indexes into one |
+| `filter` | Apply row-level selection (σ) to an index: retain only k-mers matching the ingroup/outgroup predicates. Output is a new single-layer index — compaction is a consequence, not the goal. Supports the shared [kmer filtering](implementation/filtering.md) system |
+| `query`     | Query an index with sequences and annotate matches |
+| `dump`      | Dump all indexed k-mers as CSV (kmer + per-genome counts or presence); supports the shared [kmer filtering](implementation/filtering.md) system; `--head N` limits output to the first N k-mers |
+| `annotate`  | Add or update genome metadata from a CSV file; or dump metadata as CSV |
+| `phylo`     | Compute pairwise evolutionary-distance proxies between genomes (`--metric jaccard\|mash\|hamming\|bray-curtis\|relfreq-bray-curtis\|euclidean\|relfreq-euclidean\|hellinger\|hellinger-euclidean`); optionally build NJ/UPGMA trees; `--presence-threshold N` sets the minimum count to consider a k-mer present when computing Jaccard/Mash on count indexes (default 1); optionally a central-position SNP/Sankoff calibration with exports for TNT/PhyG/IQ-TREE (see [evolutionary distances](theory/evolutionary_distances.md)) |
+| `unitig`    | Build a global de Bruijn graph across all partitions and enumerate its unitigs as FASTA; supports the shared [kmer filtering](implementation/filtering.md) system |
+| `select`    | Project and/or aggregate genome columns into a new or in-place index; the column-axis counterpart of `filter` (see [select](implementation/select.md)) |
+| `estimate`  | Estimate approximate-index parameters (z, evidence bits, FP rates) before indexing |
+| `reindex`   | Convert an index's evidence in-place: exact ↔ approx |
+| `utils`     | Miscellaneous index utilities: `--new-label NEW=OLD` renames a genome label; `--upgrade-index` adds missing `layer_meta.json` to old indexes |
+| `pack`      | Pack per-column matrix files into single-file format to reduce query I/O |
+
+## Constraints
+
+- Target scale: individual genome datasets, tens of Gbases
+- Maximum efficiency in computation, memory, and disk usage
+- k odd, k ∈ [11, 31], fixed at runtime; kmer fits in a u64 (2 bits/base)
+- Canonical form: `min(kmer, revcomp(kmer))` reduces strand-symmetric space by half
+- Input formats for `index`/`superkmer`: FASTA (`.fa`, `.fasta`), FASTQ (`.fq`, `.fastq`), GenBank flat file (`.gb`, `.gbk`, `.gbff`), all optionally gzip-compressed; directories expanded recursively; streaming stdin via `-`
+- Input formats for `query`: FASTA, FASTQ, optionally gzip-compressed; streaming stdin via `-`
+
+## Parameter constraints (enforced at CLI)
+
+All constraints below are checked by `CommonArgs::validate()` at the start of `superkmer` and `index`. Invalid values exit immediately with an error.
+
+| Parameter | Constraint | Reason |
+|-----------|-----------|--------|
+| k (`--kmer-size`) | odd | even k allows palindromic k-mers: kmer == revcomp(kmer), breaking the canonical form invariant |
+| k (`--kmer-size`) | k ∈ [11, 31] | k > 31 overflows u64 at 2 bits/base; k < 11 gives insufficient specificity |
+| m (`--minimizer-size`) | odd | same palindrome argument as k |
+| m (`--minimizer-size`) | 3 ≤ m ≤ k−1 | minimizer must be strictly shorter than the kmer |
+| z (`-z`, Findere, `index --approx` only) | z ≤ k−1 | effective indexed kmer size is k−z+1; z ≥ k would make it ≤ 0 |
+
+## Genome label constraints
+
+Genome labels are arbitrary Unicode strings with the following restrictions:
+
+| Character | Forbidden | Reason |
+|-----------|-----------|--------|
+| `/` | yes | filesystem path separator |
+| `=` | yes | `--new-label` parser separator |
+| `\0` | yes | null byte |
+| `\n` `\r` `\t` | yes | break CSV output |
+| spaces | **allowed** | use shell quoting: `--new-label 'new label=old label'` |
+
+Empty labels are also rejected. Labels derived automatically from the index directory name (when `--label` is omitted) are not validated since they come from the filesystem and are already safe.
+
+## Priority operations
+
+- Kmer counting (frequencies)
+- Fast search / query
+- Set operations: union, intersection, difference

DevDocMD/index.refs.md

@@ -0,0 +1,13 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: index.md
+
+## Code couvert
+
+- `obikmer/src/main.rs` — point d'entrée CLI, contraintes globales
+- `obikmer/src/cli.rs` — structure des arguments communs
+
+## Notes
+
+Document de niveau projet (vue d'ensemble, motivations, contraintes fondamentales).
+Pas de code Rust spécifique à vérifier au-delà des contraintes générales (k impair, formats d'entrée).
+À mettre à jour si de nouvelles sous-commandes ou formats sont ajoutés.

DevDocMD/installation.md

@@ -0,0 +1,84 @@
+# Installation
+
+## Prerequisites
+
+### Rust toolchain
+
+`obikmer` requires **Rust 1.85 or later** (edition 2024). Install or update via [rustup](https://rustup.rs):
+
+```bash
+curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+rustup update stable
+```
+
+### C build environment (required for hwloc)
+
+`obikmer` embeds [hwloc](https://www.open-mpi.org/projects/hwloc/) (Hardware Locality) for NUMA-aware thread placement on multi-socket machines. hwloc is built from source at compile time via the `vendored` feature of the `hwlocality` crate. This requires a standard C build environment.
+
+#### Linux (Debian/Ubuntu)
+
+```bash
+apt install build-essential automake libtool autoconf pkg-config
+```
+
+#### Linux (RHEL/Rocky/AlmaLinux)
+
+```bash
+dnf install gcc make automake libtool autoconf pkgconfig
+```
+
+#### HPC clusters
+
+Most HPC clusters provide these tools via the module system:
+
+```bash
+module load gcc automake libtool autoconf
+```
+
+If in doubt, check whether `autoreconf --version` and `libtool --version` return successfully.
+
+#### macOS
+
+```bash
+brew install automake libtool autoconf pkg-config
+```
+
+## Building
+
+```bash
+git clone <repository-url>
+cd obikmer/src
+cargo build --release
+```
+
+The compiled binary is at `target/release/obikmer`.
+
+### Building on HPC clusters (network filesystems)
+
+HPC home directories are typically on a network filesystem (Lustre, NFS) optimised for large sequential reads — not for the thousands of small file operations that Cargo generates during compilation. Building directly on such a filesystem can be extremely slow (0.1% CPU utilisation, tens of minutes for what should take seconds).
+
+**Always redirect the build directory to a local scratch disk:**
+
+```bash
+CARGO_TARGET_DIR=/scratch/$USER/cargo-target cargo build --release
+```
+
+Adapt the path to the local scratch available on your cluster (`/var/tmp`, `/tmp`, `/scratch/local`, etc.). Once built, copy the binary to a permanent location:
+
+```bash
+cp /scratch/$USER/cargo-target/release/obikmer ~/bin/
+```
+
+## NUMA support
+
+NUMA-aware thread placement is active automatically on multi-socket Linux machines (detected at runtime via hwloc). No special build flag is required — the detection is built in and falls back gracefully to the single-pool adaptive strategy on:
+
+- macOS (Apple Silicon, unified memory)
+- single-socket Linux machines
+- any system where hwloc reports only one NUMA node
+
+## Verifying the installation
+
+```bash
+obikmer --help
+```

DevDocMD/kmers.md

@@ -0,0 +1,34 @@
+# Kmers and super-kmers
+
+## Kmers
+
+A **kmer** is a DNA subsequence of fixed length k. Two constraints govern the choice of k:
+
+- **k ∈ [11, 31]**: the range ensures the kmer is long enough to be specific and short enough to fit in a single machine word (u64 at 2 bits/base requires k ≤ 32; k < 11 yields insufficient specificity).
+- **k is odd**: an odd-length sequence cannot equal its own reverse complement (no palindromes). This guarantees that the canonical form `min(kmer, revcomp(kmer))` is always strictly defined — the two orientations are always distinct — which is required for strand-independent counting.
+
+Both constraints are **enforced at CLI entry** by `CommonArgs::validate()` in `superkmer` and `index`. Passing an invalid k exits immediately with an error message.
+
+## Super-kmers
+
+A **super-kmer** is a maximal run of consecutive kmers from a DNA read, each overlapping the next by k−1 nucleotides, sharing the same **canonical minimizer**. The **canonical minimizer** of a kmer is the m-mer (m < k) whose canonical hash `hash_kmer(min(m-mer, revcomp(m-mer)))` is smallest over all m-mers in the kmer window. The hash function is a `mix64`-based bijection; selection is purely hash-ordered with no degeneracy filter. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
+
+### Canonical super-kmers
+
+A **canonical super-kmer** is the lexicographic minimum of a super-kmer and its reverse complement:
+
+```
+canonical(super-kmer) = min(super-kmer, revcomp(super-kmer))
+```
+
+When a read and its reverse-complement are both sequenced, they produce super-kmers that are reverse complements of each other. Both map to the same canonical form: the same genomic region is represented by a single canonical super-kmer regardless of which strand was read.
+
+### Expected length of a super-kmer
+
+For a random minimizer of length m over k-mers of length k, the density of minimizer positions is approximately 2/(k−m+2) [@Zheng2020-ji; @Golan2025-xf], so the expected number of consecutive k-mers per super-kmer is (k−m+2)/2. A run of n k-mers spans n + k − 1 nucleotides, giving:
+
+$$L_{\text{nt}} = \frac{k-m+2}{2} + k - 1$$
+
+For k=31, m=13: expected ≈ 40 nt. In practice super-kmers rarely exceed a few dozen nucleotides.[^superkmer_length]
+
+[^superkmer_length]: The expected length formula and the density approximation 2/(k−m+2) should be verified against the values reported in [@Zheng2020-ji] and [@Golan2025-xf].

DevDocMD/kmers.refs.md

@@ -0,0 +1,13 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: kmers.md
+
+## Code couvert
+
+- `obikseq/src/kmer.rs` — type Kmer, propriétés, forme canonique
+- `obikseq/src/superkmer.rs` — type SuperKmer, longueur attendue
+- `obiskbuilder/src/lib.rs` — extraction de superkmers par minimiseur
+
+## Notes
+
+Chevauche `theory/encoding.md` (encodage 2 bits) et `theory/minimizer.md` (choix du minimiseur).
+Vérifier que la définition de SuperKmer est cohérente avec les invariants actuels de `obikseq`.

DevDocMD/references.bib

@@ -0,0 +1,261 @@
+%% This BibTeX bibliography file was created using BibDesk.
+%% https://bibdesk.sourceforge.io/
+
+%% Created for Eric Coissac at 2026-04-18 08:19:36 +0200
+
+
+%% Saved with string encoding Unicode (UTF-8)
+
+
+
+@article{Zheng2020-ji,
+	abstract = {MOTIVATION: Minimizers are methods to sample k-mers from a
+               string, with the guarantee that similar set of k-mers will be
+               chosen on similar strings. It is parameterized by the k-mer
+               length k, a window length w and an order on the k-mers.
+               Minimizers are used in a large number of softwares and pipelines
+               to improve computation efficiency and decrease memory usage.
+               Despite the method's popularity, many theoretical questions
+               regarding its performance remain open. The core metric for
+               measuring performance of a minimizer is the density, which
+               measures the sparsity of sampled k-mers. The theoretical optimal
+               density for a minimizer is 1/w, provably not achievable in
+               general. For given k and w, little is known about asymptotically
+               optimal minimizers, that is minimizers with density O(1/w).
+               RESULTS: We derive a necessary and sufficient condition for
+               existence of asymptotically optimal minimizers. We also provide a
+               randomized algorithm, called the Miniception, to design
+               minimizers with the best theoretical guarantee to date on density
+               in practical scenarios. Constructing and using the Miniception is
+               as easy as constructing and using a random minimizer, which
+               allows the design of efficient minimizers that scale to the
+               values of k and w used in current bioinformatics software
+               programs. AVAILABILITY AND IMPLEMENTATION: Reference
+               implementation of the Miniception and the codes for analysis can
+               be found at https://github.com/kingsford-group/miniception.
+               SUPPLEMENTARY INFORMATION: Supplementary data are available at
+               Bioinformatics online.},
+	author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
+	doi = {10.1093/bioinformatics/btaa472},
+	issn = {1367-4803,1367-4811},
+	journal = {Bioinformatics (Oxford, England)},
+	language = {en},
+	month = jul,
+	number = {Suppl_1},
+	pages = {i119--i127},
+	pmc = {PMC8248892},
+	pmid = 32657376,
+	publisher = {Oxford University Press (OUP)},
+	title = {Improved design and analysis of practical minimizers},
+	url = {http://dx.doi.org/10.1093/bioinformatics/btaa472},
+	volume = 36,
+	year = 2020,
+	bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btaa472}}
+
+@article{Zheng2021-cc,
+	abstract = {MOTIVATION: Minimizers are efficient methods to sample k-mers
+               from genomic sequences that unconditionally preserve sufficiently
+               long matches between sequences. Well-established methods to
+               construct efficient minimizers focus on sampling fewer k-mers on
+               a random sequence and use universal hitting sets (sets of k-mers
+               that appear frequently enough) to upper bound the sketch size. In
+               contrast, the problem of sequence-specific minimizers, which is
+               to construct efficient minimizers to sample fewer k-mers on a
+               specific sequence such as the reference genome, is less studied.
+               Currently, the theoretical understanding of this problem is
+               lacking, and existing methods do not specialize well to sketch
+               specific sequences. RESULTS: We propose the concept of polar
+               sets, complementary to the existing idea of universal hitting
+               sets. Polar sets are k-mer sets that are spread out enough on the
+               reference, and provably specialize well to specific sequences.
+               Link energy measures how well spread out a polar set is, and with
+               it, the sketch size can be bounded from above and below in a
+               theoretically sound way. This allows for direct optimization of
+               sketch size. We propose efficient heuristics to construct polar
+               sets, and via experiments on the human reference genome, show
+               their practical superiority in designing efficient
+               sequence-specific minimizers. AVAILABILITY AND IMPLEMENTATION: A
+               reference implementation and code for analyses under an
+               open-source license are at
+               https://github.com/kingsford-group/polarset. SUPPLEMENTARY
+               INFORMATION: Supplementary data are available at Bioinformatics
+               online.},
+	author = {Zheng, Hongyu and Kingsford, Carl and Mar{\c c}ais, Guillaume},
+	doi = {10.1093/bioinformatics/btab313},
+	issn = {1367-4803,1367-4811},
+	journal = {Bioinformatics (Oxford, England)},
+	language = {en},
+	month = jul,
+	number = {Suppl\_1},
+	pages = {i187--i195},
+	pmc = {PMC8686682},
+	pmid = 34252928,
+	publisher = {Oxford University Press (OUP)},
+	title = {Sequence-specific minimizers via polar sets},
+	url = {http://dx.doi.org/10.1093/bioinformatics/btab313},
+	volume = 37,
+	year = 2021,
+	bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btab313}}
+
+@article{Pan2024-hb,
+	abstract = {MOTIVATION: The minimizer concept is a data structure for
+               sequence sketching. The standard canonical minimizer selects a
+               subset of k-mers from the given DNA sequence by comparing the
+               forward and reverse k-mers in a window simultaneously according
+               to a predefined selection scheme. It is widely employed by
+               sequence analysis such as read mapping and assembly. k-mer
+               density, k-mer repetitiveness (e.g. k-mer bias), and
+               computational efficiency are three critical measurements for
+               minimizer selection schemes. However, there exist trade-offs
+               between kinds of minimizer variants. Generic, effective, and
+               efficient are always the requirements for high-performance
+               minimizer algorithms. RESULTS: We propose a simple minimizer
+               operator as a refinement of the standard canonical minimizer. It
+               takes only a few operations to compute. However, it can improve
+               the k-mer repetitiveness, especially for the lexicographic order.
+               It applies to other selection schemes of total orders (e.g.
+               random orders). Moreover, it is computationally efficient and the
+               density is close to that of the standard minimizer. The refined
+               minimizer may benefit high-performance applications like binning
+               and read mapping. AVAILABILITY AND IMPLEMENTATION: The source
+               code of the benchmark in this work is available at the github
+               repository https://github.com/xp3i4/mini\_benchmark.},
+	author = {Pan, Chenxu and Reinert, Knut},
+	doi = {10.1093/bioinformatics/btae045},
+	issn = {1367-4803,1367-4811},
+	journal = {Bioinformatics (Oxford, England)},
+	language = {en},
+	month = feb,
+	number = 2,
+	pmc = {PMC10868324},
+	pmid = 38269626,
+	publisher = {Oxford University Press (OUP)},
+	title = {A simple refined DNA minimizer operator enables 2-fold faster computation},
+	url = {http://dx.doi.org/10.1093/bioinformatics/btae045},
+	volume = 40,
+	year = 2024,
+	bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btae045}}
+
+@article{Kille2023-px,
+	abstract = {MOTIVATION: The Jaccard similarity on k-mer sets has shown to be
+               a convenient proxy for sequence identity. By avoiding expensive
+               base-level alignments and comparing reduced sequence
+               representations, tools such as MashMap can scale to massive
+               numbers of pairwise comparisons while still providing useful
+               similarity estimates. However, due to their reliance on minimizer
+               winnowing, previous versions of MashMap were shown to be biased
+               and inconsistent estimators of Jaccard similarity. This directly
+               impacts downstream tools that rely on the accuracy of these
+               estimates. RESULTS: To address this, we propose the minmer
+               winnowing scheme, which generalizes the minimizer scheme by use
+               of a rolling minhash with multiple sampled k-mers per window. We
+               show both theoretically and empirically that minmers yield an
+               unbiased estimator of local Jaccard similarity, and we implement
+               this scheme in an updated version of MashMap. The minmer-based
+               implementation is over 10 times faster than the minimizer-based
+               version under the default ANI threshold, making it well-suited
+               for large-scale comparative genomics applications. AVAILABILITY
+               AND IMPLEMENTATION: MashMap3 is available at
+               https://github.com/marbl/MashMap.},
+	author = {Kille, Bryce and Garrison, Erik and Treangen, Todd J and Phillippy, Adam M},
+	doi = {10.1093/bioinformatics/btad512},
+	issn = {1367-4803,1367-4811},
+	journal = {Bioinformatics (Oxford, England)},
+	language = {en},
+	month = sep,
+	number = 9,
+	pmc = {PMC10505501},
+	pmid = 37603771,
+	publisher = {Oxford University Press (OUP)},
+	title = {Minmers are a generalization of minimizers that enable unbiased local Jaccard estimation},
+	url = {http://dx.doi.org/10.1093/bioinformatics/btad512},
+	volume = 39,
+	year = 2023,
+	bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btad512}}
+
+@incollection{Golan2025-xf,
+	address = {Cham},
+	author = {Golan, Shay and Shur, Arseny M},
+	booktitle = {Lecture Notes in Computer Science},
+	doi = {10.1007/978-3-031-82670-2\_25},
+	isbn = {9783031826696,9783031826702},
+	issn = {0302-9743,1611-3349},
+	language = {en},
+	pages = {347--360},
+	publisher = {Springer Nature Switzerland},
+	series = {Lecture Notes in Computer Science},
+	title = {Expected density of random minimizers},
+	url = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
+	year = 2025,
+	bdsk-url-1 = {http://dx.doi.org/10.1007/978-3-031-82670-2_25},
+	bdsk-url-2 = {http://dx.doi.org/10.1007/978-3-031-82670-2%5C_25}}
+
+@article{Mohamadi2017-ok,
+	abstract = {Motivation: Many bioinformatics algorithms are designed for the
+               analysis of sequences of some uniform length, conventionally
+               referred to as k -mers. These include de Bruijn graph assembly
+               methods and sequence alignment tools. An efficient algorithm to
+               enumerate the number of unique k -mers, or even better, to build
+               a histogram of k -mer frequencies would be desirable for these
+               tools and their downstream analysis pipelines. Among other
+               applications, estimated frequencies can be used to predict genome
+               sizes, measure sequencing error rates, and tune runtime
+               parameters for analysis tools. However, calculating a k -mer
+               histogram from large volumes of sequencing data is a challenging
+               task. Results: Here, we present ntCard, a streaming algorithm for
+               estimating the frequencies of k -mers in genomics datasets. At
+               its core, ntCard uses the ntHash algorithm to efficiently compute
+               hash values for streamed sequences. It then samples the
+               calculated hash values to build a reduced representation
+               multiplicity table describing the sample distribution. Finally,
+               it uses a statistical model to reconstruct the population
+               distribution from the sample distribution. We have compared the
+               performance of ntCard and other cardinality estimation
+               algorithms. We used three datasets of 480 GB, 500 GB and 2.4 TB
+               in size, where the first two representing whole genome shotgun
+               sequencing experiments on the human genome and the last one on
+               the white spruce genome. Results show ntCard estimates k -mer
+               coverage frequencies >15× faster than the state-of-the-art
+               algorithms, using similar amount of memory, and with higher
+               accuracy rates. Thus, our benchmarks demonstrate ntCard as a
+               potentially enabling technology for large-scale genomics
+               applications. Availability and Implementation: ntCard is written
+               in C ++ and is released under the GPL license. It is freely
+               available at https://github.com/bcgsc/ntCard. Contact:
+               hmohamadi@bcgsc.ca or ibirol@bcgsc.ca. Supplementary information:
+               Supplementary data are available at Bioinformatics online.},
+	author = {Mohamadi, Hamid and Khan, Hamza and Birol, Inanc},
+	date-modified = {2026-04-18 08:19:36 +0200},
+	doi = {10.1093/bioinformatics/btw832},
+	issn = {1367-4803,1367-4811},
+	journal = {Bioinformatics (Oxford, England)},
+	language = {en},
+	month = may,
+	number = 9,
+	pages = {1324--1330},
+	pmc = {PMC5408799},
+	pmid = 28453674,
+	publisher = {Oxford University Press (OUP)},
+	title = {ntCard: a streaming algorithm for cardinality estimation in genomics data},
+	url = {http://dx.doi.org/10.1093/bioinformatics/btw832},
+	volume = 33,
+	year = 2017,
+	bdsk-url-1 = {http://dx.doi.org/10.1093/bioinformatics/btw832}}
+
+@misc{Mash-distances-doc,
+	author = {{Marbl Lab}},
+	howpublished = {Mash documentation},
+	title = {Mash Distance},
+	url = {https://mash.readthedocs.io/en/latest/distances.html},
+	urldate = {2026-07-09},
+	year = 2026}
+
+@article{Fan2015-mash-formula,
+	author = {Fan, Huan and Ives, Anthony R and Surget-Groba, Yann and Cannon, Charles H},
+	doi = {10.1186/s12864-015-1647-5},
+	journal = {BMC Genomics},
+	number = 1,
+	title = {An assembly and alignment-free method of phylogeny reconstruction from next-generation sequencing data},
+	url = {https://doi.org/10.1186/s12864-015-1647-5},
+	volume = 16,
+	year = 2015}

DevDocMD/theory/encoding.md

@@ -0,0 +1,42 @@
+# DNA encoding
+
+## 2-bit nucleotide encoding
+
+All nucleotides are encoded on 2 bits, MSB-first within each word. Nucleotides are numbered 0-based from the 5′ end across all sequence types:
+
+| Base | Encoding |
+|------|----------|
+| A    | `00`     |
+| C    | `01`     |
+| G    | `10`     |
+| T    | `11`     |
+
+The Watson-Crick complement of any base is its bitwise NOT on 2 bits: `complement(base) = ~base & 0b11`.
+
+## Kmer encoding
+
+A kmer fits in a single `u64`. Nucleotide 0 occupies bits 63–62, nucleotide i occupies bits 63−2i and 62−2i, and the low 64−2k bits are zero. Extraction of nucleotide i (0 ≤ i < k): `(kmer >> (62 - 2*i)) & 0b11`.
+
+Reverse complement is computed by **bit manipulation in four steps**, with no lookup table:
+
+!!! abstract "Algorithm — Kmer reverse complement"
+    ```text
+    procedure KmerRevcomp(kmer, k):
+        x ← ~kmer                                           -- complement all bases
+        x ← swap_bytes(x)                                   -- reverse byte order
+        x ← ((x >> 4) & 0x0F0F0F0F0F0F0F0F)
+           | ((x & 0x0F0F0F0F0F0F0F0F) << 4)               -- swap nibbles within each byte
+        x ← ((x >> 2) & 0x3333333333333333)
+           | ((x & 0x3333333333333333) << 2)                -- swap 2-bit pairs within each nibble
+        return x << (64 - 2*k)                              -- re-align to MSB
+    ```
+
+The three reorder passes together reverse the order of all 2-bit base codes across the 64-bit word. The bitwise NOT in the first step complements each base (A↔T, C↔G). The final left shift clears the low 64−2k padding bits.
+
+The **canonical form** is the lexicographic minimum of the kmer and its reverse complement:
+
+```
+canonical(kmer) = min(kmer, revcomp(kmer))
+```
+
+This halves the kmer space and ensures strand-independent counting.

DevDocMD/theory/encoding.refs.md

@@ -0,0 +1,11 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/encoding.md
+
+## Code couvert
+
+- `obikseq/src/kmer.rs` — encodage 2 bits/base, revcomp, forme canonique
+
+## Notes
+
+Document purement théorique. Peu de risque de dérive sauf si l'encodage interne de Kmer change.
+Vérifier que la table d'encodage A=00, C=01, G=10, T=11 est toujours celle du code.

DevDocMD/theory/entropy.md

@@ -0,0 +1,84 @@
+# Kmer entropy filter
+
+Low-complexity kmers (polyA, polyT, tandem repeats) are detected and excluded during phase 1. The filter computes a **normalized Shannon entropy** over sub-words of multiple sizes, corrected for one source of bias: the small number of observations within a single kmer relative to the number of possible sub-words.
+
+## Sub-word frequencies
+
+For a kmer of length k and a sub-word size ws (1 ≤ ws ≤ ws_max, typically ws_max = 6), extract the $n_{\text{words}} = k - ws + 1$ overlapping sub-words by sliding a window of length ws:
+
+$$w_i = \text{kmer}[i \mathinner{..} i+ws-1], \quad i = 0, \ldots, n_{\text{words}}-1$$
+
+Each sub-word is tallied under its own raw 2-bit-packed value — **no canonicalization**. Let $f_j$ be the count of raw word $j$ among the $n_{\text{words}}$ sub-words ($\sum_j f_j = n_{\text{words}}$), over the $4^{ws}$ possible raw words.
+
+An earlier version of this filter first folded each sub-word into a circular+reverse-complement equivalence class, then "unfolded" the observed class frequency back onto its members to correct for unequal class sizes. That machinery bought nothing it was claimed for — see *Why no equivalence classes* below — while measurably weakening detection of the very sequences the filter exists to catch, so it was removed.
+
+## Corrected Shannon entropy
+
+$$H_{\text{corr}} = \log(n_{\text{words}}) - \frac{1}{n_{\text{words}}} \sum_j f_j \log f_j$$
+
+This is a plain Shannon entropy over the observed raw-word frequencies.
+
+## Maximum entropy correction for small samples
+
+With only $n_{\text{words}}$ observations over $4^{ws}$ possible raw words, the achievable maximum entropy is bounded by the most uniform integer distribution over $4^{ws}$ categories.
+
+Let $c = \lfloor n_{\text{words}} / 4^{ws} \rfloor$ and $r = n_{\text{words}} \bmod 4^{ws}$. The most uniform integer distribution assigns frequency $c+1$ to $r$ categories and $c$ to the remaining $4^{ws} - r$, with the convention $0 \log 0 = 0$:
+
+$$H_{\max} = -\left[(4^{ws} - r)\,\frac{c}{n_{\text{words}}}\log\frac{c}{n_{\text{words}}} + r\,\frac{c+1}{n_{\text{words}}}\log\frac{c+1}{n_{\text{words}}}\right]$$
+
+When $n_{\text{words}} < 4^{ws}$: $c=0$, $r=n_{\text{words}}$, and the formula reduces to $H_{\max} = \log(n_{\text{words}})$ — a single unified expression covers both regimes. A truly random sequence achieves $H_{\text{corr}} \approx H_{\max}$.
+
+## Normalized entropy
+
+$$\hat{H}(ws) = \frac{H_{\text{corr}}}{H_{\max}} \in [0, 1]$$
+
+## Final score
+
+The filter computes $\hat{H}(ws)$ for each word size ws from 1 to ws_max and returns the **minimum**:
+
+$$\text{entropy}(kmer) = \min_{ws=1}^{ws_{\max}} \hat{H}(ws)$$
+
+A value near 0 indicates low complexity (e.g. AAAA…); near 1 indicates high complexity. A kmer is rejected if $\text{entropy}(kmer) < \theta$, where $\theta$ is a collection parameter (default 0.7). The minimum across word sizes ensures that any scale of repetition is detected independently: polyA is caught at ws=1, dinucleotide repeats at ws=2, etc.
+
+## Why no equivalence classes
+
+A prior design folded each sub-word into the canonical form of its circular-rotation + reverse-complement equivalence class before tallying, on the reasoning that (a) it guarantees $\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))$, and (b) collapsing phase-shifted repeats (e.g. `ATG` ≡ `TGA` ≡ `GAT`) into one class better reflects that they are "the same" low-complexity pattern.
+
+Both properties already hold for the raw, unfolded entropy above, without any class machinery:
+
+- **Reverse complement**: for any K of length n, window $j$ of $\text{revcomp}(K)$ equals $\text{revcomp}$ of window $(n{-}ws{-}j)$ of K. This is a bijection between the window sets under which each window maps to its own revcomp — and revcomp is itself a bijection (involution) on the space of raw ws-mers. So the multiset of raw-word frequencies for $\text{revcomp}(K)$ is exactly a relabeling of the multiset for K, and Shannon entropy — a function of the frequency multiset alone — is exactly invariant. No folding required, for any K.
+- **Tandem repeats**: a period-p repeat sampled by a stride-1 sliding window naturally cycles through its own rotations as raw tokens (e.g. `ATGATGATG…` yields the raw words `ATG`, `TGA`, `GAT` in rotation as the window slides). The low diversity this represents (few distinct raw words out of $4^{ws}$ possible) is already visible in the raw frequency distribution — no folding needed to detect it.
+
+What the fold-then-unfold step actually did was credit each observed class with the frequency of equivalence-class members that were **never observed on the read strand**, inflating $H_{\text{corr}}$ for genuine repeats. Worked example: k=31, ws=3, kmer = `ATG` repeated ($n_{\text{words}}=29$, all 29 windows fall into one class of size 6 under the old scheme — 3 rotations × forward/revcomp):
+
+| | $H_{\text{corr}}$ | normalized |
+|---|---|---|
+| old (folded, class size 6) | $\log 6 \approx 1.79$ | $\approx 0.53$ |
+| current (raw, unfolded) | $\log 3 \approx 1.10$ | $\approx 0.33$ |
+
+The gap is not a rounding artifact: per sub-word order, the folded score for this same repeat swings from 0.53 (ws=3, aligned with the period) up to **1.03** (ws=5, misaligned with the period) — i.e. a period-3 repeat could score *above* the theoretical maximum for a random sequence, depending on which ws happens to divide the repeat's period. The raw formula stays flat at ≈0.33–0.40 across ws=2..6 regardless of alignment, which is the robustness the "minimum across ws" design was meant to provide in the first place.
+
+## Interpretation as an effective number of classes
+
+$H_{\text{corr}}$ is a standard Shannon entropy over raw words, so the classical perplexity interpretation holds directly: $N_{\text{eff}} = e^{H_{\text{corr}}}$ is the number of equiprobable raw words that would yield the same entropy.
+
+For the normalised score $\hat{H}$, dividing by $H_{\max}$ changes the logarithm base:
+
+$$\hat{H} = \frac{\log N_{\text{eff}}}{\log N_{\max}} = \log_{N_{\max}} N_{\text{eff}} \quad \Longleftrightarrow \quad N_{\text{eff}} = N_{\max}^{\,\hat{H}}$$
+
+The property is preserved: $\hat{H}$ is the logarithm (in base $N_{\max}$) of the effective number of equi-represented raw words.
+
+In the large-sample limit ($n_{\text{words}} \gg 4^{ws}$), $N_{\max} \approx 4^{ws}$, giving:
+
+$$N_{\text{eff}} \approx 4^{ws \cdot \hat{H}}$$
+
+This has a clean interpretation: $ws \cdot \hat{H}$ is the **effective word length** (in bases) of a perfectly uniform distribution that would produce the same entropy. At $\hat{H} = 1$ the full space of $4^{ws}$ words is used; at $\hat{H} = 0.5$ with ws=2, only $4^1 = 4$ effective words out of 16 are occupied.
+
+In our actual regime, $n_{\text{words}}$ is small and $4^{ws}$ can exceed $n_{\text{words}}$, so $H_{\max} < \log(4^{ws})$ due to the small-sample correction. The exact effective count is $N_{\max}^{\hat{H}}$, not $4^{ws \cdot \hat{H}}$.
+
+## Properties
+
+The entropy score is a function of the kmer sequence alone — it does not depend on the surrounding context or on the position within any genome. Two consequences:
+
+- **Orientation invariance**: $\text{entropy}(K) = \text{entropy}(\text{revcomp}(K))$ — see *Why no equivalence classes* above for why this holds without any explicit strand-folding step.
+- **Context independence**: the same kmer is always rejected or always kept, regardless of which genome it occurs in, where in that genome it appears, or which strand is considered. The filter defines a fixed partition of the kmer space into low-complexity and valid kmers.

DevDocMD/theory/entropy.refs.md

@@ -0,0 +1,16 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/entropy.md
+
+## Code couvert
+
+- `obikentropy/src/table.rs`, `obikentropy/src/tracker.rs` — formule d'entropie et tables de correction petits effectifs
+- `obikentropy/src/kmer_entropy.rs` — entropie d'un kmer isolé (`KmerEntropy`)
+- `obiskbuilder/src/rolling_stat.rs` — composition de `obikentropy::EntropyTracker` dans le suivi streaming (sélection de minimiseur + entropie)
+- `obiskbuilder/src/iter.rs`, `obiskbuilder/src/stream_iter.rs` — application du filtre lors du scatter (phase 1)
+
+## Notes
+
+Le repli en classes d'équivalence circulaires + brin inverse (décrit dans une version antérieure de ce document) a été supprimé : voir la section « Why no equivalence classes » de `entropy.md` pour la justification théorique et numérique.
+
+Vérifier que les paramètres `theta` et `level_max` dans le CLI
+(`obikmer/src/cli.rs` → `CommonArgs`) correspondent bien à ce qui est décrit.

DevDocMD/theory/indexing.md

@@ -0,0 +1,28 @@
+# Partitioning and indexing architecture
+
+The canonical minimizer of a super-kmer is hashed to produce a **p-bit routing value** (p is a collection-level parameter):
+
+```
+canonical minimizer → hash(minimizer) → p-bit value → PART → partition directory
+```
+
+PART is computed once at phase 1 to open the correct partition file, then discarded. It is recomputed on the fly at query time. It is never stored in the super-kmer header.
+
+Each partition holds one MPHF instance (phase 6) that indexes kmers as plain u64 values — the minimizer plays no role inside the partition.
+
+## Why hashing is necessary
+
+The canonical minimizer is an m-mer (m ∈ {9, 11, 13, 15}), encoded in 2m bits (18 to 30 bits). Its distribution over the $4^m$ possible values is **not uniform**: because the minimizer is the lexicographic minimum of a window of m-mers, small values are systematically over-represented [@Zheng2020-ji; @Zheng2021-cc; @Pan2024-hb; @Kille2023-px; @Golan2025-xf]. Routing directly by the raw minimizer value would produce severely unbalanced partitions.
+
+A hash function with good avalanche properties redistributes this skewed distribution uniformly over the $2^p$ partition slots. The key reason this works well is the **entropy gap**: p is chosen to be much smaller than 2m, so the hash compresses many distinct minimizer values into each partition slot. Even under strong bias in the minimizer distribution, as long as its effective entropy exceeds p bits — which holds comfortably since the set of distinct minimizers in any real dataset is far larger than $2^p$ — the load imbalance across partitions is negligible.
+
+## Parameter choices
+
+| m  | 2m (bits) | Typical p | Partitions |
+|----|-----------|-----------|------------|
+| 9  | 18        | 6–8       | 64–256     |
+| 11 | 22        | 8–10      | 256–1 024  |
+| 13 | 26        | 10–12     | 1 024–4 096|
+| 15 | 30        | 10–14     | 1 024–16 384|
+
+The hard constraint is p ≤ 2m: one cannot extract more bits of uniform randomness from a source than it contains. In practice p is chosen well below 2m, leaving a large entropy margin that absorbs the minimizer bias. For k=31, m=13, p=10: 1 024 partitions with comfortable balance.

DevDocMD/theory/indexing.refs.md

@@ -0,0 +1,12 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/indexing.md
+
+## Code couvert
+
+- `obikpartitionner/src/partition.rs` — routage par hash de minimiseur, choix des paramètres
+- `obikpartitionner/src/lib.rs` — structure KmerPartition, nombre de partitions
+
+## Notes
+
+Vérifier que la doc mentionne bien que le nombre de partitions est une puissance de 2
+(converti par `partitions_to_bits` dans `obikmer/src/cli.rs`).

DevDocMD/theory/minimizer.md

@@ -0,0 +1,73 @@
+# Minimizer selection
+
+## Definition
+
+A **minimizer** of a k-mer window is the m-mer (m < k) with the smallest value under some total order ≺ among all k − m + 1 overlapping m-mers in the window. The minimizer is always taken in **canonical form** (lexicographic minimum of forward and reverse complement) to ensure strand-independence.
+
+The minimizer partitions the sequence into **super-kmers**: maximal contiguous runs of overlapping k-mers that share the same minimizer. A single minimizer anchors each super-kmer, enabling partitioned storage and indexing.
+
+## Lexicographic ordering and its bias
+
+The classical definition uses lexicographic order on the canonical m-mer value. In 2-bit encoding (A=00, C=01, G=10, T=11), the canonical form is $\min_{\text{lex}}(\text{fwd}, \text{rc})$, so AT-rich m-mers have systematically small values:
+
+$$\text{canonical}(\text{AAAA}\cdots\text{A}) = \text{canonical}(\text{TTTT}\cdots\text{T}) = 0$$
+
+Since small values always win the lex comparison, low-complexity AT-rich m-mers dominate as minimizers across large genomic regions. On real metagenomics data with k=31, m=11 and 256 partitions, this produces a max/min partition ratio of ≈ 2.75 — and a single pathological partition when the hash function has a fixed point at 0.
+
+## Random minimizer
+
+A **random minimizer** replaces lex order with a hash order: define $H : \{0,1\}^{2m} \to \{0,1\}^{64}$ and select the m-mer with the **minimum $H$ value** in the window.
+
+The key property: because $H$ is a bijection with well-distributed outputs, each distinct m-mer in the window has equal probability of holding the minimum hash value. Selection probability is no longer correlated with nucleotide composition.
+
+## Why the canonical form remains lexicographic
+
+An apparent alternative is to redefine the canonical form of each m-mer as the strand with the smaller hash value:
+
+$$\text{canonical}_H(v) = \arg\min(H(\text{fwd}),\ H(\text{rc}))$$
+
+This must be rejected. The hash of this new canonical is $\min(H(\text{fwd}), H(\text{rc}))$ — the minimum of two i.i.d. Uniform$[0, 2^{64})$ values. Its distribution is:
+
+$$F(x) = 1 - \left(1 - \frac{x}{2^{64}}\right)^2$$
+
+with density $f(x) = 2(1 - x/2^{64})$, which is approximately **twice as large near 0 than near $2^{64}$**. The low-order partition bits inherit this bias: partition 0 receives roughly twice as many super-kmers as the last partition.
+
+The lex canonical form does not have this problem: $\text{canonical}_{\text{lex}}(v)$ is a fixed, deterministic representative of each equivalence class, and $H(\text{canonical}_{\text{lex}})$ is uniformly distributed over $[0, 2^{64})$ independently of the min/max relationship between the two strands.
+
+## Partition key independence
+
+A further subtlety arises when the selection hash is used directly as the partition key. The selected minimizer is the m-mer with the **minimum** $H$ value in a window of $W = k - m + 1$ positions. The minimum of $W$ i.i.d. Uniform$[0,2^{64})$ values has distribution:
+
+$$F(x) = 1 - \left(1 - \frac{x}{2^{64}}\right)^W \approx \frac{Wx}{2^{64}}$$
+
+concentrated near 0 relative to the full range. Using this minimum-hash directly as the partition key creates the same bias as lex ordering, just distributed differently.
+
+The correct approach is to decouple selection from partition routing:
+
+- **Selection** uses $H(\text{canonical}_{\text{lex}}(m\text{-mer}))$ to pick the minimizer in the window.
+- **Partition routing** recomputes $H(\text{canonical}_{\text{lex}}(\text{minimizer}))$ from the stored minimizer position. This is the hash of a specific kmer value, not the minimum of a window — it is uniformly distributed over $[0, 2^{64})$.
+
+## Seed and fixed-point elimination
+
+The splitmix64 finalizer has a fixed point at 0:
+
+$$\text{mix64}(0) = 0$$
+
+Since $\text{canonical}_{\text{lex}}(\text{AAAA}\cdots\text{A}) = 0$, using unseeded mix64 causes all-A m-mers to win every window comparison, recreating a pathological partition identical to the lex-ordering bias.
+
+The fix is a non-zero XOR seed applied before mixing:
+
+$$H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}$$
+
+where $\varphi$ is the golden ratio. This maps 0 to $\text{mix64}(s)$, a well-distributed non-zero value. No canonical m-mer value has a systematically small $H$.
+
+!!! abstract "Hash function $H$"
+    ```
+    H(x):
+        x ← x  ⊕  0x9e3779b97f4a7c15
+        x ← x  ⊕  (x >> 30)
+        x ← x  ×  0xbf58476d1ce4e5b9
+        x ← x  ⊕  (x >> 27)
+        x ← x  ×  0x94d049bb133111eb
+        return x ⊕ (x >> 31)
+    ```

DevDocMD/theory/minimizer.refs.md

@@ -0,0 +1,12 @@
+<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
+# Coverage: theory/minimizer.md
+
+## Code couvert
+
+- `obiskbuilder/src/lib.rs` — sélection du minimiseur par hash seedé (splitmix64 finalizer)
+- `obikseq/src/superkmer.rs` — forme canonique du minimiseur, fenêtre glissante
+
+## Notes
+
+Vérifier que la fonction de hash décrite (splitmix64 finalizer avec graine) correspond
+au code actuel. Vérifier aussi que la définition de « minimiseur canonique » est toujours cohérente.

DevDocMD/vancouver.csl

@@ -0,0 +1,377 @@
+<?xml version="1.0" encoding="utf-8"?>
+<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" demote-non-dropping-particle="sort-only" initialize-with-hyphen="false" page-range-format="minimal">
+  <info>
+    <title>Vancouver</title>
+    <id>http://www.zotero.org/styles/vancouver</id>
+    <link href="http://www.zotero.org/styles/vancouver" rel="self"/>
+    <link href="http://www.nlm.nih.gov/bsd/uniform_requirements.html" rel="documentation"/>
+    <author>
+      <name>Michael Berkowitz</name>
+      <email>mberkowi@gmu.edu</email>
+    </author>
+    <contributor>
+      <name>Sean Takats</name>
+      <email>stakats@gmu.edu</email>
+    </contributor>
+    <contributor>
+      <name>Sebastian Karcher</name>
+    </contributor>
+    <category citation-format="numeric"/>
+    <category field="generic-base"/>
+    <category field="medicine"/>
+    <summary>Vancouver style as outlined by International Committee of Medical Journal Editors Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Sample References</summary>
+    <updated>2025-05-17T20:55:38-04:00</updated>
+    <rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
+  </info>
+  <locale xml:lang="en">
+    <date form="text" delimiter=" ">
+      <date-part name="year"/>
+      <date-part name="month" form="short" strip-periods="true"/>
+      <date-part name="day"/>
+    </date>
+    <terms>
+      <term name="collection-editor" form="long">
+        <single>editor</single>
+        <multiple>editors</multiple>
+      </term>
+      <term name="presented at">presented at</term>
+      <term name="available at">available from</term>
+      <term name="section" form="short">sect.</term>
+    </terms>
+  </locale>
+  <locale xml:lang="fr">
+    <date form="text" delimiter=" ">
+      <date-part name="day"/>
+      <date-part name="month" form="short" strip-periods="true"/>
+      <date-part name="year"/>
+    </date>
+  </locale>
+  <macro name="author">
+    <names variable="author">
+      <name sort-separator=" " initialize-with="" name-as-sort-order="all" delimiter=", " delimiter-precedes-last="always"/>
+      <label form="long" prefix=", "/>
+      <substitute>
+        <text macro="webpage-title"/>
+        <names variable="editor"/>
+      </substitute>
+    </names>
+  </macro>
+  <macro name="editor">
+    <names variable="editor" suffix=".">
+      <name sort-separator=" " initialize-with="" name-as-sort-order="all" delimiter=", " delimiter-precedes-last="always"/>
+      <label form="long" prefix=", "/>
+    </names>
+  </macro>
+  <macro name="chapter-marker">
+    <choose>
+      <if type="chapter paper-conference entry-dictionary entry-encyclopedia" match="any">
+        <text term="in" text-case="capitalize-first"/>
+      </if>
+    </choose>
+  </macro>
+  <macro name="webpage-title">
+    <!--If a webpage has a container, we're assuming the citation is "part of a website" as per ch. 25 Citing Medicine https://www.ncbi.nlm.nih.gov/books/NBK7274/?report=reader -->
+    <choose>
+      <if type="webpage" variable="container-title" match="all">
+        <group delimiter=" ">
+          <text variable="container-title"/>
+          <text term="internet" prefix="[" suffix="]" text-case="capitalize-first"/>
+        </group>
+      </if>
+    </choose>
+  </macro>
+  <macro name="publisher">
+    <choose>
+      <!--discard publisher info for articles-->
+      <if type="article-journal article-magazine article-newspaper" match="none">
+        <group delimiter=": " suffix=";">
+          <choose>
+            <if type="thesis">
+              <text variable="publisher-place" prefix="[" suffix="]"/>
+            </if>
+            <else-if type="speech"/>
+            <else>
+              <text variable="publisher-place"/>
+            </else>
+          </choose>
+          <text variable="publisher"/>
+        </group>
+      </if>
+    </choose>
+  </macro>
+  <macro name="access">
+    <choose>
+      <if variable="URL">
+        <group delimiter=": ">
+          <text term="available at" text-case="capitalize-first"/>
+          <text variable="URL"/>
+        </group>
+      </if>
+    </choose>
+  </macro>
+  <macro name="accessed-date">
+    <choose>
+      <if variable="URL">
+        <group prefix="[" suffix="]" delimiter=" ">
+          <text term="cited" text-case="lowercase"/>
+          <date variable="accessed" form="text"/>
+        </group>
+      </if>
+    </choose>
+  </macro>
+  <macro name="container-title">
+    <choose>
+      <if type="article-journal article-magazine chapter paper-conference article-newspaper review review-book entry-dictionary entry-encyclopedia" match="any">
+        <group suffix="." delimiter=" ">
+          <choose>
+            <if type="article-journal review review-book" match="any">
+              <text variable="container-title" form="short" strip-periods="true"/>
+            </if>
+            <else>
+              <text variable="container-title" strip-periods="true"/>
+            </else>
+          </choose>
+          <choose>
+            <if variable="URL">
+              <text term="internet" prefix="[" suffix="]" text-case="capitalize-first"/>
+            </if>
+          </choose>
+        </group>
+        <text macro="edition" prefix=" "/>
+      </if>
+      <!--add event-name and event-place once they become available-->
+      <else-if type="bill legislation" match="any">
+        <group delimiter=", ">
+          <group delimiter=". ">
+            <text variable="container-title"/>
+            <group delimiter=" ">
+              <text term="section" form="short" text-case="capitalize-first"/>
+              <text variable="section"/>
+            </group>
+          </group>
+          <text variable="number"/>
+        </group>
+      </else-if>
+      <else-if type="speech">
+        <group delimiter=": " suffix=";">
+          <group delimiter=" ">
+            <text variable="genre" text-case="capitalize-first"/>
+            <text term="presented at"/>
+          </group>
+          <text variable="event"/>
+        </group>
+      </else-if>
+      <else>
+        <group delimiter=", " suffix=".">
+          <choose>
+            <if variable="collection-title" match="none">
+              <group delimiter=" ">
+                <label variable="volume" form="short" text-case="capitalize-first"/>
+                <text variable="volume"/>
+              </group>
+            </if>
+          </choose>
+          <text variable="container-title"/>
+        </group>
+      </else>
+    </choose>
+  </macro>
+  <macro name="title">
+    <choose>
+      <if type="webpage" variable="container-title" match="all"/>
+      <else>
+        <text variable="title"/>
+        <choose>
+          <if type="article-journal article-magazine chapter paper-conference article-newspaper review review-book entry-dictionary entry-encyclopedia" match="none">
+            <choose>
+              <if variable="URL">
+                <text term="internet" prefix=" [" suffix="]" text-case="capitalize-first"/>
+              </if>
+            </choose>
+            <text macro="edition" prefix=". "/>
+          </if>
+        </choose>
+      </else>
+    </choose>
+    <choose>
+      <if type="thesis">
+        <text variable="genre" prefix=" [" suffix="]"/>
+      </if>
+    </choose>
+  </macro>
+  <macro name="edition">
+    <choose>
+      <if is-numeric="edition">
+        <group delimiter=" ">
+          <number variable="edition" form="ordinal"/>
+          <text term="edition" form="short"/>
+        </group>
+      </if>
+      <else>
+        <text variable="edition" suffix="."/>
+      </else>
+    </choose>
+  </macro>
+  <macro name="date">
+    <choose>
+      <if type="article-journal article-magazine article-newspaper review review-book" match="any">
+        <group suffix=";" delimiter=" ">
+          <date variable="issued" form="text"/>
+          <text macro="accessed-date"/>
+        </group>
+      </if>
+      <else-if type="bill legislation" match="any">
+        <group delimiter=", ">
+          <date variable="issued" delimiter=" ">
+            <date-part name="month" form="short" strip-periods="true"/>
+            <date-part name="day"/>
+          </date>
+          <date variable="issued">
+            <date-part name="year"/>
+          </date>
+        </group>
+      </else-if>
+      <else-if type="report">
+        <date variable="issued" delimiter=" ">
+          <date-part name="year"/>
+          <date-part name="month" form="short" strip-periods="true"/>
+        </date>
+        <text macro="accessed-date" prefix=" "/>
+      </else-if>
+      <else-if type="patent">
+        <group suffix=".">
+          <group delimiter=", ">
+            <text variable="number"/>
+            <date variable="issued">
+              <date-part name="year"/>
+            </date>
+          </group>
+          <text macro="accessed-date" prefix=" "/>
+        </group>
+      </else-if>
+      <else-if type="speech">
+        <group delimiter="; ">
+          <group delimiter=" ">
+            <date variable="issued" delimiter=" ">
+              <date-part name="year"/>
+              <date-part name="month" form="short" strip-periods="true"/>
+              <date-part name="day"/>
+            </date>
+            <text macro="accessed-date"/>
+          </group>
+          <text variable="event-place"/>
+        </group>
+      </else-if>
+      <else>
+        <group suffix=".">
+          <date variable="issued">
+            <date-part name="year"/>
+          </date>
+          <text macro="accessed-date" prefix=" "/>
+        </group>
+      </else>
+    </choose>
+  </macro>
+  <macro name="pages">
+    <choose>
+      <if type="article-journal article-magazine article-newspaper review review-book" match="any">
+        <text variable="page" prefix=":"/>
+      </if>
+      <else-if type="book" match="any">
+        <text variable="number-of-pages" prefix=" "/>
+        <choose>
+          <if is-numeric="number-of-pages">
+            <label variable="number-of-pages" form="short" prefix=" " plural="never"/>
+          </if>
+        </choose>
+      </else-if>
+      <else>
+        <group prefix=" " delimiter=" ">
+          <label variable="page" form="short" plural="never"/>
+          <text variable="page"/>
+        </group>
+      </else>
+    </choose>
+  </macro>
+  <macro name="journal-location">
+    <choose>
+      <if type="article-journal article-magazine review review-book" match="any">
+        <text variable="volume"/>
+        <text variable="issue" prefix="(" suffix=")"/>
+      </if>
+    </choose>
+  </macro>
+  <macro name="webpage-part">
+    <choose>
+      <if type="webpage" variable="container-title" match="all">
+        <text variable="title"/>
+      </if>
+    </choose>
+  </macro>
+  <macro name="collection-details">
+    <choose>
+      <if type="article-journal article-magazine article-newspaper review review-book" match="none">
+        <choose>
+          <if variable="collection-title">
+            <group delimiter=" " prefix="(" suffix=")">
+              <names variable="collection-editor" suffix=".">
+                <name sort-separator=" " initialize-with="" name-as-sort-order="all" delimiter=", " delimiter-precedes-last="always"/>
+                <label form="long" prefix=", "/>
+              </names>
+              <group delimiter="; ">
+                <text variable="collection-title"/>
+                <group delimiter=" ">
+                  <label variable="volume" form="short"/>
+                  <text variable="volume"/>
+                </group>
+              </group>
+            </group>
+          </if>
+        </choose>
+      </if>
+    </choose>
+  </macro>
+  <macro name="report-details">
+    <choose>
+      <if type="report">
+        <text variable="number" prefix="Report No.: "/>
+      </if>
+    </choose>
+  </macro>
+  <citation collapse="citation-number">
+    <sort>
+      <key variable="citation-number"/>
+    </sort>
+    <layout prefix="(" suffix=")" delimiter=",">
+      <text variable="citation-number"/>
+    </layout>
+  </citation>
+  <bibliography et-al-min="7" et-al-use-first="6" second-field-align="flush">
+    <layout>
+      <text variable="citation-number" suffix="."/>
+      <group delimiter=". " suffix=". ">
+        <text macro="author"/>
+        <text macro="title"/>
+      </group>
+      <group delimiter=" " suffix=". ">
+        <group delimiter=": ">
+          <text macro="chapter-marker"/>
+          <group delimiter=" ">
+            <text macro="editor"/>
+            <text macro="container-title"/>
+          </group>
+        </group>
+        <text macro="publisher"/>
+        <group>
+          <text macro="date"/>
+          <text macro="journal-location"/>
+          <text macro="pages"/>
+        </group>
+        <text macro="webpage-part"/>
+      </group>
+      <text macro="collection-details" suffix=". "/>
+      <text macro="report-details" suffix=". "/>
+      <text macro="access"/>
+    </layout>
+  </bibliography>
+</style>