large refactoring
This commit is contained in:
Eric Coissac
@@ -0,0 +1,125 @@
|
||||
# Chunk reader — implementation
|
||||
|
||||
`obiread` exposes two distinct sequence reading paths, each optimised for a different use case.
|
||||
|
||||
## Two reading paths
|
||||
|
||||
| Path | API | Output unit | Per-record identity | Use case |
|
||||
|------|-----|-------------|---------------------|----------|
|
||||
| **Record path** | `read_sequence_chunks` → `parse_chunk` | `SeqRecord` (id + raw sequence + normalised rope) | yes | `query` — must read complete records |
|
||||
| **Stream path** | `open_nuc_stream` | `NucPage` (flat normalised byte buffer) | no | `index`, `superkmer` — bulk throughput |
|
||||
|
||||
The record path uses `Rope`-backed chunks and is described in detail below.
|
||||
The stream path (`NucStream` / `NucPage`) is described in the scatter section of [pipeline](pipeline.md).
|
||||
|
||||
---
|
||||
|
||||
## Record path: chunk reader
|
||||
|
||||
The chunk reader reads FASTA or FASTQ files in fixed-size blocks and yields self-contained chunks, each ending on a complete sequence record boundary. `parse_chunk` then converts each chunk into a `Vec<SeqRecord>`, where each record carries its identifier, raw sequence bytes, and a normalised rope ready for superkmer building.
|
||||
|
||||
This path is mandatory for `query`, where superkmers must be tracked back to their originating sequence (id, kmer offset) for output annotation.
|
||||
|
||||
## Output type: Rope
|
||||
|
||||
Each chunk is a `Rope` — a segmented byte sequence: a `Vec` of blocks, where each block is a `Vec<Cell<u8>>`. The consumer iterates over the blocks via a forward or backward cursor.
|
||||
|
||||
`Rope::split_off(pos)` splits at an absolute byte offset in O(log n) (binary search over block-start index). If `pos` falls inside a block, that block is split in two via `Vec::split_off` — no `memcpy` in the common case.
|
||||
|
||||
## SeqChunkIter
|
||||
|
||||
```rust
|
||||
pub struct SeqChunkIter<R: Read> { /* private */ }
|
||||
|
||||
impl<R: Read> Iterator for SeqChunkIter<R> {
|
||||
type Item = io::Result<Rope>;
|
||||
}
|
||||
|
||||
pub fn fasta_chunks<R: Read>(source: R) -> SeqChunkIter<R>
|
||||
pub fn fastq_chunks<R: Read>(source: R) -> SeqChunkIter<R>
|
||||
```
|
||||
|
||||
`next()` loop:
|
||||
|
||||
```text
|
||||
1. read one block of block_size bytes → push onto Rope
|
||||
2. call splitter(rope) → Option<abs_offset>
|
||||
if Some(pos):
|
||||
tail = rope.split_off(pos) ← O(log n), may split one block
|
||||
chunk = mem::replace(&mut rope, tail)
|
||||
return Some(Ok(chunk))
|
||||
3. if EOF and rope non-empty: return Some(Ok(rope)) as final chunk
|
||||
4. if EOF and rope empty: return None
|
||||
```
|
||||
|
||||
The `Splitter` function signature is `fn(&Rope) -> Option<usize>`. It returns the absolute byte offset of the start of the last complete record, or `None` if no boundary was found in the accumulated rope (need more data).
|
||||
|
||||
## Boundary detection — FASTA
|
||||
|
||||
Backward scan with a 2-state machine. Searches (right to left) for `>` followed by `\n` or `\r` (i.e., a `>` that is preceded by a newline in forward order):
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
direction LR
|
||||
[*] --> Scanning
|
||||
Scanning --> FoundGt : '>'
|
||||
FoundGt --> Scanning : other
|
||||
FoundGt --> [*] : '\\n' / '\\r' ✓
|
||||
```
|
||||
|
||||
Returns the byte offset of the `>` that starts the last complete record. Returns `None` if only one `>` is found (cannot confirm there is a prior complete record).
|
||||
|
||||
## Boundary detection — FASTQ
|
||||
|
||||
FASTQ records have a rigid 4-line structure (`@header`, sequence, `+`, quality). The `@` character (ASCII 64, Phred score 31) can appear legitimately in quality lines, making any forward heuristic unreliable. The backward scanner verifies the full structural context before accepting a candidate `@`.
|
||||
|
||||
7-state machine (states 0–6), scanning from **right to left**. Each time a `+` is found, its position is saved as `restart`; any state mismatch resets the scan to that position.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
direction LR
|
||||
|
||||
[*] --> Scanning
|
||||
|
||||
Scanning --> FoundPlus : '+' (save restart)
|
||||
FoundPlus --> AfterNlPlus : '\\n' / '\\r'
|
||||
FoundPlus --> Scanning : other → backtrack
|
||||
|
||||
AfterNlPlus --> AfterNlPlus : séparateur
|
||||
AfterNlPlus --> InSequence : lettre / - / . / [ / ]
|
||||
AfterNlPlus --> Scanning : other → backtrack
|
||||
|
||||
InSequence --> AfterSequence : '\\n' / '\\r'
|
||||
InSequence --> InSequence : lettre / - / . / [ / ]
|
||||
InSequence --> Scanning : other → backtrack
|
||||
|
||||
AfterSequence --> AfterSequence : '\\n' / '\\r'
|
||||
AfterSequence --> InHeader : other
|
||||
|
||||
InHeader --> FoundAt : '@' (save cut)
|
||||
InHeader --> Scanning : '\\n' / '\\r' → backtrack
|
||||
InHeader --> InHeader : other
|
||||
|
||||
FoundAt --> [*] : '\\n' / '\\r' ✓
|
||||
FoundAt --> InHeader : other
|
||||
```
|
||||
|
||||
`restart` is updated each time a `+` is found. When any state fails its expected input, the scan jumps back to `restart` and continues from there — guaranteeing that a `@` in a quality line cannot be accepted as a record start, because the `\n+\n` structure immediately following it (going backward) will not be found.
|
||||
|
||||
Returns the byte offset of the `@` that starts the last complete record.
|
||||
|
||||
---
|
||||
|
||||
## Future work — parallel gzip decompression in `xopen`
|
||||
|
||||
`obiread::xopen` (`xopen.rs`) decompresses gzip via `niffler` → `flate2`, which is single-threaded (standard DEFLATE has no parallel-decodable structure). For large local gzip inputs this single-threaded decompression can become the throughput bottleneck feeding the `query`/`index`/`superkmer` pipelines, since chunk/page production for a given file is serialized ahead of the worker pool.
|
||||
|
||||
Candidate: special-case local, on-disk, gzip-magic-detected paths in `open_raw`/`xopen` to use [`rapidgzip-rs`](https://github.com/alekseizarubin/rapidgzip-rs) (`ReaderBuilder::new().parallelism(n).open(path)`, implements `Read + Seek`) instead of `niffler`, keeping `niffler` for every other case: `stdin` (`-`), HTTP(S) sources, and all non-gzip formats (bzip2, xz, zstd — less used in practice here).
|
||||
|
||||
Constraints identified so far (not yet validated against real data):
|
||||
- Branch point must move earlier than the current `decompress()` call in `open_raw` — rapidgzip's fast path needs the file **path**, not an already-opened generic `Read`, so the gzip/local-file detection has to happen before the generic `File::open` + `niffler::send::get_reader` path is taken.
|
||||
- `stdin` and HTTP sources are not seekable — they stay on `niffler` regardless; the gain only applies to local on-disk `.gz` files.
|
||||
- `rapidgzip-sys` vendors a native C++ engine: requires CMake ≥ 3.17, a C++17 compiler, and `nasm` on x86 targets — a real build-toolchain addition, not just a pure-Rust crate.
|
||||
- Low maturity of the Rust binding at review time (2 GitHub stars, ~15 commits, April 2026 latest release) — the underlying C++ engine is validated (HPDC 2023 paper), but the binding itself has limited production track record.
|
||||
|
||||
Decision: parked for now. Before adopting, validate on real data: throughput vs. `niffler` on representative large `.gz` inputs, and byte-for-byte correctness of decompressed output.
|
||||
@@ -0,0 +1,12 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/chunkreader.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obiread/src/chunk.rs` — SeqChunkIter, détection de frontières FASTA/FASTQ, state machines
|
||||
- `obikrope/src/lib.rs` — type Rope (Vec<Bytes>), opérations zero-copy
|
||||
|
||||
## Notes
|
||||
|
||||
Document stable (la stratégie de chunking rope ne devrait pas avoir changé).
|
||||
Vérifier que le split FASTA/FASTQ reste correct si de nouveaux formats ont été ajoutés.
|
||||
@@ -0,0 +1,181 @@
|
||||
# Approximate evidence: fingerprint-based index
|
||||
|
||||
## Motivation
|
||||
|
||||
`evidence.bin` maps each MPHF slot to the position of the k-mer that owns it,
|
||||
enabling zero-FP verification. On the bacterial BCT dataset (2048 partitions,
|
||||
k=31, ~33 M k-mers/partition) it accounts for 66 % of the lookup-layer footprint:
|
||||
|
||||
| file | size/partition | fraction |
|
||||
|---|---|---|
|
||||
| evidence.bin | 132 MB | 66 % |
|
||||
| unitigs.bin | 58 MB | 29 % |
|
||||
| mphf.bin | 10 MB | 5 % |
|
||||
|
||||
`evidence.bin` is a bijection from MPHF-space to unitig-position-space and
|
||||
costs at minimum ⌈log₂ N⌉ bits per slot — an information-theoretic floor with
|
||||
only ~22 % packing headroom. Compression is not a path to elimination.
|
||||
|
||||
The approximate index replaces `evidence.bin` + `unitigs.bin.idx` with a
|
||||
`fingerprint.bin` file. The MPHF and `unitigs.bin` are kept unchanged. Set
|
||||
operations still require an exact index; the approximate index targets query
|
||||
workloads that can tolerate a bounded false-positive rate.
|
||||
|
||||
---
|
||||
|
||||
## The Findere model
|
||||
|
||||
A B-bit fingerprint stored per MPHF slot provides the discrimination that
|
||||
`evidence.bin` would otherwise provide through full k-mer reconstruction.
|
||||
|
||||
For a foreign k-mer query, the MPHF maps it to some slot `s`. The fingerprint
|
||||
stored at `s` belongs to the legitimate k-mer at that slot. The FP event is:
|
||||
|
||||
```
|
||||
P(FP per k-mer) = 1 / 2^b
|
||||
```
|
||||
|
||||
The Findere trick reduces the indexed k-mer size. When the user specifies k_user
|
||||
and z, the index physically stores k-mers of size `s = k_user − z + 1`. At query
|
||||
time, the same s-mer size is used. After collecting per-position s-mer results
|
||||
over the full query sequence, a sliding window of size z aggregates z consecutive
|
||||
s-mer hits into one confirmed k_user-mer hit, reducing the per-window FP rate:
|
||||
|
||||
```
|
||||
P(FP per k_user-mer) = 1 / 2^(b·z)
|
||||
```
|
||||
|
||||
`IndexConfig::kmer_size` stores `s = k_user − z + 1`, not k_user. Both indexing
|
||||
and querying use this stored size via `set_k(idx.kmer_size())`.
|
||||
|
||||
Parameters b and z are stored in `layer_meta.json` (`EvidenceKind::Approx { b, z }`).
|
||||
|
||||
---
|
||||
|
||||
## `FingerprintVec` on disk
|
||||
|
||||
`fingerprint.bin` layout:
|
||||
|
||||
```
|
||||
magic: b"FPVF" (4 bytes)
|
||||
b: u8 (bits per slot, 1..=64)
|
||||
padding: [0u8; 3]
|
||||
n: u64 LE (number of slots)
|
||||
data: packed bits, ceil(n·b/8) bytes, Lsb0 order
|
||||
```
|
||||
|
||||
`FingerprintVec` is memory-mapped. The match check against a query k-mer:
|
||||
|
||||
```rust
|
||||
fn matches(&self, slot: usize, fingerprint: u64) -> bool {
|
||||
self.get(slot) == (fingerprint & self.mask)
|
||||
}
|
||||
```
|
||||
|
||||
`build_approx_evidence` iterates `unitigs.bin` sequentially, writes
|
||||
`kmer.seq_hash()` into the slot assigned by the MPHF, then saves `fingerprint.bin`
|
||||
and `layer_meta.json`. No `.idx` file is produced; random access into
|
||||
`unitigs.bin` is not needed.
|
||||
|
||||
At build time, `find_approx` in `MphfLayer`:
|
||||
|
||||
```rust
|
||||
let slot = self.mphf.index(&kmer.raw());
|
||||
if fingerprint.matches(slot, kmer.seq_hash()) { Some(slot) } else { None }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## `EvidenceKind` and metadata
|
||||
|
||||
`layer_meta.json` records which evidence bundle is present:
|
||||
|
||||
```rust
|
||||
pub enum EvidenceKind {
|
||||
Exact,
|
||||
Approx { b: u8, z: u8 },
|
||||
}
|
||||
```
|
||||
|
||||
`MphfLayer::open` reads this tag and dispatches `find` to `find_exact` or
|
||||
`find_approx` transparently. `find_exact` panics on an approximate layer;
|
||||
`find_approx` panics on an exact layer — mode mixing is a programming error.
|
||||
|
||||
---
|
||||
|
||||
## Parameter resolution (`resolve_approx_params`)
|
||||
|
||||
The identity `b·z = ⌈−log₂(fp)⌉` lets any two of (b, z, fp) derive the third.
|
||||
`resolve_approx_params` implements a 2-of-3 rule with conservative ceiling
|
||||
rounding:
|
||||
|
||||
| given | derived |
|
||||
|---|---|
|
||||
| b, z | fp = 1/2^(b·z) |
|
||||
| z, fp | b = ⌈−log₂(fp) / z⌉ |
|
||||
| b, fp | z = ⌈−log₂(fp) / b⌉ |
|
||||
| z only | b = 8 (default), fp derived |
|
||||
| b only | z = 1 (default), fp derived |
|
||||
| fp only | b = 8 (default), z derived |
|
||||
| none | b = 8, z = 1, fp = 1/256 |
|
||||
|
||||
When all three are given, b and z are authoritative and fp is recomputed.
|
||||
|
||||
---
|
||||
|
||||
## CLI flags
|
||||
|
||||
Both `index` and `reindex` accept the same flags:
|
||||
|
||||
| flag | type | meaning |
|
||||
|---|---|---|
|
||||
| `--approx` | bool | enable fingerprint evidence |
|
||||
| `--evidence-bits` (`b`) | u8 | fingerprint bits per slot |
|
||||
| `-z` | u8 | Findere z parameter |
|
||||
| `--fp` | f64 | target FP rate per z-window |
|
||||
| `--block-size` | usize | unitig block size for exact `.idx`; ignored in approx mode |
|
||||
|
||||
`--approx` must be set explicitly; the other three flags are optional and
|
||||
resolved by the 2-of-3 rule. Omitting all three produces b=8, z=1.
|
||||
|
||||
---
|
||||
|
||||
## `reindex` command
|
||||
|
||||
`reindex` converts an existing index between exact and approximate evidence
|
||||
in-place across all partitions and layers, running partitions in parallel via
|
||||
Rayon.
|
||||
|
||||
Conversion to approximate (`--approx`):
|
||||
|
||||
- Builds `fingerprint.bin` from `unitigs.bin` + `mphf.bin`.
|
||||
- Removes `evidence.bin` and `unitigs.bin.idx`.
|
||||
- Updates `layer_meta.json` with `EvidenceKind::Approx { b, z }`.
|
||||
|
||||
Conversion to exact (default, no `--approx`):
|
||||
|
||||
- Builds `evidence.bin` + `unitigs.bin.idx` from `unitigs.bin` + `mphf.bin`.
|
||||
- Removes `fingerprint.bin`.
|
||||
- Updates `layer_meta.json` with `EvidenceKind::Exact`.
|
||||
|
||||
The root `index.meta` is updated with the new evidence kind on success.
|
||||
`mphf.bin` and `unitigs.bin` are never modified.
|
||||
|
||||
---
|
||||
|
||||
## `estimate` command
|
||||
|
||||
`estimate` is a dry-run that resolves and prints (b, z, fp) without touching
|
||||
any index. It accepts the same `--evidence-bits`, `-z`, and `--fp` flags and
|
||||
additionally accepts `-k` to display the effective indexed k-mer length:
|
||||
|
||||
```
|
||||
k (user): 31
|
||||
k (indexed, s=k-z+1): 27
|
||||
z: 5
|
||||
evidence bits (b): 8
|
||||
FP per s-mer: 3.906e-3 (1/2^8)
|
||||
FP per k-mer window: 9.537e-7 (1/2^(8·5))
|
||||
```
|
||||
|
||||
Useful for choosing parameters before committing to an index build.
|
||||
@@ -0,0 +1,22 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/evidence_elimination.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, stockage b bits/slot, matches()
|
||||
- `obilayeredmap/src/mphf_layer.rs` — build_approx_evidence(dir, b, z), find_approx()
|
||||
- `obilayeredmap/src/meta.rs` — EvidenceKind::Approx { b, z }, LayerMeta
|
||||
- `obikindex/src/reindex.rs` — KmerIndex::reindex(), conversion exact↔approx en place
|
||||
- `obikmer/src/cmd/reindex.rs` — CLI reindex, options --approx, -z, --evidence-bits, --fp, --block-size
|
||||
- `obikmer/src/cmd/index.rs` — resolve_approx_params(), options --approx, -z, --evidence-bits, --fp
|
||||
- `obikmer/src/cmd/estimate.rs` — commande estimate (dry-run des paramètres)
|
||||
|
||||
## Notes
|
||||
|
||||
Ce document était à l'origine une discussion de design (4 approches). L'implémentation
|
||||
a maintenant convergé vers l'approche fingerprint (Findere-style).
|
||||
FORT RISQUE DE DÉRIVE — le contenu est probablement un mélange de design et d'implémentation :
|
||||
- Le modèle FP = 1/2^(b·z) et les règles de résolution (2-of-3 parmi b, z, fp) sont implémentés
|
||||
- La commande `reindex` permet la conversion a posteriori exact↔approx
|
||||
- La commande `estimate` fait le dry-run des paramètres
|
||||
Cette page doit être réécrite pour documenter l'implémentation Findere réelle plutôt que les alternatives abandonnées.
|
||||
@@ -0,0 +1,320 @@
|
||||
# Kmer filtering and ingroup/outgroup predicates
|
||||
|
||||
The `filter`, `dump`, and `unitig` commands share the same filtering system,
|
||||
implemented as a shared `FilterArgs` clap argument group embedded in each command
|
||||
via `#[command(flatten)]`. Filters select k-mers based on per-genome quorum
|
||||
counts, optionally restricted to **ingroup** and **outgroup** genome sets derived
|
||||
from genome metadata. All rules described here apply identically to all three commands.
|
||||
|
||||
`filter` additionally accepts `--min-total-count` / `--max-total-count` filters
|
||||
that operate on the sum of counts across all genomes.
|
||||
|
||||
## Predicate syntax
|
||||
|
||||
Each `--ingroup` and `--outgroup` flag takes a predicate of the form:
|
||||
|
||||
```
|
||||
key OP value1|value2|…
|
||||
```
|
||||
|
||||
| Operator | Meaning |
|
||||
|----------|---------|
|
||||
| `*` or `all` | wildcard — every genome matches unconditionally |
|
||||
| `key=v1\|v2` | exact match — genome's `key` equals `v1` or `v2` |
|
||||
| `key!=v` | negation — genome's `key` equals none of the values |
|
||||
| `key~path` | path ancestry — genome's `key` is `path` or a descendant |
|
||||
| `key!~path` | not a descendant |
|
||||
|
||||
Multiple values separated by `|` are always OR-ed within the predicate.
|
||||
|
||||
### Path matching (`~` and `!~`)
|
||||
|
||||
Metadata values can represent hierarchical concept paths such as
|
||||
`/Eukaryota/Viridiplantae/Streptophyta/Betulaceae/Betula/nana`.
|
||||
|
||||
Stored taxonomy values always start with `/` (the root of the path).
|
||||
Query patterns do **not** need to start with `/` — a leading `/` is an optional
|
||||
start anchor, not a requirement.
|
||||
|
||||
| Pattern form | Semantics |
|
||||
|---|---|
|
||||
| `A/B` | contiguous sub-path A then B, anywhere in the value |
|
||||
| `/A/B` | value starts with A then B |
|
||||
| `A/B$` | value ends with A then B |
|
||||
| `/A/B$` | value is exactly A then B |
|
||||
| `A@x/B` | A with class `x` followed by B with any class |
|
||||
|
||||
- `taxon~/Betulaceae/Betula` matches any path that starts with `Betulaceae` then `Betula`.
|
||||
- `taxon~Betula` matches any path containing `Betula` as a segment, anywhere.
|
||||
|
||||
### Missing metadata key → NA
|
||||
|
||||
If a genome does not carry the queried metadata key, the predicate returns **NA**.
|
||||
NA propagates through the group evaluation logic (see below), and genomes that
|
||||
cannot be classified are **ignored** in all quorum counts.
|
||||
|
||||
## Group semantics
|
||||
|
||||
### Multiple predicates
|
||||
|
||||
| Flag | Combination rule |
|
||||
|------|-----------------|
|
||||
| `--ingroup` (repeated) | **AND** — genome must satisfy all predicates |
|
||||
| `--outgroup` (repeated) | **OR** — genome satisfies any predicate |
|
||||
|
||||
### Three-value logic
|
||||
|
||||
Each predicate returns `true`, `false`, or `NA` (absent key).
|
||||
|
||||
- AND: `false` absorbs everything; `NA` propagates unless already `false`.
|
||||
- OR: `true` absorbs everything; `NA` propagates unless already `true`.
|
||||
|
||||
### Classification and priority
|
||||
|
||||
For each genome:
|
||||
|
||||
1. Evaluate `AND(ingroup predicates)` → `in_result`
|
||||
2. Evaluate `OR(outgroup predicates)` → `out_result`
|
||||
3. If `in_result = true` → **Ingroup** (ingroup wins over outgroup)
|
||||
4. Else if `out_result = true` → **Outgroup**
|
||||
5. Otherwise → **Uncategorized** (ignored in all quorum counts)
|
||||
|
||||
### Implicit groups
|
||||
|
||||
| `--ingroup` | `--outgroup` | Effective behaviour |
|
||||
|-------------|--------------|---------------------|
|
||||
| not set | not set | all genomes form the ingroup |
|
||||
| set | not set | only ingroup quorum flags apply |
|
||||
| not set | set | only outgroup quorum flags apply |
|
||||
| set | set | both constraints apply simultaneously |
|
||||
|
||||
## Quorum flags
|
||||
|
||||
| Flag | Applies to | Meaning |
|
||||
|------|-----------|---------|
|
||||
| `--min-count N` | ingroup | k-mer present in at least N ingroup genomes (N may be negative, see below) |
|
||||
| `--max-count N` | ingroup | k-mer present in at most N ingroup genomes (N may be negative, see below) |
|
||||
| `--min-frac F` | ingroup | k-mer present in at least fraction F of ingroup genomes |
|
||||
| `--max-frac F` | ingroup | k-mer present in at most fraction F of ingroup genomes |
|
||||
| `--min-outgroup-count N` | outgroup | k-mer present in at least N outgroup genomes (N may be negative, see below) |
|
||||
| `--max-outgroup-count N` | outgroup | k-mer present in at most N outgroup genomes (N may be negative, see below) |
|
||||
| `--min-outgroup-frac F` | outgroup | k-mer present in at least fraction F of outgroup genomes |
|
||||
| `--max-outgroup-frac F` | outgroup | k-mer present in at most fraction F of outgroup genomes |
|
||||
| `--min-total-count N` | all genomes | sum of per-genome counts ≥ N (`filter` only) |
|
||||
| `--max-total-count N` | all genomes | sum of per-genome counts ≤ N (`filter` only) |
|
||||
| `--presence-threshold N` | all | per-genome count > N to be considered "present" (default 0) |
|
||||
|
||||
### Negative counts — offset from group size
|
||||
|
||||
The four integer count flags (`--min-count`, `--max-count`, `--min-outgroup-count`,
|
||||
`--max-outgroup-count`) accept **negative** values, interpreted as an offset counted
|
||||
down from the group size `n`, resolved at run time once `n` is known:
|
||||
|
||||
| Value | Effective threshold |
|
||||
|-------|---------------------|
|
||||
| `N ≥ 0` | literal absolute count `N` |
|
||||
| `-x` (x > 0) | `max(1, n − x)` — "all but x" |
|
||||
|
||||
`-1` literally means *all but one*, `-2` *all but two*, and so on. This expresses
|
||||
a quorum relative to the group size that a plain fraction cannot state exactly
|
||||
(e.g. "present in every genome except at most one" is `n−1`, which is `0.9` for
|
||||
`n = 10` but `0.857…` for `n = 7`).
|
||||
|
||||
The threshold is **floored at 1**, never 0: the negative form always keeps
|
||||
constraining the group. Without the floor, `--min-count -1` on a singleton
|
||||
ingroup (`n = 1`) would resolve to `0` ("at least 0") and silently drop the
|
||||
constraint; the floor makes it `1` ("present in that one genome") instead.
|
||||
|
||||
To express a count of `0` (e.g. "absent from the ingroup"), use the literal `0`,
|
||||
not a negative — `0` and `-0` are indistinguishable, so the offset form starts at
|
||||
`-1`.
|
||||
|
||||
> **Edge case** — on an *empty* group (`n = 0`, e.g. a predicate matching no
|
||||
> genome), a negative count still resolves to `1`, an impossible constraint that
|
||||
> rejects every k-mer. This is consistent with an empty group letting nothing
|
||||
> through, but differs from the "no constraint" behaviour of the fraction flags.
|
||||
|
||||
**Conditional defaults** — the defaults for `--min-frac` and `--max-outgroup-count` depend on two conditions:
|
||||
whether the corresponding group was declared, **and** whether any quorum flag for that group was explicitly set.
|
||||
|
||||
> **Rule**: declaring a group activates the smart default **only if no quorum flag for that group is explicitly set**.
|
||||
> As soon as any quorum flag for a group is present on the command line, all defaults for that group revert to no-op values.
|
||||
|
||||
| `--ingroup` | Any ingroup quorum flag? | `--min-frac` default |
|
||||
|-------------|--------------------------|----------------------|
|
||||
| not set | — | 0.0 (no-op) |
|
||||
| set | no | **1.0** — all ingroup genomes must carry the k-mer |
|
||||
| set | yes | 0.0 — user controls quorum explicitly |
|
||||
|
||||
| `--outgroup` | Any outgroup quorum flag? | `--max-outgroup-count` default |
|
||||
|--------------|---------------------------|-------------------------------|
|
||||
| not set | — | outgroup size (no-op) |
|
||||
| set | no | **0** — no outgroup genome may carry the k-mer |
|
||||
| set | yes | outgroup size — user controls quorum explicitly |
|
||||
|
||||
"Any ingroup quorum flag" means any of: `--min-count`, `--max-count`, `--min-frac`, `--max-frac`.
|
||||
"Any outgroup quorum flag" means any of: `--min-outgroup-count`, `--max-outgroup-count`, `--min-outgroup-frac`, `--max-outgroup-frac`.
|
||||
|
||||
**Why this rule?** Setting any quorum flag signals explicit intent — the defaults are there to help when the user omits quorum entirely, not to interfere with deliberate constraints. Mixing implicit and explicit quorum on the same group would risk silent incoherence (e.g. `--max-count 0` with an implicit `--min-frac 1.0`).
|
||||
|
||||
All other bounds default to 0 / group size / 0.0 / 1.0 regardless of whether groups are declared.
|
||||
|
||||
### Validation
|
||||
|
||||
After resolving defaults, the following are checked and cause an immediate error:
|
||||
|
||||
| Condition | Error |
|
||||
|-----------|-------|
|
||||
| `--min-count > --max-count` | incoherent bounds |
|
||||
| `--min-frac > --max-frac` | incoherent bounds |
|
||||
| `--min-outgroup-count > --max-outgroup-count` | incoherent bounds |
|
||||
| `--min-outgroup-frac > --max-outgroup-frac` | incoherent bounds |
|
||||
| any fraction outside `[0.0, 1.0]` | invalid value |
|
||||
|
||||
The check applies to the **effective** values (after defaults are resolved), so an explicit `--max-frac 0.5` with an implicit `--min-frac 1.0` would have been caught — but the rule above prevents that situation from arising in the first place.
|
||||
|
||||
Fractions are computed over the size of the classified group, not over total
|
||||
genome count. An empty group (no genome classified as ingroup/outgroup) never
|
||||
triggers a filter failure.
|
||||
|
||||
### Conservative rounding of fraction thresholds
|
||||
|
||||
When a fraction threshold `F` is applied to a group of size `N`, the effective
|
||||
integer threshold is determined by the direction of the bound:
|
||||
|
||||
| Bound | Effective count | Rounding | Rationale |
|
||||
|-------|----------------|----------|-----------|
|
||||
| `--min-frac F` | k-mer in ≥ ⌈F·N⌉ genomes | **ceil** | stricter — a kmer present in exactly ⌊F·N⌋ genomes does not meet the fraction |
|
||||
| `--max-frac F` | k-mer in ≤ ⌊F·N⌋ genomes | **floor** | stricter — a kmer present in ⌈F·N⌉ genomes already exceeds the fraction |
|
||||
|
||||
The same rule applies symmetrically to `--min-outgroup-frac` (ceil) and
|
||||
`--max-outgroup-frac` (floor). The outgroup direction is not inverted: the
|
||||
conservative rounding depends only on whether the bound is a minimum or a
|
||||
maximum, not on which group it applies to.
|
||||
|
||||
**Example** — `--min-frac 0.5` with an ingroup of 3 genomes:
|
||||
`⌈0.5 × 3⌉ = ⌈1.5⌉ = 2` → at least 2 of 3 ingroup genomes must carry the k-mer.
|
||||
|
||||
**Implementation note** — the filter evaluates `n / denom < min_frac` directly
|
||||
(integer `n`, float comparison) rather than pre-computing `⌈F·N⌉`. This is
|
||||
mathematically equivalent for integer counts: `n / N < F` ↔ `n < F·N` ↔
|
||||
`n ≤ ⌈F·N⌉ − 1` ↔ `n < ⌈F·N⌉`. No explicit rounding is needed.
|
||||
|
||||
## Examples
|
||||
|
||||
Keep k-mers specific to *Betula nana* — present in at least 2 *B. nana* genomes
|
||||
and absent from every other genome in the index:
|
||||
|
||||
```sh
|
||||
obikmer filter src --output dst \
|
||||
--ingroup "species=Betula_nana" \
|
||||
--outgroup "*" \
|
||||
--min-count 2 \
|
||||
--max-outgroup-count 0
|
||||
```
|
||||
|
||||
Keep k-mers found in at least 2 *Betula nana* genomes and absent from all
|
||||
other *Betula*:
|
||||
|
||||
```sh
|
||||
obikmer filter src --output dst \
|
||||
--ingroup "species=Betula_nana" \
|
||||
--outgroup "genus=Betula" \
|
||||
--min-count 2 \
|
||||
--max-outgroup-count 0
|
||||
```
|
||||
|
||||
Use taxonomic paths — keep k-mers present in ≥ 50 % of the *Betula* clade
|
||||
and in fewer than 10 % of everything outside *Betulaceae*:
|
||||
|
||||
```sh
|
||||
obikmer filter src --output dst \
|
||||
--ingroup "taxon~/Betulaceae/Betula" \
|
||||
--outgroup "taxon!~/Betulaceae" \
|
||||
--min-frac 0.5 \
|
||||
--max-outgroup-frac 0.1
|
||||
```
|
||||
|
||||
Multiple outgroup predicates (OR): exclude k-mers present in *Alnus* or *Carpinus*:
|
||||
|
||||
```sh
|
||||
obikmer filter src --output dst \
|
||||
--ingroup "genus=Betula" \
|
||||
--outgroup "genus=Alnus" \
|
||||
--outgroup "genus=Carpinus" \
|
||||
--max-outgroup-count 0
|
||||
```
|
||||
|
||||
Noise-tolerant core — keep k-mers present in *all but one* ingroup genome
|
||||
(`-1` = `n−1`) and absent from *all but one* of the outgroup:
|
||||
|
||||
```sh
|
||||
obikmer filter src --output dst \
|
||||
--ingroup "genus=Betula" \
|
||||
--outgroup "*" \
|
||||
--min-count -1 \
|
||||
--max-outgroup-count -1
|
||||
```
|
||||
|
||||
To dump only k-mers specific to *Betula nana*:
|
||||
|
||||
```sh
|
||||
obikmer dump myindex \
|
||||
--ingroup "species=Betula_nana" \
|
||||
--outgroup "*" \
|
||||
--min-count 1 \
|
||||
--max-outgroup-count 0
|
||||
```
|
||||
|
||||
To enumerate unitigs of the *Betula*-specific subgraph:
|
||||
|
||||
```sh
|
||||
obikmer unitig myindex \
|
||||
--ingroup "genus=Betula" \
|
||||
--outgroup "*" \
|
||||
--min-count 2 \
|
||||
--max-outgroup-count 0
|
||||
```
|
||||
|
||||
## Command-specific options
|
||||
|
||||
### `dump --head N`
|
||||
|
||||
Stops output after the first N k-mers that pass all active filters.
|
||||
Iteration terminates immediately — subsequent partitions and layers are not scanned.
|
||||
Useful for quick inspection of large indexes without loading the entire dataset.
|
||||
|
||||
```sh
|
||||
obikmer dump myindex --head 100
|
||||
obikmer dump myindex --head 20 --ingroup "species=Betula_nana" --min-count 1
|
||||
```
|
||||
|
||||
### `phylo --presence-threshold N`
|
||||
|
||||
When computing Jaccard distance on a **count index**, a k-mer is considered present in a genome if its count is ≥ N (default 1).
|
||||
This option is independent of the `--presence-threshold` used in filtering.
|
||||
|
||||
```sh
|
||||
# Jaccard treating kmers with count ≥ 2 as present
|
||||
obikmer phylo myindex --metric jaccard --presence-threshold 2
|
||||
```
|
||||
|
||||
This parameter has no effect on presence/absence indexes (where values are already 0/1) or on metrics other than Jaccard.
|
||||
|
||||
## Implementation
|
||||
|
||||
- **`obikpartitionner::filter::GroupQuorumFilter`** — implements `KmerFilter`
|
||||
using pre-computed ingroup and outgroup index vectors. The heavy logic
|
||||
(predicate parsing, three-value evaluation, genome classification) happens
|
||||
once before any iteration; each k-mer row evaluation is a simple index
|
||||
lookup and counter.
|
||||
|
||||
- **`obikmer::cmd::predicate::FilterArgs`** — shared `clap` argument group
|
||||
embedded via `#[command(flatten)]` in `FilterArgs`, `DumpArgs`, and
|
||||
`UnitigArgs`. `FilterArgs::build_filters()` returns a ready-to-use filter
|
||||
list.
|
||||
|
||||
- **`obikpartitionner::KmerPartition::iter_partition_kmers`** — accepts
|
||||
`filters: &[Box<dyn KmerFilter>]` and applies them per-kmer before invoking
|
||||
the callback. `filter`, `dump`, and `unitig` all go through this single
|
||||
entry point.
|
||||
@@ -0,0 +1,91 @@
|
||||
# Kmer — implementation
|
||||
|
||||
## Types and layout
|
||||
|
||||
`KmerOf<L>` is a `#[repr(transparent)]` newtype over `u64` parameterized by a `KmerLength` marker:
|
||||
|
||||
```rust
|
||||
#[repr(transparent)]
|
||||
pub struct KmerOf<L: KmerLength>(u64, PhantomData<L>);
|
||||
```
|
||||
|
||||
Three marker types implement `KmerLength`:
|
||||
|
||||
| Marker | `len()` source | Used for |
|
||||
|--------|---------------|---------|
|
||||
| `KLen` | `params::k()` | k-mers |
|
||||
| `MLen` | `params::m()` | minimizers |
|
||||
| `ConstLen<N>` | const generic `N` | tests |
|
||||
|
||||
Public aliases:
|
||||
|
||||
```rust
|
||||
pub type Kmer = KmerOf<KLen>; // k-mer, global k
|
||||
pub type Minimizer = CanonicalKmerOf<MLen>; // canonical m-mer, global m
|
||||
```
|
||||
|
||||
Nucleotides are packed 2 bits each, **left-aligned**, MSB-first. Nucleotide 0 occupies bits 63–62; nucleotide i occupies bits 63−2i and 62−2i. The low 64−2·len bits are always zero. The length is **not stored** — every operation reads it from `L::len()`.
|
||||
|
||||
| 63–62 | 61–60 | … | 63−2(k−1)−1 to 63−2(k−1) | 63−2k down to 0 |
|
||||
|-------|-------|---|--------------------------|-----------------|
|
||||
| nt 0 | nt 1 | … | nt k−1 | zero padding |
|
||||
|
||||
## Global parameters
|
||||
|
||||
`params::set_k(k)` / `params::k()` and `params::set_m(m)` / `params::m()` are backed by `OnceLock<usize>` in production (write-once, panic on conflict) and by `thread_local! { Cell<usize> }` in test builds (per-thread, freely writable). `params::init(k, m)` sets both in one call.
|
||||
|
||||
## Encoding
|
||||
|
||||
`KmerOf::<L>::from_ascii(ascii)` encodes the first `L::len()` bytes using the shared `ENC` table (see [SuperKmer — ASCII encoding](superkmer.md#ascii-encoding-and-decoding)):
|
||||
|
||||
```rust
|
||||
for i in 0..k {
|
||||
val = (val << 2) | encode_base(ascii[i]) as u64;
|
||||
}
|
||||
KmerOf(val << (64 - 2 * k), PhantomData)
|
||||
```
|
||||
|
||||
Zero allocation — result lives on the stack.
|
||||
|
||||
## Decoding
|
||||
|
||||
`write_ascii(writer)` writes k ASCII characters to any `W: Write` using the shared `DEC4` table: one lookup per 4 nucleotides, one partial lookup for the remainder. No allocation in the hot path.
|
||||
|
||||
`to_ascii()` is a convenience wrapper that allocates and returns a `Vec<u8>`; intended for tests and display only.
|
||||
|
||||
## Reverse complement
|
||||
|
||||
Computed as pure arithmetic — no lookup table, no memory access:
|
||||
|
||||
```rust
|
||||
let x = !self.0; // complement
|
||||
let x = x.swap_bytes(); // reverse bytes
|
||||
let x = ((x >> 4) & 0x0F0F0F0F0F0F0F0F) | ((x & 0x0F0F0F0F0F0F0F0F) << 4); // swap nibbles
|
||||
let x = ((x >> 2) & 0x3333333333333333) | ((x & 0x3333333333333333) << 2); // swap 2-bit groups
|
||||
KmerOf(x << (64 - 2 * k), PhantomData)
|
||||
```
|
||||
|
||||
After complementing, bytes are reversed (`swap_bytes`), then nibbles, then 2-bit groups — restoring 2-bit nucleotides to their correct positions in reverse order. A final left-shift realigns to MSB. Zero allocation — result lives on the stack.
|
||||
|
||||
## Canonical form and `CanonicalKmerOf`
|
||||
|
||||
`canonical()` returns a `CanonicalKmerOf<L>` — a distinct newtype that carries the same `u64` layout but enforces the invariant that the stored value equals `min(kmer, revcomp)`:
|
||||
|
||||
```rust
|
||||
pub fn canonical(&self) -> CanonicalKmerOf<L> {
|
||||
let rc = self.revcomp();
|
||||
CanonicalKmerOf(if self.0 <= rc.0 { self.0 } else { rc.0 }, PhantomData)
|
||||
}
|
||||
```
|
||||
|
||||
Lexicographic minimum of forward and reverse-complement, comparing the raw `u64` values directly (left-aligned encoding makes this equivalent to nucleotide-wise comparison). Zero allocation — result lives on the stack.
|
||||
|
||||
`CanonicalKmerOf::from_raw_unchecked(raw)` is the only other public constructor, for trusted paths such as deserialisation.
|
||||
|
||||
## Sliding window helpers
|
||||
|
||||
`push_right(nuc)` / `push_left(nuc)` shift the window by one base in O(1). `is_overlapping(other)` checks whether the last k−1 nucleotides of `self` equal the first k−1 of `other`.
|
||||
|
||||
## Hashing
|
||||
|
||||
`hash_kmer(raw: u64) -> u64` computes `mix64(raw ^ 0x9e3779b97f4a7c15)`, the seeded splitmix64 finalizer. `CanonicalKmerOf::seq_hash()` delegates to `hash_kmer`.
|
||||
@@ -0,0 +1,13 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/kmer.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikseq/src/kmer.rs` — layout mémoire (repr(transparent) u64), encodage/décodage, revcomp, forme canonique
|
||||
- `obikseq/src/params.rs` — k global (set_k / k())
|
||||
|
||||
## Notes
|
||||
|
||||
Document d'implémentation stable. L'algorithme de revcomp bit-à-bit est décrit —
|
||||
vérifier qu'il correspond à `revcomp_raw` dans `obiskio/src/unitig_index.rs` (copie locale)
|
||||
et à l'implémentation dans `obikseq/src/kmer.rs`.
|
||||
@@ -0,0 +1,196 @@
|
||||
# Merge command
|
||||
|
||||
## Purpose
|
||||
|
||||
`obikmer merge` combines multiple existing kmer indexes into a single index. The result contains all kmers from all sources, with per-genome presence/absence or count data for every genome across every layer.
|
||||
|
||||
---
|
||||
|
||||
## Modes
|
||||
|
||||
```rust
|
||||
pub enum MergeMode { Presence, Count }
|
||||
```
|
||||
|
||||
Default mode is `Presence`. `Count` mode requires **all** source indexes to have `with_counts=true`; mixing count and non-count sources is rejected at validation.
|
||||
|
||||
| Mode | Column type | Constraint |
|
||||
|---|---|---|
|
||||
| `Presence` | `PersistentBitMatrix` (one bit per genome per slot) | none |
|
||||
| `Count` | `PersistentCompactIntMatrix` (one u32 per genome per slot) | all sources `with_counts=true` |
|
||||
|
||||
---
|
||||
|
||||
## Input / output constraints
|
||||
|
||||
All source indexes must satisfy:
|
||||
|
||||
- `IndexState::Indexed` (fully built — `index.done` sentinel present)
|
||||
- Same `kmer_size`, `minimizer_size`, `n_partitions`
|
||||
- Same evidence kind: all `Exact`, or all `Approx` with identical `(b, z)` parameters
|
||||
- If `Count` mode: all sources must have `with_counts=true`
|
||||
|
||||
`--force`: if the output directory already exists, it is deleted before the merge begins.
|
||||
|
||||
---
|
||||
|
||||
## Evidence compatibility
|
||||
|
||||
`validate_evidence_compat(sources)` is called before any I/O. It compares each source's `EvidenceKind` against `sources[0]`:
|
||||
|
||||
- All `Exact` → accepted, output uses `Exact`
|
||||
- All `Approx { b, z }` with same `(b, z)` → accepted, output uses those parameters
|
||||
- Any other combination → `OKIError::IncompatibleEvidence`, with a message directing the user to run `reindex` first
|
||||
|
||||
Mixed exact/approx is a hard error, not a silent conversion.
|
||||
|
||||
```rust
|
||||
fn validate_evidence_compat(sources: &[&KmerIndex]) -> OKIResult<EvidenceKind>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Genome label deduplication
|
||||
|
||||
`compute_labels(sources, rename_duplicates)` assigns final genome labels across all sources before any file is written. The first occurrence of a label keeps the original name. Subsequent occurrences receive `.1`, `.2`, … suffixes when `rename_duplicates` is true, or trigger `OKIError::DuplicateGenomeLabel` otherwise.
|
||||
|
||||
---
|
||||
|
||||
## Algorithm
|
||||
|
||||
### 1. Validation
|
||||
|
||||
Check all sources against the constraints above. Abort on any mismatch.
|
||||
|
||||
### 2. Bootstrap output from first source
|
||||
|
||||
Recursive file copy of `sources[0]` → `output`. Immediately after the copy:
|
||||
|
||||
- `index.meta` is rewritten with the final genome list (all sources, possibly renamed) and the effective evidence kind.
|
||||
- In `Presence` mode, any `counts/` directories inherited from source_0 are removed.
|
||||
- `spectrums/` from source_0 is removed and rebuilt from scratch across all sources, applying the (possibly renamed) labels.
|
||||
|
||||
This establishes the partition layout, all existing MPHFs, unitigs, and evidence files. The first source's genomes occupy columns 0 … `n_dst_genomes - 1` in the destination.
|
||||
|
||||
### 3. For each subsequent source (parallel across partitions)
|
||||
|
||||
`KmerPartition::merge_partition(i, sources, mode, n_dst_genomes, block_bits)` is called for each partition index `i`. `block_bits` is taken from `dst.meta.config.block_bits`.
|
||||
|
||||
Each entry in `sources` is `(&KmerPartition, n_genomes)` where `n_genomes` is the column count that source contributes (> 1 when the source is itself a merged index).
|
||||
|
||||
**First merge, Presence mode**: when `n_dst_genomes == 1`, `Layer::<()>::init_presence_matrix` is called on every existing destination layer before any source column is appended. This creates `presence/col_000000.pbiv` set all-true (genome 0 is present in every slot).
|
||||
|
||||
**Pass 1 — classify kmers**
|
||||
|
||||
Iterate all kmers from all source partitions (via `UnitigFileReader` + canonical kmer iteration). For each kmer, probe the destination `LayeredMap<()>`:
|
||||
|
||||
- **Hit**: kmer already in the destination; record for Pass 2.
|
||||
- **Miss**: push kmer into a `GraphDeBruijn` accumulator.
|
||||
|
||||
**New layer construction**
|
||||
|
||||
If the accumulator is non-empty, compute de Bruijn unitigs and call `Layer::<()>::build(&new_layer_dir, block_bits)`. All kmers absent from the destination — across **all** sources — accumulate into a **single** graph, producing one new layer per merge operation (not one per source).
|
||||
|
||||
**Pass 2 — fill column builders**
|
||||
|
||||
For each source and each of its layers, re-iterate unitigs and look up stored values via `SrcLayerData::lookup(kmer, src_n)`:
|
||||
|
||||
- `SrcLayerData::SetMembership` — no data directory exists; every kmer returns `vec![1; n_genomes]`
|
||||
- `SrcLayerData::Presence` — reads `PersistentBitMatrix` from `presence/`
|
||||
- `SrcLayerData::Count` — reads `PersistentCompactIntMatrix` from `counts/`
|
||||
|
||||
Hits are routed to `exist_builders[dst_layer][src_col]`; misses are routed to `new_src_builders[src_col]`.
|
||||
|
||||
**Column prepending for new layers**
|
||||
|
||||
Before source columns are written to the new layer, `n_dst_genomes` absent columns (all-zero / all-false) are prepended — one per genome already in the index — so the column count invariant holds immediately after layer creation.
|
||||
|
||||
**Close and update metadata**
|
||||
|
||||
Close all builders; update `presence/meta.json` or `counts/meta.json` with `{"n": N, "n_cols": n_dst_genomes + n_src_total}`; increment `PartitionMeta::n_layers` if a new layer was added.
|
||||
|
||||
### 4. Update index metadata
|
||||
|
||||
`index.meta` was already written during bootstrap with the complete genome list and evidence kind. No further update is needed after the partition loop.
|
||||
|
||||
---
|
||||
|
||||
## `append_genome_column`
|
||||
|
||||
Defined on two concrete specialisations of `Layer<D>`:
|
||||
|
||||
```rust
|
||||
impl Layer<PersistentCompactIntMatrix> {
|
||||
pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> u32) -> OLMResult<()>
|
||||
}
|
||||
|
||||
impl Layer<PersistentBitMatrix> {
|
||||
pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> bool) -> OLMResult<()>
|
||||
}
|
||||
```
|
||||
|
||||
Each appends one column file to the matrix subdirectory (`counts/` or `presence/`). In `merge_partition`, columns are written directly via `PersistentBitVecBuilder` / `PersistentCompactIntVecBuilder` rather than through these helpers, but the invariant they enforce is the same.
|
||||
|
||||
---
|
||||
|
||||
## Column count invariant
|
||||
|
||||
After any merge, **every layer in every partition has exactly `n_genomes` columns**, where `n_genomes` is the total genome count in the index at that point.
|
||||
|
||||
Maintained by three mechanisms:
|
||||
|
||||
1. **Existing layers**: `n_src_total` columns appended (one per source genome).
|
||||
2. **New layers created during merge**: `n_dst_genomes` absent columns prepended before source columns.
|
||||
3. **First merge, Presence mode**: `init_presence_matrix` retroactively creates `presence/col_0` all-true for genome 0.
|
||||
|
||||
The invariant is a precondition of `LayeredStore` aggregation traits: `col_weights()` and all partial distance methods assume every inner store has the same column count.
|
||||
|
||||
---
|
||||
|
||||
## Error variants relevant to merge
|
||||
|
||||
| Variant | Condition |
|
||||
|---|---|
|
||||
| `OKIError::NotIndexed(path)` | Source not in `Indexed` state |
|
||||
| `OKIError::IncompatibleConfig` | Mismatched `kmer_size`, `minimizer_size`, or `n_partitions` |
|
||||
| `OKIError::MismatchedMode` | Count mode but a source has `with_counts=false` |
|
||||
| `OKIError::IncompatibleEvidence(msg)` | Mixed exact/approx or different approx `(b, z)` |
|
||||
| `OKIError::DuplicateGenomeLabel(label)` | Duplicate label and `rename_duplicates=false` |
|
||||
|
||||
---
|
||||
|
||||
## On-disk impact
|
||||
|
||||
After merging `G` genomes (sources_0 contributes `G0`, subsequent sources the rest):
|
||||
|
||||
```
|
||||
partitions/
|
||||
part_00000/
|
||||
index/
|
||||
meta.json ← n_layers updated if new layer added
|
||||
layer_0/
|
||||
mphf.bin ← unchanged
|
||||
unitigs.bin ← unchanged
|
||||
evidence.bin ← unchanged
|
||||
presence/ ← created on first merge (Presence mode)
|
||||
meta.json {"n": N, "n_cols": G}
|
||||
col_000000.pbiv ← all-true (genome 0 … G0-1)
|
||||
col_000001.pbiv ← next source
|
||||
...
|
||||
counts/ ← extended (Count mode)
|
||||
meta.json {"n": N, "n_cols": G}
|
||||
col_000000.pciv ← genome 0 counts (from original build)
|
||||
col_000001.pciv ← next source
|
||||
...
|
||||
layer_N/ ← new layer (if new kmers found)
|
||||
mphf.bin
|
||||
unitigs.bin
|
||||
evidence.bin
|
||||
presence/ or counts/
|
||||
meta.json {"n": N1, "n_cols": G}
|
||||
col_000000.pbiv ← all-false (absent for existing genomes)
|
||||
...
|
||||
spectrums/
|
||||
<label>.json ← one file per genome, rebuilt from all sources
|
||||
index.meta ← complete genome list + evidence kind written at bootstrap
|
||||
```
|
||||
@@ -0,0 +1,19 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/merge.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikindex/src/merge.rs` — `KmerIndex::merge()`, validation de compatibilité d'évidence, `validate_evidence_compat()`
|
||||
- `obikpartitionner/src/merge_layer.rs` — `merge_partition()`, construction de la nouvelle layer, paramètre `block_bits`
|
||||
- `obikpartitionner/src/rebuild_layer.rs` — `rebuild_partition()`, paramètre `block_bits`
|
||||
- `obilayeredmap/src/layer.rs` — `Layer::append_genome_column()` (PersistentCompactIntMatrix et PersistentBitMatrix)
|
||||
- `obicompactvec/src/intmatrix.rs` — `append_column` pour PersistentCompactIntMatrix
|
||||
- `obicompactvec/src/bitmatrix.rs` — `append_column` pour PersistentBitMatrix
|
||||
|
||||
## Notes
|
||||
|
||||
FORT RISQUE DE DÉRIVE. Changements récents :
|
||||
- Ajout de la validation de compatibilité d'évidence : merge exact+approx → erreur (OKIError::IncompatibleEvidence)
|
||||
- `merge_partition` reçoit maintenant `block_bits: u8`
|
||||
- La commande `reindex` a été ajoutée comme outil de conversion exact↔approx avant merge
|
||||
Vérifier que la doc décrit la politique de merge mixed-evidence et le recours à `reindex`.
|
||||
@@ -0,0 +1,207 @@
|
||||
# Merge parallelism and memory pressure
|
||||
|
||||
## Problem observed
|
||||
|
||||
Running `obikmer merge` over 109 indexes (108 sources + 1 bootstrap) on a 192-core machine
|
||||
produces a fatal OOM during the `merge_partitions` stage:
|
||||
|
||||
```
|
||||
memory allocation of 9126805520 bytes failed
|
||||
```
|
||||
|
||||
A single allocation of ~8.5 GB fails. This is not an aggregate; it is one `malloc` call
|
||||
from hashbrown during a HashMap resize.
|
||||
|
||||
---
|
||||
|
||||
## Root cause
|
||||
|
||||
### The merge pipeline per partition
|
||||
|
||||
```
|
||||
source unitigs.bin
|
||||
→ iter_indexed_canonical_kmers()
|
||||
→ GraphDeBruijn::push() ← HashSet<u64> + 1 byte flags, all in RAM
|
||||
→ compute_degrees_and_mark_starts()
|
||||
→ try_for_each_unitig()
|
||||
→ unitigs.bin (new layer)
|
||||
→ Layer::build() → MPHF + evidence
|
||||
```
|
||||
|
||||
`GraphDeBruijn` is a `FastHashMap<CanonicalKmer, AtomicU8>` — a `HashSet<u64>` with
|
||||
one flag byte per node. Neighbor lookup is implicit: 4 probes into the same map.
|
||||
No edges are stored. The full kmer set of one partition must reside in RAM
|
||||
simultaneously to compute degrees and mark unitig starts.
|
||||
|
||||
The matrix builders that follow (pass 2) are mmapped files — they do **not** consume
|
||||
significant RAM. The pressure is entirely in pass 1.
|
||||
|
||||
### Unbounded Rayon parallelism
|
||||
|
||||
With 192 cores, Rayon ran up to 192 partitions concurrently. Each partition built its
|
||||
own `GraphDeBruijn` accumulating all kmers absent from the destination. Peak memory =
|
||||
192 × peak_partition_hashset.
|
||||
|
||||
### The 8.5 GB single allocation
|
||||
|
||||
hashbrown allocates the entire backing array in one call when rehashing.
|
||||
At load factor 7/8: `capacity × (sizeof(K,V) + 1 control byte)`.
|
||||
For `(u64, AtomicU8)` with alignment: ~16 bytes per slot.
|
||||
|
||||
```
|
||||
9 127 MB / 16 bytes ≈ 570 M slots → ~380 M new kmers in one partition
|
||||
```
|
||||
|
||||
Plausible for the largest partition of 108 Salix/Betula sources (~450 Mbp each).
|
||||
|
||||
---
|
||||
|
||||
## Partition size distribution
|
||||
|
||||
`obikmer utils --partition-stats` measures the sum of `unitigs.bin` file sizes
|
||||
per partition across all source indexes (pure `stat()` syscalls, negligible cost).
|
||||
|
||||
Observed on a 9-genome pilot (256 partitions):
|
||||
|
||||
| Stat | Value |
|
||||
|---|---|
|
||||
| min | 30.5 MB |
|
||||
| max | 232.1 MB |
|
||||
| mean | 40.1 MB |
|
||||
| median | 37.2 MB |
|
||||
| p95 | 47.1 MB |
|
||||
| max/median ratio | 6.23× |
|
||||
|
||||
The distribution is **bimodal with a heavy tail**:
|
||||
- 238/256 partitions in a narrow 30–50 MB band
|
||||
- 4 structurally extreme partitions (3–6× the median): 221, 233, 135, 191
|
||||
|
||||
These correspond to minimizers over-represented in repetitive regions shared across
|
||||
all sources. They are extreme in every run on this dataset.
|
||||
|
||||
With 109 sources, outlier partitions do not scale linearly: only kmers **absent from
|
||||
the destination** enter the GraphDeBruijn, and inter-source overlap is high for closely
|
||||
related species. Partition 221 is the likely trigger for the 8.5 GB crash.
|
||||
|
||||
---
|
||||
|
||||
## Solution: LFD scheduling + memory budget semaphore
|
||||
|
||||
### Principle
|
||||
|
||||
Pre-sort partitions by **decreasing estimated size** (First Fit Decreasing — FFD),
|
||||
then schedule them through a **continuous memory budget semaphore**. Each worker
|
||||
acquires an estimated cost before starting and releases it on completion.
|
||||
|
||||
Large partitions run first when the full budget is available; small partitions fill
|
||||
the gaps. No hard outlier threshold is needed.
|
||||
|
||||
### `MemoryBudget` (`obisys`)
|
||||
|
||||
```rust
|
||||
pub struct MemoryBudget { … }
|
||||
|
||||
impl MemoryBudget {
|
||||
pub fn new(total: u64) -> Self;
|
||||
pub fn acquire(&self, cost: u64); // blocks until budget available
|
||||
pub fn release(&self, cost: u64);
|
||||
pub fn peak_active(&self) -> usize;
|
||||
}
|
||||
```
|
||||
|
||||
Non-deadlock guarantee: when `active == 0`, acquire always succeeds regardless of cost.
|
||||
Without this, a partition whose estimated cost exceeds the total budget would block forever.
|
||||
|
||||
### Adaptive expansion factor
|
||||
|
||||
The expansion factor converts raw `unitigs.bin` bytes into an estimated GraphDeBruijn
|
||||
RAM footprint. hashbrown stores each kmer as `(u64, AtomicU8)` ≈ 16 bytes/kmer at 7/8
|
||||
load factor; unitig files encode ≈ 2 bits/base. The ratio depends on average unitig
|
||||
length (short unitigs: ~2×; long unitigs: up to ~50×).
|
||||
|
||||
**Phase 1 — sequential pilot (worst partition)**
|
||||
|
||||
The largest partition runs alone first. Its actual `g.len()` seeds the expansion factor
|
||||
before any parallel job starts. `FALLBACK_EXPANSION = 4×` is used only for empty partitions.
|
||||
|
||||
```rust
|
||||
let worst_g_len = dst_partition.merge_partition(worst_id, …)?;
|
||||
// ↑ now returns SKResult<usize> (was SKResult<()>)
|
||||
|
||||
let seed_expansion = worst_g_len as u64 * 16 * 1000 / worst_bytes;
|
||||
let max_expansion = AtomicU64::new(seed_expansion);
|
||||
```
|
||||
|
||||
**Phase 2 — parallel with adaptive updates**
|
||||
|
||||
```rust
|
||||
order[1..].into_par_iter().for_each(|&i| {
|
||||
let cost = partition_sizes[i] * max_expansion.load(Relaxed) / 1000;
|
||||
budget.acquire(cost);
|
||||
let g_len = dst_partition.merge_partition(i, …)?;
|
||||
budget.release(cost); // releases estimated cost, not actual
|
||||
|
||||
let actual = g_len as u64 * 16 * 1000 / partition_sizes[i];
|
||||
max_expansion.fetch_max(actual, Relaxed); // always pessimistic (max)
|
||||
});
|
||||
```
|
||||
|
||||
`budget.release(cost)` uses the estimated cost, not the actual one. The budget tracks
|
||||
reservations, not physical RAM; each partition pays what it promised at acquisition.
|
||||
|
||||
**On the safety margin**
|
||||
|
||||
There is no separate multiplier `k`. It is redundant with `budget_fraction`: both
|
||||
reduce effective concurrency by the same amount. A single parameter is easier to
|
||||
calibrate. `budget_fraction = 0.5` (default) reserves half of available RAM for the
|
||||
OS, MPHF build, pass 2, and estimation error.
|
||||
|
||||
`--budget-fraction` is exposed as a CLI flag — the only escape hatch for pathological
|
||||
cases (extreme repetitive content, unusually long unitigs) that still cause OOM.
|
||||
|
||||
### RAM source
|
||||
|
||||
`obisys::available_memory_bytes()` — wraps `sysinfo::System::available_memory()`,
|
||||
falls back to `total / 2` on macOS when the memory compressor returns 0.
|
||||
|
||||
---
|
||||
|
||||
## Diagnostic report
|
||||
|
||||
After the parallel phase, `merge_partition` emits a structured report via `tracing::info!`:
|
||||
|
||||
```
|
||||
─── merge_partitions memory report ───
|
||||
available RAM : 512.0 GB budget 50% = 256.0 GB
|
||||
expansion factor — seed: 4.2× final max: 6.1× (mean: 1.8× median: 1.6×)
|
||||
peak concurrent workers: 42
|
||||
expansion factor distribution (256 partitions with data):
|
||||
0.50× – 1.25× │██████████████████████████████ 148
|
||||
1.25× – 2.00× │████████████████████████ 82
|
||||
…
|
||||
5.50× – 6.25× │█ 2
|
||||
top partitions by actual expansion factor:
|
||||
partition 221 : 6.10× (232.1 MB unitigs → 48M kmers, reserved at 4.20×)
|
||||
partition 135 : 5.82× (127.3 MB unitigs → 24M kmers, reserved at 4.20×)
|
||||
…
|
||||
──────────────────────────────────────
|
||||
```
|
||||
|
||||
Fields useful for diagnosis:
|
||||
|
||||
| Field | Interpretation |
|
||||
|---|---|
|
||||
| `seed` vs `final max` expansion | gap indicates partitions with higher expansion than the worst-by-size |
|
||||
| `reserved at X×` | the factor used at acquisition; if much lower than actual, the budget was under-reserved for that partition |
|
||||
| `peak concurrent workers` | effective parallelism achieved under the budget constraint |
|
||||
| `mean` / `median` expansion | typical dataset characteristic; stable across runs on the same data |
|
||||
|
||||
---
|
||||
|
||||
## Parameters
|
||||
|
||||
| Parameter | Default | CLI flag | Notes |
|
||||
|---|---|---|---|
|
||||
| `fallback_expansion` | 4× | — | seed for empty partitions only |
|
||||
| `budget_fraction` | 0.5 | `--budget-fraction` | reduce if OOM persists |
|
||||
| RAM source | `obisys::available_memory_bytes()` | — | falls back to `total/2` on macOS |
|
||||
@@ -0,0 +1,177 @@
|
||||
# MPHF selection — two-phase indexing architecture
|
||||
|
||||
## Why two phases are needed
|
||||
|
||||
Kmer indexing per partition proceeds in two phases. The separation is necessary because the exact number of surviving unique kmers is not known until after counting and filtering low-abundance kmers.
|
||||
|
||||
### Phase 1 — provisional MPHF + kmer spectrum
|
||||
|
||||
Implemented in `obikpartitionner::KmerPartition::count_kmer()` → `count_partition()`.
|
||||
|
||||
1. **External sort**: read the dereplicated superkmer file; extract the raw `u64` canonical kmer value for every kmer of every superkmer. Sort in RAM-bounded chunks (adaptive budget: 40% of available RAM ÷ n_threads, minimum 1 M kmers per chunk), then k-way merge with inline dedup. Result: `sorted_unique.bin` — a flat array of f0 distinct sorted `u64` values. Exact kmer count f0 is known at this point.
|
||||
2. **Build provisional MPHF** (ptr_hash, same configuration as phase 2) over `sorted_unique.bin` using `new_from_par_iter`. Delete `sorted_unique.bin` immediately after. Persist to `mphf1.bin`.
|
||||
3. **Create `counts1.bin`**: `PersistentCompactIntVec` with f0 slots, zero-initialised.
|
||||
4. **Accumulation pass**: re-read the dereplicated superkmer file; for each kmer in each superkmer, compute `slot = mphf.index(kmer.raw())` and increment `counts1[slot]` by the superkmer's COUNT.
|
||||
5. **Build kmer frequency spectrum** from `counts1`: histogram `{count → n_kmers}`, totals f0 (distinct kmers) and f1 (total abundance). Written to `kmer_spectrum_raw.json` per partition, then merged globally.
|
||||
|
||||
Files produced per partition:
|
||||
|
||||
```
|
||||
part_XXXXX/
|
||||
mphf1.bin — ptr_hash provisional MPHF (discarded after phase 2)
|
||||
counts1.bin — PersistentCompactIntVec, f0 × u32 kmer counts
|
||||
kmer_spectrum_raw.json — local frequency spectrum
|
||||
```
|
||||
|
||||
### Phase 2 — definitive MPHF
|
||||
|
||||
After filtering (applying a min-count threshold derived from the spectrum) and building the local De Bruijn graph + unitigs (see [Construction pipeline](pipeline.md)), the exact filtered kmer set is available via `unitigs.bin`.
|
||||
|
||||
`MphfLayer::build(dir, block_bits, mode: &IndexMode, fill_slot)` is called on the unitig directory:
|
||||
|
||||
1. **Pass 1** (parallel): a `CanonicalKmerIter` — clonable via `Arc<Mmap>`, no file reopening — is passed directly to `new_from_par_iter` via `par_bridge()`. No `.idx` is read or created at this stage; parallelism is at partition/layer level, not within a single MPHF. Produces `mphf.bin`.
|
||||
2. **Pass 2** (sequential): iterate with `iter_indexed_canonical_kmers`; fill evidence files; call `fill_slot(slot, kmer)` callback per kmer. For Exact/Hybrid, `.idx` is written at the end of this pass — never earlier.
|
||||
|
||||
`mphf1.bin` and `counts1.bin` are no longer needed after phase 2 and can be deleted.
|
||||
|
||||
---
|
||||
|
||||
## MPHF candidates
|
||||
|
||||
**boomphf** (BBHash algorithm, maintained by 10X Genomics):
|
||||
|
||||
- ~3.7 bits/key; mature crate, used in production bioinformatics (Pufferfish, Piscem)
|
||||
- Supports streaming construction (no exact count needed)
|
||||
- Drawback: largest space footprint; streaming advantage is irrelevant at phase 2 since the exact count is available
|
||||
|
||||
**ptr_hash** (PtrHash algorithm, Groot Koerkamp, SEA 2025):
|
||||
|
||||
- ~2.4 bits/key; fastest queries (≥2.1× over alternatives, 8–12 ns/key for u64) and fastest construction (≥3.1×)
|
||||
- Requires exact key count at construction — available at both phases after pass 1
|
||||
- Published February 2025; accepted given performance profile and the fact that each MPHF is independently rebuildable from its unitig file
|
||||
|
||||
**FMPH/FMPHGO** (`ph` crate, Beling, ACM JEA 2023):
|
||||
|
||||
- ~2.1 bits/key — most compact; good query speed; deterministic construction
|
||||
- `GOFunction` (group-oriented variant) was the original phase-1 choice; eliminated when the external sort made the exact count available at phase 1 as well
|
||||
|
||||
## MPHF choice per phase
|
||||
|
||||
**Both phases**: **ptr_hash**, same type alias and construction parameters. The external sort (phase 1) and the unitig index (phase 2) both provide the exact key count before MPHF construction, so ptr_hash's requirement is satisfied in both cases. Using a single MPHF implementation removes the `ph` crate dependency.
|
||||
|
||||
boomphf: eliminated — largest space overhead, streaming advantage no longer needed. FMPH/GOFunction: eliminated — exact count available, ptr_hash is faster at equivalent compactness.
|
||||
|
||||
---
|
||||
|
||||
## Space at scale
|
||||
|
||||
For 1 024 partitions × 100 M kmers/partition (phase 2 index, after filtering):
|
||||
|
||||
| MPHF | bits/key | Total MPHF size |
|
||||
|----------|----------|-----------------|
|
||||
| boomphf | 3.7 | ~47 GB |
|
||||
| ptr_hash | 2.4 | ~31 GB |
|
||||
| FMPH | 2.1 | ~27 GB |
|
||||
|
||||
For a human genome at 30× coverage with 1 024 partitions, realistic partition sizes are 3–30 M unique kmers → 1–8 MB per phase-2 MPHF, well within RAM.
|
||||
|
||||
---
|
||||
|
||||
## ptr_hash configuration (phase 2)
|
||||
|
||||
```rust
|
||||
type Mphf = PtrHash<
|
||||
u64, // key: canonical kmer raw encoding
|
||||
CubicEps, // bucket fn: 2.4 bits/key, λ=3.5, α=0.99
|
||||
CachelineEfVec<Vec<CachelineEf>>, // remap: 11.6 bits/entry (Elias-Fano)
|
||||
Xx64, // hasher: XXH3-64 with seed
|
||||
Vec<u8>, // pilots
|
||||
>;
|
||||
```
|
||||
|
||||
**Hasher — `Xx64`**: canonical kmer raw values are left-aligned u64 with structural zeros in low bits (42 zeros for k=11, 2 zeros for k=31). `FxHash` (single multiply) distributes these poorly; `Xx64` (XXH3-64, seeded) handles structured input correctly.
|
||||
|
||||
**Bucket function — `CubicEps`**: λ=3.5, α=0.99. Balanced tradeoff: 2× slower construction than `Linear/λ=3.0`, 20% less space. `default_compact` (λ=4.0) saves a further 12.5% at 2× more construction time — not chosen.
|
||||
|
||||
**Remap — `CachelineEfVec`**: Elias-Fano variant packing 44 sorted 40-bit values per 64-byte cacheline (11.6 bits/value vs 32 for `Vec<u32>`). One cacheline per query; space win dominates at billion-scale key counts.
|
||||
|
||||
---
|
||||
|
||||
## Multilayer index architecture
|
||||
|
||||
### Layer structure
|
||||
|
||||
Each layer is a self-contained unit. See [obilayeredmap](obilayeredmap.md) for the full on-disk layout. The MPHF-relevant files are:
|
||||
|
||||
```
|
||||
layer_i/
|
||||
unitigs.bin — packed 2-bit nucleotide sequences (kmer evidence source)
|
||||
unitigs.bin.idx — random-access block index (block_bits controls granularity)
|
||||
mphf.bin — ptr_hash phase-2 MPHF
|
||||
evidence.bin — n × (chunk_id: 25 bits | rank: 7 bits) per slot [exact mode]
|
||||
fingerprint.bin — n × b-bit fingerprints per slot [approx mode]
|
||||
[no layer_meta.json — mode stored once in partition-level meta.json]
|
||||
```
|
||||
|
||||
Layers are **disjoint**: a canonical kmer belongs to exactly one layer. Layer 0 is built from dataset A. Adding dataset B:
|
||||
|
||||
1. For each kmer in B: probe existing layers. If found, the kmer is already indexed.
|
||||
2. Collect kmers of B not present in any layer → set `B \ A`.
|
||||
3. Build layer 1 from `B \ A` (dereplicate → count → De Bruijn → unitigs → `MphfLayer::build`).
|
||||
|
||||
### Evidence modes
|
||||
|
||||
Three evidence modes are supported via `IndexMode`, stored once in `PartitionMeta` at partition root. There is no `layer_meta.json`.
|
||||
|
||||
**Exact** (`IndexMode::Exact`): `evidence.bin` stores one `(chunk_id, rank)` pair per MPHF slot. Verification reconstructs the kmer and compares to the query. Zero false positives. `.idx` required at query time.
|
||||
|
||||
**Approx** (`IndexMode::Approx { b, z }`): `fingerprint.bin` stores a b-bit hash per slot. False-positive rate 1/2^b per query; Findere z-parameter reduces window FP to ≈ 1/2^(b·z). No `.idx` written or needed.
|
||||
|
||||
**Hybrid** (`IndexMode::Hybrid { b, z }`): both `fingerprint.bin` and `evidence.bin` + `.idx`. `find()` uses the fingerprint (O(1)); `find_strict()` uses exact evidence (O(1)).
|
||||
|
||||
### Build functions
|
||||
|
||||
```
|
||||
MphfLayer::build(dir, block_bits, mode: &IndexMode, fill_slot)
|
||||
Pass 1: CanonicalKmerIter + par_bridge() → build mphf.bin (no .idx used)
|
||||
Pass 2: sequential iter → fill evidence files + call fill_slot
|
||||
.idx written last for Exact/Hybrid (query-time only)
|
||||
|
||||
MphfLayer::build_exact_evidence(dir, block_bits)
|
||||
Post-hoc: builds evidence.bin + .idx from existing mphf.bin + unitigs.bin
|
||||
Uses open_sequential(); no .idx required on entry
|
||||
|
||||
MphfLayer::build_approx_evidence(dir, b, z)
|
||||
Post-hoc: builds fingerprint.bin from existing mphf.bin + unitigs.bin
|
||||
Uses open_sequential(); never writes .idx
|
||||
```
|
||||
|
||||
There is no `build_evidence` dispatch wrapper. Callers choose the appropriate post-hoc build directly.
|
||||
|
||||
In `obikpartitionner`, `build_index_layer` receives `block_bits: u8` from `IndexConfig::block_bits` and forwards it directly to `Layer::build` and `Layer::build_approx_evidence`.
|
||||
|
||||
### Membership verification
|
||||
|
||||
ptr_hash maps any input to a valid slot — it does not natively detect absent keys. Membership is verified using the evidence entry:
|
||||
|
||||
- **Exact**: decode `(chunk_id, rank)` from `evidence.bin`; reconstruct the kmer via `unitigs.verify_canonical_kmer`; compare to query.
|
||||
- **Approx**: compare `kmer.seq_hash()` to the b-bit fingerprint stored at the slot.
|
||||
|
||||
A mismatch in either mode means the kmer is absent from this layer; probe the next layer.
|
||||
|
||||
### Query algorithm
|
||||
|
||||
```
|
||||
fn query(kmer) → Option<(layer_index, slot)>:
|
||||
for (i, layer) in layers.iter().enumerate():
|
||||
slot = layer.mphf.index(kmer)
|
||||
if layer.evidence.matches(slot, kmer): // exact or approx dispatch
|
||||
return Some((i, slot))
|
||||
return None
|
||||
```
|
||||
|
||||
`MphfLayer::find` dispatches on `LayerEvidence` at O(1) — no panicking `find_exact`/`find_approx` methods. `find_strict` always performs an exact check: O(1) for Exact/Hybrid, O(n) sequential scan for Approx. Expected probe depth: 1 for kmers in layer 0. Each probe is a ptr_hash lookup (~10 ns) plus one evidence check.
|
||||
|
||||
### Merging layers
|
||||
|
||||
Two layer chains can be merged by re-indexing their union through the full pipeline. This is expensive (full rebuild) but produces an optimal single-layer index. Merge is a maintenance operation, not a query-path requirement.
|
||||
@@ -0,0 +1,16 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/mphf.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/mphf_layer.rs` — type Mphf (PtrHash + CubicEps + CachelineEfVec + Xx64), construction en 2 passes, `build()`, `build_exact_evidence()`, `build_approx_evidence()`, `build_evidence()`
|
||||
- `obikpartitionner/src/index_layer.rs` — `build_index_layer()` avec passage de `block_bits`
|
||||
|
||||
## Notes
|
||||
|
||||
FORT RISQUE DE DÉRIVE. Changements récents :
|
||||
- `build_exact_evidence(dir, block_bits)` — `block_bits` maintenant paramétrisé (défaut 0)
|
||||
- `build_approx_evidence(dir, b, z)` — nouvelle fonction pour l'évidence fingerprint
|
||||
- `build_evidence(dir, kind, block_bits)` — dispatch selon EvidenceKind
|
||||
- Construction en 2 phases : pass 1 (Rayon parallèle) + pass 2 (callback `fill_slot`)
|
||||
Vérifier que la doc décrit correctement les deux nouvelles routes d'évidence et le paramètre `block_bits`.
|
||||
@@ -0,0 +1,533 @@
|
||||
# obicompactvec — Complete Reference
|
||||
|
||||
## Module structure
|
||||
|
||||
```
|
||||
src/obicompactvec/src/
|
||||
lib.rs public re-exports
|
||||
views.rs BitSliceView<'a>, IntSliceView<'a> — zero-copy read views
|
||||
traits.rs ColumnWeights, CountPartials, BitPartials (matrix aggregation)
|
||||
bitvec.rs PersistentBitVec, PersistentBitVecBuilder, BitIter
|
||||
reader.rs PersistentCompactIntVec (read-only)
|
||||
builder.rs PersistentCompactIntVecBuilder (read-write)
|
||||
tempintvec.rs TempCompactIntVec, TempCompactIntVecBuilder (temp-file-backed)
|
||||
tempbitvec.rs TempBitVec, TempBitVecBuilder (temp-file-backed)
|
||||
bitmatrix.rs PersistentBitMatrix, PersistentBitMatrixBuilder
|
||||
intmatrix.rs PersistentCompactIntMatrix, PersistentCompactIntMatrixBuilder
|
||||
colgroup.rs ColGroup, MatrixGroupOps trait
|
||||
format.rs file format constants, encode/decode helpers
|
||||
layer_meta.rs LayerMeta (column metadata)
|
||||
meta.rs matrix metadata
|
||||
```
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
views --> bitvec
|
||||
views --> builder
|
||||
views --> tempbitvec
|
||||
views --> tempintvec
|
||||
views --> bitmatrix
|
||||
views --> intmatrix
|
||||
format --> reader
|
||||
format --> builder
|
||||
reader --> intmatrix
|
||||
reader --> tempintvec
|
||||
builder --> intmatrix
|
||||
builder --> tempintvec
|
||||
bitvec --> tempbitvec
|
||||
bitvec --> bitmatrix
|
||||
tempintvec --> intmatrix
|
||||
tempintvec --> bitmatrix
|
||||
tempbitvec --> intmatrix
|
||||
tempbitvec --> bitmatrix
|
||||
colgroup --> intmatrix
|
||||
colgroup --> bitmatrix
|
||||
layer_meta --> bitmatrix
|
||||
layer_meta --> intmatrix
|
||||
meta --> bitmatrix
|
||||
meta --> intmatrix
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Compact int encoding
|
||||
|
||||
All integer vectors use the same two-tier encoding regardless of storage backend.
|
||||
|
||||
**Primary array** — one `u8` per slot:
|
||||
|
||||
- Values **0–254** are stored directly. No overhead.
|
||||
- Value **255 is a sentinel**: the slot's actual value is ≥ 255 and lives in the overflow store.
|
||||
|
||||
**Overflow store** — maps slot index to a `u32` value ≥ 255:
|
||||
|
||||
- In `PersistentCompactIntVecBuilder`: a `HashMap<usize, u32>` in RAM.
|
||||
- In `PersistentCompactIntVec` (reader): a sorted `[(slot: u64, value: u32)]` array in the mmap, with a sparse L1-resident index for binary search.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
slot --> P["primary[slot]: u8"]
|
||||
P -->|"< 255"| V["value = byte (0–254)"]
|
||||
P -->|"= 255 sentinel"| OV["overflow store"]
|
||||
OV -->|"Builder"| HM["HashMap<usize, u32>\nin RAM"]
|
||||
OV -->|"PersistentCompactIntVec"| SA["sorted [(slot,value)] in mmap\n+ sparse L1 index"]
|
||||
```
|
||||
|
||||
**Key property — sentinel 255 = +∞ on `u8`:**
|
||||
|
||||
- `min(a, 255) = a` for all `a ≤ 254` → correct when only one side is overflow
|
||||
- `max(a, 255) = 255` → correct sentinel when either side is overflow
|
||||
- Only the **both-overflow** case requires reading actual values from the overflow store.
|
||||
|
||||
In practice, k (overflow count) ≪ n (total slots). Observed genomic data: ~0.07% of kmer slots are in overflow.
|
||||
|
||||
---
|
||||
|
||||
## View types
|
||||
|
||||
The previous trait hierarchy (`BitSlice`, `BitSliceMut`, `IntSlice`, `IntSliceMut`) has been replaced by two concrete zero-copy view structs with inherent methods. Views are **`Copy`** — passing them is free. All read operations live on these two types.
|
||||
|
||||
### `BitSliceView<'a>`
|
||||
|
||||
```rust
|
||||
#[derive(Clone, Copy)]
|
||||
pub struct BitSliceView<'a> { pub(crate) words: &'a [u64], pub(crate) n: usize }
|
||||
```
|
||||
|
||||
Bit `i` is at `words[i >> 6]` bit `i & 63` (LSB-first). Padding bits in the last word are zero.
|
||||
|
||||
| Method | Cost |
|
||||
|---|---|
|
||||
| `len()`, `is_empty()` | O(1) |
|
||||
| `get(slot)` | O(1) |
|
||||
| `count_ones()` | POPCNT per word, O(n/64) |
|
||||
| `count_zeros()` | `n − count_ones()`, O(n/64) |
|
||||
| `iter() -> BitSliceIter<'a>` | O(1) setup, O(n) iteration |
|
||||
| `partial_jaccard_dist(other: BitSliceView)` | `(a&b).popcount`, `(a\|b).popcount` per word, O(n/64) |
|
||||
| `jaccard_dist(other: BitSliceView)` | from partial, O(n/64) |
|
||||
| `hamming_dist(other: BitSliceView)` | `(a^b).popcount` per word, O(n/64) |
|
||||
|
||||
`BitSliceIter<'a>`: word-level scan; one word per 64 iterations.
|
||||
|
||||
### `IntSliceView<'a>`
|
||||
|
||||
```rust
|
||||
#[derive(Clone, Copy)]
|
||||
pub struct IntSliceView<'a> {
|
||||
pub(crate) primary: &'a [u8],
|
||||
pub(crate) overflow_raw: &'a [u8], // sorted [(slot:u64, value:u32)] entries
|
||||
pub(crate) n_overflow: usize,
|
||||
pub(crate) n: usize,
|
||||
}
|
||||
```
|
||||
|
||||
`overflow_raw` contains `n_overflow` entries of `OVERFLOW_ENTRY_SIZE` bytes each, sorted by slot. The sort invariant is established at `close()`/`freeze()` time.
|
||||
|
||||
| Method | Cost |
|
||||
|---|---|
|
||||
| `len()`, `is_empty()` | O(1) |
|
||||
| `primary_bytes()` | O(1) |
|
||||
| `overflow_entries() -> impl Iterator<(usize,u32)>` | O(n_overflow) iteration |
|
||||
| `get(slot)` | O(1) primary; binary search O(log k) for overflow slots |
|
||||
| `iter() -> IntSliceViewIter<'a>` | merge scan, O(n + k) |
|
||||
| `sum()` | byte scan + overflow, O(n + k) |
|
||||
| `count_nonzero()` | byte scan, O(n) |
|
||||
| Distance methods (`bray_dist`, `euclidean_dist`, `jaccard_dist`, …) | O(n + k) |
|
||||
|
||||
`IntSliceViewIter<'a>`: merge scan using `overflow_pos` index. Requires sorted overflow — guaranteed by the construction lifecycle.
|
||||
|
||||
**Builder `view()` vs reader `view()`:** `PersistentCompactIntVecBuilder` stores overflow as an unsorted `HashMap`, not raw bytes. Its `view()` returns an `IntSliceView` with `overflow_raw = &[]` and `n_overflow = 0`. This is intentional — the view is primarily useful after `freeze()`. During building, callers that need overflow use `overflow_entries()` directly.
|
||||
|
||||
---
|
||||
|
||||
## Concrete types
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class BitSliceView {
|
||||
+words: &[u64]
|
||||
+n: usize
|
||||
+get(slot) bool
|
||||
+count_ones() u64
|
||||
+iter() BitSliceIter
|
||||
+jaccard_dist/hamming_dist(other: BitSliceView)
|
||||
}
|
||||
class IntSliceView {
|
||||
+primary: &[u8]
|
||||
+overflow_raw: &[u8]
|
||||
+n_overflow: usize
|
||||
+n: usize
|
||||
+get(slot) u32
|
||||
+iter() IntSliceViewIter
|
||||
+overflow_entries() Iterator
|
||||
+bray_dist/euclidean_dist/…(other: IntSliceView)
|
||||
}
|
||||
class PersistentBitVec {
|
||||
-mmap: Mmap
|
||||
-n: usize
|
||||
+view() BitSliceView
|
||||
+get(slot) bool
|
||||
+count_ones/zeros() u64
|
||||
+iter() BitIter
|
||||
+partial_jaccard_dist(&Self) (u64,u64)
|
||||
+jaccard_dist/hamming_dist(&Self) …
|
||||
}
|
||||
class PersistentBitVecBuilder {
|
||||
-mmap: MmapMut
|
||||
-n: usize
|
||||
+view() BitSliceView
|
||||
+set(slot, bool)
|
||||
+or/and/xor/not(BitSliceView)
|
||||
+copy_from(BitSliceView)
|
||||
+close() / finish() → PersistentBitVec
|
||||
}
|
||||
class PersistentCompactIntVec {
|
||||
-mmap: Mmap
|
||||
-n: usize
|
||||
-n_overflow: usize
|
||||
-step: usize
|
||||
-index: Vec~(usize,usize)~
|
||||
+view() IntSliceView
|
||||
+get(slot) u32
|
||||
+iter() Iter
|
||||
+sum/count_nonzero() u64
|
||||
+bray_dist/euclidean_dist/… (&Self)
|
||||
}
|
||||
class PersistentCompactIntVecBuilder {
|
||||
-mmap: MmapMut
|
||||
-n: usize
|
||||
-overflow: HashMap~usize,u32~
|
||||
+view() IntSliceView
|
||||
+set(slot, u32) / get(slot) u32
|
||||
+inc / inc_present / inc_present_fast
|
||||
+inc_predicate / inc_predicate_fast
|
||||
+add/min/max/diff/mask_with(…View)
|
||||
+primary_bytes/primary_bytes_mut()
|
||||
+close() / finish() → PersistentCompactIntVec
|
||||
}
|
||||
|
||||
PersistentBitVec --> BitSliceView : view()
|
||||
PersistentBitVecBuilder --> BitSliceView : view()
|
||||
PersistentCompactIntVec --> IntSliceView : view()
|
||||
PersistentCompactIntVecBuilder --> IntSliceView : view() (primary only)
|
||||
PersistentBitVecBuilder --> PersistentBitVec : close() then open()
|
||||
PersistentCompactIntVecBuilder --> PersistentCompactIntVec : close() then open()
|
||||
```
|
||||
|
||||
### `PersistentBitVec` / `PersistentBitVecBuilder`
|
||||
|
||||
`PersistentBitVec` is the read-only type. `view()` returns a `BitSliceView<'_>` over the mmap word array. Direct inherent methods delegate to the view: `count_ones()`, `count_zeros()`, `partial_jaccard_dist(&Self)`, `jaccard_dist(&Self)`, `hamming_dist(&Self)`.
|
||||
|
||||
`BitIter<'a>` — exported iterator for `PersistentBitVec::iter()`:
|
||||
|
||||
```rust
|
||||
pub struct BitIter<'a> { pub(crate) words: &'a [u64], pub(crate) slot: usize, pub(crate) n: usize }
|
||||
```
|
||||
|
||||
`PersistentBitVecBuilder` is the read-write type. Mutation operations accept `BitSliceView<'_>`:
|
||||
|
||||
| Method | Cost |
|
||||
|---|---|
|
||||
| `set(slot, bool)` | O(1) |
|
||||
| `view() -> BitSliceView<'_>` | O(1) |
|
||||
| `or/and/xor(BitSliceView)` | word-level, O(n/64), SIMD-friendly |
|
||||
| `not()` | `w ^= u64::MAX` per word, re-masks last word | O(n/64) |
|
||||
| `copy_from(BitSliceView)` | `copy_from_slice` | O(n/64) |
|
||||
|
||||
### `PersistentCompactIntVec` / `PersistentCompactIntVecBuilder`
|
||||
|
||||
`PersistentCompactIntVec` is the read-only type. `view()` returns an `IntSliceView<'_>` over the mmap primary and overflow arrays. Inherent `iter()` is a merge scan (`Iter` struct). Inherent `sum()` and `count_nonzero()` use fast byte-scan helpers.
|
||||
|
||||
`PersistentCompactIntVecBuilder` is the read-write type. Mutation methods on the builder fall into two categories:
|
||||
|
||||
**Point mutations:**
|
||||
|
||||
| Method | Note |
|
||||
|---|---|
|
||||
| `set(slot, u32)` | writes primary[slot] or 255+overflow |
|
||||
| `get(slot) -> u32` | reads primary byte or HashMap |
|
||||
| `inc(slot)` | `get` + `set`, O(1) |
|
||||
|
||||
**Bulk computation methods** — accept view arguments:
|
||||
|
||||
| Method | Semantics | Overflow |
|
||||
|---|---|---|
|
||||
| `inc_present(BitSliceView)` | `+= 1` at each 1-bit | via `inc`, safe for any group size |
|
||||
| `inc_present_fast(BitSliceView)` | same, raw u8 `+= 1` | `debug_assert` no 255 reached |
|
||||
| `inc_predicate(IntSliceView, pred)` | `+= 1` where `pred(col[s])` | two-pass, safe |
|
||||
| `inc_predicate_fast(IntSliceView, pred)` | same, raw u8 | `debug_assert` no 255 reached |
|
||||
| `add(IntSliceView)` | `self[s] += other[s]` | primary fast path + overflow fallback |
|
||||
| `min(IntSliceView)` | byte min + both-overflow fixup | see algorithm below |
|
||||
| `max(IntSliceView)` | pre-pass + byte max | see algorithm below |
|
||||
| `diff(IntSliceView)` | saturating sub | self<255 hot path |
|
||||
| `mask_with(BitSliceView)` | zeros slots where mask bit = 0 | O(n_zeros) |
|
||||
|
||||
**`inc_present_fast` / `inc_predicate_fast` invariant:** caller guarantees no counter reaches 255 during the operation (group size < 255 for `inc_present_fast`, or chunk size < 255 for `inc_predicate_fast`). Violation is caught by `debug_assert` in dev builds.
|
||||
|
||||
**`min` algorithm:**
|
||||
|
||||
Exploits 255 = +∞: byte-level min is correct unless both sides are overflow.
|
||||
|
||||
```
|
||||
snapshot self_ov: Vec<(slot,val)>
|
||||
snapshot other_ov: HashMap<slot,val>
|
||||
clear_overflow()
|
||||
Pass 1 — byte min, SIMD-vectorizable, O(n)
|
||||
Pass 2 — both-overflow fixup, O(k_self):
|
||||
for (slot, self_val) in self_ov:
|
||||
if slot ∈ other_ov: set(slot, min(self_val, other_ov[slot]))
|
||||
```
|
||||
|
||||
**`max` algorithm:**
|
||||
|
||||
Cannot do byte max first — `max(255, b<255)=255` overwrites self's original overflow value. Pre-pass reads self's value at other's overflow slots before the byte pass.
|
||||
|
||||
```
|
||||
Pre-pass O(k_other): for (slot, other_val) in other.overflow_entries():
|
||||
set(slot, max(self.get(slot), other_val))
|
||||
Pass 1 — byte max, SIMD-vectorizable, O(n)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Matrix types
|
||||
|
||||
Four matrix types, two encodings × two formats:
|
||||
|
||||
| | Columnar format | Packed format |
|
||||
|---|---|---|
|
||||
| **Bit** | `PersistentBitMatrix` (Columnar variant) | `PersistentBitMatrix` (Packed variant) |
|
||||
| **Int** | `PersistentCompactIntMatrix` (Columnar variant) | `PersistentCompactIntMatrix` (Packed variant) |
|
||||
|
||||
Both matrix types are enums (`Columnar` / `Packed` / `Implicit` for bit) behind a transparent API. `col_view(c)` returns the appropriate view directly:
|
||||
|
||||
```rust
|
||||
// PersistentBitMatrix
|
||||
pub fn col_view(&self, c: usize) -> BitSliceView<'_>
|
||||
|
||||
// PersistentCompactIntMatrix
|
||||
pub fn col_view(&self, c: usize) -> IntSliceView<'_>
|
||||
```
|
||||
|
||||
No wrapper enums (`BitColView`, `IntColView`): the caller receives a `Copy` view struct immediately usable with any view method or bulk builder method.
|
||||
|
||||
`pack_compact_int_matrix` and `pack_bit_matrix` convert columnar → packed format.
|
||||
|
||||
---
|
||||
|
||||
## Aggregation traits (matrix level)
|
||||
|
||||
### ColumnWeights
|
||||
|
||||
```rust
|
||||
trait ColumnWeights: Send + Sync {
|
||||
fn col_weights(&self) -> Array1<u64>; // sum per column
|
||||
fn partial_kmer_counts(&self) -> Array1<u64>; // default = col_weights()
|
||||
}
|
||||
```
|
||||
|
||||
`partial_kmer_counts` is overridden for count matrices to return `count_nonzero` per column (distinct kmers) rather than total count.
|
||||
|
||||
### CountPartials
|
||||
|
||||
Abstract required methods: `partial_bray`, `partial_euclidean`, `partial_threshold_jaccard`, `partial_relfreq_bray`, `partial_relfreq_euclidean`, `partial_hellinger`.
|
||||
|
||||
**Additivity rule:** self-contained partials (`partial_bray`, `partial_euclidean`, `partial_threshold_jaccard`) can be element-wise summed across all `(partition, layer)` pairs. Normalised partials (`partial_relfreq_*`, `partial_hellinger`) require the **global** `col_weights` (accumulated across all layers and all partitions) as parameter.
|
||||
|
||||
**`partial_threshold_jaccard` returns `(inter, union)`** because `union[i,j]` depends on both columns simultaneously.
|
||||
|
||||
Provided finalisations:
|
||||
|
||||
| Finalisation | Formula |
|
||||
|---|---|
|
||||
| `bray_dist_matrix()` | `1 − 2·partial_bray[i,j] / (w[i] + w[j])` |
|
||||
| `euclidean_dist_matrix()` | `√partial_euclidean[i,j]` |
|
||||
| `threshold_jaccard_dist_matrix(t)` | `1 − inter[i,j] / union[i,j]` |
|
||||
| `relfreq_bray_dist_matrix()` | `1 − partial_relfreq_bray[i,j]` |
|
||||
| `relfreq_euclidean_dist_matrix()` | `√partial_relfreq_euclidean[i,j]` |
|
||||
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
|
||||
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
|
||||
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
|
||||
|
||||
### BitPartials
|
||||
|
||||
Required: `partial_jaccard() -> (Array2<u64>, Array2<u64>)`, `partial_hamming() -> Array2<u64>`. Both additive across layers and partitions.
|
||||
|
||||
Provided finalisations also include `jaccard_dist_matrix()`, `hamming_dist_matrix()`, and `mash_dist_matrix(k)`.
|
||||
|
||||
### Mash distance
|
||||
|
||||
`mash_dist_matrix`/`threshold_mash_dist_matrix` add no new additive primitive: both are a pointwise transform of the existing Jaccard distance matrix, per the Mash mutation-rate estimator [@Mash-distances-doc; @Fan2015-mash-formula]:
|
||||
|
||||
```
|
||||
D = -1/k · ln(2J / (1+J)), J = 1 - d_jaccard
|
||||
```
|
||||
|
||||
`J ≤ 0` (i.e. `d_jaccard ≥ 1`, no shared k-mers) maps to `D = 1` (maximal distance) rather than the `ln` singularity at `J = 0`.
|
||||
|
||||
---
|
||||
|
||||
## Temp-file-backed types
|
||||
|
||||
**All inter-function results use temp-file-backed types** so the OS can page them out under memory pressure. This matters in practice: processing dozens of layers × hundreds of partitions in parallel would otherwise accumulate gigabytes of live anonymous memory.
|
||||
|
||||
### Lifecycle
|
||||
|
||||
```
|
||||
TempCompactIntVecBuilder::new(n) → writable mmap in TempDir
|
||||
↓ (inc_present_fast / inc_predicate_fast / add / mask_with / …)
|
||||
.freeze() → TempCompactIntVec (read-only mmap + TempDir)
|
||||
↓ (optional)
|
||||
.make_persistent(path) → PersistentCompactIntVec (permanent file)
|
||||
```
|
||||
|
||||
Same pattern for `TempBitVecBuilder` → `TempBitVec` → `PersistentBitVec`.
|
||||
|
||||
**Drop order**: `TempCompactIntVec { vec: PersistentCompactIntVec, _temp: TempDir }` — Rust drops fields in declaration order. `vec` (mmap) released before `_temp` (directory deleted). No explicit `drop()` needed.
|
||||
|
||||
### TempCompactIntVec / TempCompactIntVecBuilder
|
||||
|
||||
```rust
|
||||
pub struct TempCompactIntVec {
|
||||
vec: PersistentCompactIntVec,
|
||||
_temp: TempDir, // dropped after vec
|
||||
}
|
||||
|
||||
pub(crate) struct TempCompactIntVecBuilder {
|
||||
builder: PersistentCompactIntVecBuilder,
|
||||
temp: TempDir,
|
||||
}
|
||||
```
|
||||
|
||||
`TempCompactIntVec`: read access via `get(slot)`, `sum()`, `iter()`, `view() -> IntSliceView<'_>`.
|
||||
|
||||
`TempCompactIntVecBuilder`: full delegation to inner `PersistentCompactIntVecBuilder` — all bulk computation methods (`inc_present_fast`, `inc_predicate_fast`, `add`, `min`, `max`, `diff`, `mask_with`) are exposed as `pub(crate)`.
|
||||
|
||||
### TempBitVec / TempBitVecBuilder
|
||||
|
||||
```rust
|
||||
pub struct TempBitVec {
|
||||
vec: PersistentBitVec,
|
||||
_temp: TempDir,
|
||||
}
|
||||
|
||||
pub(crate) struct TempBitVecBuilder {
|
||||
builder: PersistentBitVecBuilder,
|
||||
temp: TempDir,
|
||||
}
|
||||
```
|
||||
|
||||
`TempBitVec`: read access via `get(slot)`, `count_ones()`, `view() -> BitSliceView<'_>`, `iter()`.
|
||||
|
||||
`TempBitVecBuilder`: exposes `set(slot, bool)`, `or(BitSliceView)`, and:
|
||||
|
||||
```rust
|
||||
pub(crate) fn or_where(&mut self, col: IntSliceView<'_>, pred: impl Fn(u32) -> bool)
|
||||
```
|
||||
|
||||
`or_where` — two passes, no intermediate allocation:
|
||||
|
||||
```
|
||||
Pass 1 — primary bytes, O(n):
|
||||
for slot in 0..n:
|
||||
b = col.primary_bytes()[slot]
|
||||
if b < 255 AND pred(b as u32): self.set(slot, true)
|
||||
|
||||
Pass 2 — overflow, O(k):
|
||||
for (slot, val) in col.overflow_entries():
|
||||
if pred(val): self.set(slot, true)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Filter / Select API
|
||||
|
||||
### ColGroup
|
||||
|
||||
```rust
|
||||
pub struct ColGroup { pub name: String, pub indices: Vec<usize> }
|
||||
```
|
||||
|
||||
Defined **once at the index level** from column metadata. Valid in all matrices of all layers and partitions — column structure is identical across the entire hierarchy; only rows (kmer slots) are partitioned.
|
||||
|
||||
### Composition axis
|
||||
|
||||
- **Across partitions**: kmer space is partitioned → partial results **concatenated** (disjoint kmer ranges).
|
||||
- **Across layers**: same kmer space, different counts → partial results **aggregated** (add, OR, etc.).
|
||||
|
||||
### MatrixGroupOps
|
||||
|
||||
Five required primitives + two default methods derived from them. All return temp-file-backed types.
|
||||
|
||||
```rust
|
||||
pub trait MatrixGroupOps {
|
||||
// required
|
||||
fn partial_group_presence_count(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
fn partial_group_sum(&self, g: &ColGroup)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
fn partial_group_any(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempBitVec>;
|
||||
fn partial_group_min(&self, g: &ColGroup)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
fn partial_group_max(&self, g: &ColGroup)
|
||||
-> io::Result<TempCompactIntVec>;
|
||||
|
||||
// defaults derived from partial_group_presence_count
|
||||
fn partial_group_all(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempBitVec>; // slot=1 iff count == g.indices.len()
|
||||
fn partial_group_none(&self, g: &ColGroup, threshold: u32)
|
||||
-> io::Result<TempBitVec>; // slot=1 iff count == 0
|
||||
}
|
||||
```
|
||||
|
||||
Implemented for both `PersistentCompactIntMatrix` and `PersistentBitMatrix`.
|
||||
|
||||
For **bit matrices**: values are 0/1, so `partial_group_sum` = `partial_group_presence_count(g, 1)`; `partial_group_min` is AND (set first column then mask-with remaining); `partial_group_max` is OR via `partial_group_any` + `inc_present`.
|
||||
|
||||
**`partial_group_presence_count` — chunking for large groups:**
|
||||
|
||||
When `g.indices.len() < 255`: per-slot counts stay within `u8` range. Use `inc_present_fast` (bit) or `inc_predicate_fast(col_view(c), |v| v >= threshold)` (int) — raw u8 increment, no overflow entry written.
|
||||
|
||||
When `g.indices.len() ≥ 255`: process in chunks of 254 columns, accumulate via `.add(chunk_frozen.view())`.
|
||||
|
||||
**`partial_group_min` (int matrix)**: copy first column via `.add(col_view(first))` (start from 0 ⇒ copy), then `.min(col_view(c))` for remaining.
|
||||
|
||||
**`partial_group_max` (int matrix)**: `.max(col_view(c))` for all columns (start from 0 ⇒ first column acts as copy).
|
||||
|
||||
**`partial_group_any`** uses `or_where` on `TempBitVecBuilder` (two-pass: primary bytes then overflow entries).
|
||||
|
||||
**`partial_group_all` / `partial_group_none`** (default): call `partial_group_presence_count`, then iterate slots to produce the bit result. O(n) extra pass, not chunked.
|
||||
|
||||
### add_col_from — matrix builder integration
|
||||
|
||||
Both matrix builders accept temp-file results directly:
|
||||
|
||||
```rust
|
||||
// PersistentBitMatrixBuilder
|
||||
fn add_col_from(&mut self, src: &TempBitVec) -> io::Result<()>
|
||||
fn add_col_from_int(&mut self, src: &TempCompactIntVec) -> io::Result<()> // nonzero → 1
|
||||
|
||||
// PersistentCompactIntMatrixBuilder
|
||||
fn add_col_from(&mut self, src: &TempCompactIntVec) -> io::Result<()>
|
||||
fn add_col_from_bit(&mut self, src: &TempBitVec) -> io::Result<()> // bit → 0/1 u32
|
||||
```
|
||||
|
||||
`add_col_from` copies the temp file to the matrix directory and increments `n_cols`; `close()` writes `meta.json` with the final column count. No separate `write_meta` step needed.
|
||||
|
||||
### mask_with
|
||||
|
||||
Direct method on `PersistentCompactIntVecBuilder` (and delegation via `TempCompactIntVecBuilder`). Zeros every slot where the corresponding mask bit is 0. Iterates only zero bits — O(n_zeros), O(1) when mask is all-ones.
|
||||
|
||||
```
|
||||
for (w_idx, word) in mask.words():
|
||||
if word == u64::MAX: continue // skip all-ones words
|
||||
zeros = !word
|
||||
while zeros != 0:
|
||||
bit = trailing_zeros(zeros)
|
||||
s = w_idx * 64 + bit
|
||||
if primary[s] != 0: set(s, 0) // clears overflow entry too
|
||||
zeros &= zeros − 1
|
||||
```
|
||||
|
||||
Terminal operation for Filter (retain only selected kmer slots in a count vector) and Select (positional selection without MPHF).
|
||||
@@ -0,0 +1,396 @@
|
||||
# obilayeredmap — layered kmer index crate
|
||||
|
||||
## Purpose
|
||||
|
||||
`obilayeredmap` implements a persistent, incrementally extensible kmer index. Each layer covers a disjoint kmer set and wraps a `ptr_hash` MPHF with associated per-slot data. Adding a new dataset never rebuilds existing layers.
|
||||
|
||||
---
|
||||
|
||||
## Three usage modes
|
||||
|
||||
The MPHF + evidence infrastructure is the same for all modes. The **payload** varies.
|
||||
|
||||
| Mode | Description | Payload type | Storage |
|
||||
|---|---|---|---|
|
||||
| 1. Set | membership test only | `()` | — |
|
||||
| 2. Count | occurrences per kmer per sample | `PersistentCompactIntMatrix` | `counts/` directory |
|
||||
| 3. Presence/absence | which genomes contain each kmer | `PersistentBitMatrix` | `presence/` directory |
|
||||
|
||||
Both `PersistentCompactIntMatrix` and `PersistentBitMatrix` come from the `obicompactvec` crate.
|
||||
|
||||
---
|
||||
|
||||
## Index mode (homogeneity invariant)
|
||||
|
||||
A partitioned index is homogeneous: every layer within a partition shares the same mode. The mode is determined once at `LayeredMap::open()` from `PartitionMeta.mode` and passed to each `Layer::open()` — no per-layer file is read.
|
||||
|
||||
```rust
|
||||
#[derive(Serialize, Deserialize, Default)]
|
||||
#[serde(tag = "type", rename_all = "snake_case")]
|
||||
pub enum IndexMode {
|
||||
#[default]
|
||||
Exact,
|
||||
Approx { b: u8, z: u8 },
|
||||
Hybrid { b: u8, z: u8 },
|
||||
}
|
||||
```
|
||||
|
||||
`IndexMode` is stored once in `PartitionMeta` (`meta.json` at partition root). There is no `layer_meta.json`.
|
||||
|
||||
- **Exact**: writes `evidence.bin` + `unitigs.bin.idx`. Zero false positives.
|
||||
- **Approx**: writes `fingerprint.bin` only. FP rate per kmer = 1/2^b; with Findere z-parameter, z consecutive kmers must all match → effective window FP ≈ 1/2^(b·z). No `.idx` written or required.
|
||||
- **Hybrid**: writes both `fingerprint.bin` and `evidence.bin` + `.idx`. `find()` uses the fingerprint (fast, O(1)); `find_strict()` uses exact evidence.
|
||||
|
||||
---
|
||||
|
||||
## MphfLayer — autonomous kmer → slot mapping
|
||||
|
||||
`MphfLayer` encapsulates the MPHF and evidence store for one layer. It is independent of any payload.
|
||||
|
||||
```rust
|
||||
pub struct MphfLayer {
|
||||
mphf: Mphf,
|
||||
ev: LayerEvidence, // loaded at open() time
|
||||
n: usize,
|
||||
}
|
||||
```
|
||||
|
||||
`LayerEvidence` is an internal enum, not public:
|
||||
|
||||
```rust
|
||||
enum LayerEvidence {
|
||||
Exact { evidence: Evidence, unitigs: UnitigFileReader },
|
||||
Approx { fingerprint: FingerprintVec, unitigs_path: PathBuf },
|
||||
Hybrid { evidence: Evidence, unitigs: UnitigFileReader, fingerprint: FingerprintVec },
|
||||
}
|
||||
```
|
||||
|
||||
`MphfLayer::open(dir, mode: &IndexMode)` receives the mode from `PartitionMeta` — no per-layer file is read.
|
||||
|
||||
### Query API
|
||||
|
||||
Two public query methods, both returning `Option<usize>` (slot index):
|
||||
|
||||
```rust
|
||||
pub fn find(&self, kmer: CanonicalKmer) -> Option<usize>
|
||||
pub fn find_strict(&self, kmer: CanonicalKmer) -> Option<usize>
|
||||
```
|
||||
|
||||
- `find`: O(1) auto-dispatch. Exact/Hybrid → exact evidence check. Approx/Hybrid → fingerprint comparison.
|
||||
- `find_strict`: always exact. Exact/Hybrid → O(1) evidence check. Approx → O(n) sequential scan (no `.idx`).
|
||||
|
||||
There are no `find_exact`/`find_approx` methods; panicking dispatch is eliminated.
|
||||
|
||||
### Build surface
|
||||
|
||||
```rust
|
||||
// Full MPHF + evidence build (two-pass)
|
||||
pub(crate) fn build(dir, block_bits, mode: &IndexMode, fill_slot) -> OLMResult<usize>
|
||||
|
||||
// Evidence-only post-hoc builds (MPHF already present)
|
||||
pub fn build_exact_evidence(dir, block_bits) -> OLMResult<usize>
|
||||
pub fn build_approx_evidence(dir, b, z) -> OLMResult<usize>
|
||||
```
|
||||
|
||||
`MphfLayer::build` runs two passes over `unitigs.bin`:
|
||||
|
||||
1. **Pass 1** (parallel via rayon): a `CanonicalKmerIter` (clonable, `Arc<Mmap>`, no file reopening) is passed to `new_from_par_iter` via `par_bridge()`. Produces `mphf.bin`. No `.idx` is read or created at this stage.
|
||||
2. **Pass 2** (sequential): fill evidence files; call `fill_slot(slot, kmer)` per kmer. `.idx` is written last for Exact/Hybrid modes (query-time only).
|
||||
|
||||
There is no `build_evidence` dispatch wrapper — callers invoke `build_exact_evidence` or `build_approx_evidence` directly.
|
||||
|
||||
For empty layers (n = 0), all build variants return `Ok(0)` immediately after creating empty output files.
|
||||
|
||||
---
|
||||
|
||||
## Layer\<D: LayerData\> — MPHF + payload
|
||||
|
||||
`Layer<D>` pairs an `MphfLayer` with one payload store.
|
||||
|
||||
```rust
|
||||
pub trait LayerData: Sized {
|
||||
type Item;
|
||||
fn open(layer_dir: &Path) -> OLMResult<Self>;
|
||||
fn read(&self, slot: usize) -> Self::Item;
|
||||
}
|
||||
|
||||
pub struct Layer<D: LayerData = ()> {
|
||||
mphf: MphfLayer,
|
||||
data: D,
|
||||
}
|
||||
|
||||
pub struct Hit<T = ()> {
|
||||
pub slot: usize,
|
||||
pub data: T,
|
||||
}
|
||||
```
|
||||
|
||||
`LayerData` covers the **read path only** (`open` + `read`). Build signatures differ between modes and are not part of the trait.
|
||||
|
||||
| Type | `Item` | Description |
|
||||
|---|---|---|
|
||||
| `()` | `()` | mode 1 — membership only |
|
||||
| `PersistentCompactIntMatrix` | `Box<[u32]>` | mode 2 — count matrix (one u32 per column per slot) |
|
||||
| `PersistentBitMatrix` | `Box<[bool]>` | mode 3 — presence matrix (one bit per genome per slot) |
|
||||
|
||||
### Build signatures
|
||||
|
||||
```rust
|
||||
// mode 1
|
||||
impl Layer<()> {
|
||||
pub fn build(out_dir: &Path, block_bits: u8, mode: &IndexMode) -> OLMResult<usize>
|
||||
}
|
||||
|
||||
// mode 2
|
||||
impl Layer<PersistentCompactIntMatrix> {
|
||||
pub fn build(out_dir: &Path, block_bits: u8, mode: &IndexMode,
|
||||
count_of: impl Fn(CanonicalKmer) -> u32) -> OLMResult<usize>
|
||||
pub fn build_from_map(out_dir: &Path, block_bits: u8, mode: &IndexMode,
|
||||
counts: &HashMap<CanonicalKmer, u32>) -> OLMResult<usize>
|
||||
}
|
||||
|
||||
// mode 3
|
||||
impl Layer<PersistentBitMatrix> {
|
||||
pub fn build_presence(out_dir: &Path, block_bits: u8, mode: &IndexMode,
|
||||
n_genomes: usize,
|
||||
present_in: impl Fn(CanonicalKmer, usize) -> bool) -> OLMResult<usize>
|
||||
}
|
||||
```
|
||||
|
||||
All build impls delegate to `MphfLayer::build` via a mode-specific `fill_slot` callback. The `mode` parameter is forwarded directly — no `LayerMeta` is written.
|
||||
|
||||
Evidence-only post-hoc builds are accessible directly on `Layer<D>`:
|
||||
|
||||
```rust
|
||||
impl<D: LayerData> Layer<D> {
|
||||
pub fn build_exact_evidence(layer_dir: &Path, block_bits: u8) -> OLMResult<usize>
|
||||
pub fn build_approx_evidence(layer_dir: &Path, b: u8, z: u8) -> OLMResult<usize>
|
||||
}
|
||||
```
|
||||
|
||||
There is no `build_evidence` dispatch wrapper.
|
||||
|
||||
---
|
||||
|
||||
## FingerprintVec and FingerprintVecWriter
|
||||
|
||||
Approximate evidence is stored as a packed b-bit array, one fingerprint per MPHF slot.
|
||||
|
||||
```
|
||||
fingerprint.bin format:
|
||||
magic: b"FPVF" (4 bytes)
|
||||
b: u8 (bits per fingerprint, 1..=64)
|
||||
padding: [0u8; 3]
|
||||
n: u64 LE (number of slots)
|
||||
data: packed bits, ceil(n*b/8) bytes, Lsb0 order
|
||||
```
|
||||
|
||||
```rust
|
||||
impl FingerprintVec {
|
||||
pub fn open(path: &Path) -> OLMResult<Self>
|
||||
pub fn get(&self, slot: usize) -> u64
|
||||
pub fn matches(&self, slot: usize, fingerprint: u64) -> bool
|
||||
pub fn n(&self) -> usize
|
||||
pub fn b(&self) -> u8
|
||||
}
|
||||
```
|
||||
|
||||
`matches(slot, hash)` extracts the b-bit fingerprint stored at `slot` and compares it to the low b bits of `hash`. It is the core operation of `find_approx`.
|
||||
|
||||
---
|
||||
|
||||
## LayeredMap\<D\> — collection of layers
|
||||
|
||||
`LayeredMap<D>` wraps `Vec<Layer<D>>` for a single partition directory.
|
||||
|
||||
```rust
|
||||
pub struct LayeredMap<D: LayerData = ()> {
|
||||
root: PathBuf,
|
||||
meta: PartitionMeta,
|
||||
layers: Vec<Layer<D>>,
|
||||
}
|
||||
```
|
||||
|
||||
`PartitionMeta` (`meta.json` at the partition root) stores `n_layers`.
|
||||
|
||||
### Common methods
|
||||
|
||||
```rust
|
||||
pub fn open(root: &Path) -> OLMResult<Self>
|
||||
pub fn create(root: &Path, mode: IndexMode) -> OLMResult<Self>
|
||||
pub fn n_layers(&self) -> usize
|
||||
pub fn layer(&self, i: usize) -> &Layer<D>
|
||||
pub fn mode(&self) -> &IndexMode
|
||||
pub fn query(&self, kmer: CanonicalKmer) -> Option<(usize, Hit<D::Item>)>
|
||||
pub fn next_layer_writer(&self) -> OLMResult<UnitigFileWriter>
|
||||
```
|
||||
|
||||
`open` reads `PartitionMeta` once, extracts `mode`, and passes it to every `Layer::open` — no per-layer file is read. `create` stores the given mode in `PartitionMeta`.
|
||||
|
||||
`query` probes layers in order and returns `(layer_index, Hit)` on the first match. Expected probe depth: 1 for kmers in layer 0.
|
||||
|
||||
### push_layer
|
||||
|
||||
`push_layer` builds the next layer from a `unitigs.bin` already written via `next_layer_writer`, using `DEFAULT_BLOCK_BITS`:
|
||||
|
||||
```rust
|
||||
// mode 1
|
||||
impl LayeredMap<()> {
|
||||
pub fn push_layer(&mut self) -> OLMResult<usize>
|
||||
}
|
||||
|
||||
// mode 2
|
||||
impl LayeredMap<PersistentCompactIntMatrix> {
|
||||
pub fn push_layer(&mut self, count_of: impl Fn(CanonicalKmer) -> u32) -> OLMResult<usize>
|
||||
pub fn push_layer_from_map(&mut self, counts: &HashMap<CanonicalKmer, u32>) -> OLMResult<usize>
|
||||
}
|
||||
```
|
||||
|
||||
Mode 3 (`PersistentBitMatrix`) has no `push_layer` on `LayeredMap`; callers build directly via `Layer<PersistentBitMatrix>::build_presence`.
|
||||
|
||||
---
|
||||
|
||||
## LayeredStore\<S\> and aggregation traits
|
||||
|
||||
`LayeredStore<S>` is a generic aggregation wrapper over `Vec<S>`. It propagates three traits from `obicompactvec::traits` up the hierarchy via blanket impls:
|
||||
|
||||
```rust
|
||||
pub struct LayeredStore<S>(pub Vec<S>);
|
||||
|
||||
impl<S: ColumnWeights> ColumnWeights for LayeredStore<S> { … } // Σ col_weights across inner stores
|
||||
impl<S: CountPartials> CountPartials for LayeredStore<S> { … } // element-wise Σ partials
|
||||
impl<S: BitPartials> BitPartials for LayeredStore<S> { … } // element-wise Σ partials
|
||||
```
|
||||
|
||||
Because blanket impls compose, `LayeredStore<LayeredStore<S>>` automatically inherits all three traits when `S` does — providing the partitioned level without a separate type.
|
||||
|
||||
**Leaf implementors** (in `obicompactvec`):
|
||||
|
||||
| Type | Traits |
|
||||
|---|---|
|
||||
| `PersistentCompactIntMatrix` | `ColumnWeights` (via `sum()`) + `CountPartials` |
|
||||
| `PersistentBitMatrix` | `ColumnWeights` (via `count_ones()`) + `BitPartials` |
|
||||
|
||||
See [Kmer index architecture](../architecture/index_architecture.md) for the full trait API and the two-pass normalised-metric pattern.
|
||||
|
||||
---
|
||||
|
||||
## On-disk structure
|
||||
|
||||
```
|
||||
partition_root/ ← LayeredMap (one partition)
|
||||
meta.json — {"n_layers": N, "mode": {"type": "exact"|"approx"|"hybrid", ...}}
|
||||
layer_0/ ← Layer
|
||||
mphf.bin — ptr_hash MPHF (epserde format)
|
||||
unitigs.bin — packed 2-bit nucleotide sequences
|
||||
unitigs.bin.idx — UIDX index (Exact/Hybrid only; query-time, never built during MPHF construction)
|
||||
evidence.bin — [u32; n], LE (Exact/Hybrid only)
|
||||
fingerprint.bin — packed b-bit array (Approx/Hybrid only)
|
||||
counts/ [mode 2] PersistentCompactIntMatrix
|
||||
meta.json
|
||||
col_000000.pciv
|
||||
presence/ [mode 3] PersistentBitMatrix
|
||||
meta.json
|
||||
col_000000.pbiv …
|
||||
layer_1/
|
||||
…
|
||||
```
|
||||
|
||||
There is no `layer_meta.json`. The mode is stored once in `PartitionMeta` and is valid for all layers. `unitigs.bin.idx` is built at the end of `build_exact_evidence` — never during MPHF construction — and is consumed at query time only.
|
||||
|
||||
---
|
||||
|
||||
## Evidence encoding (exact)
|
||||
|
||||
`evidence.bin` is a flat `[u32; n]` array with no header. Each u32 encodes one slot:
|
||||
|
||||
```
|
||||
bits [31:7] = chunk_id (25 bits) — index of the unitig chunk
|
||||
bits [6:0] = rank (7 bits) — kmer index within the chunk (0-based)
|
||||
```
|
||||
|
||||
`chunk_id = raw >> 7`, `rank = raw & 0x7F`. Reconstructing the kmer: read k nucleotides at position `rank` within unitig `chunk_id` (requires `unitigs.bin.idx` for random access).
|
||||
|
||||
For k=31, m=11, the observed maximum is ~46 kmers per chunk — well within the 127-kmer u7 capacity.
|
||||
|
||||
---
|
||||
|
||||
## ptr_hash configuration
|
||||
|
||||
```rust
|
||||
type Mphf = PtrHash<
|
||||
u64, // key type: canonical kmer raw encoding
|
||||
CubicEps, // bucket fn: 2.4 bits/key, λ=3.5, α=0.99
|
||||
CachelineEfVec<Vec<CachelineEf>>, // remap: Elias-Fano
|
||||
Xx64, // hasher: XXH3-64 with seed
|
||||
Vec<u8>, // pilots
|
||||
>;
|
||||
```
|
||||
|
||||
`Xx64` is chosen over `FxHash` because canonical kmer raw values are left-aligned u64 with structural zeros in the low bits (42 zeros for k=11, 2 zeros for k=31), which single-multiply hashes distribute poorly.
|
||||
|
||||
`CubicEps` with `PtrHashParams::<CubicEps>::default()` (λ=3.5): 2× slower construction than `Linear/λ=3.0`, ~20% less space.
|
||||
|
||||
---
|
||||
|
||||
## Column append and merge support
|
||||
|
||||
These methods extend existing layers with new genome columns without touching the MPHF.
|
||||
|
||||
### Layer-level genome column append
|
||||
|
||||
```rust
|
||||
impl Layer<PersistentBitMatrix> {
|
||||
pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> bool) -> OLMResult<()>
|
||||
}
|
||||
|
||||
impl Layer<PersistentCompactIntMatrix> {
|
||||
pub fn append_genome_column(layer_dir: &Path, value_of: impl Fn(usize) -> u32) -> OLMResult<()>
|
||||
}
|
||||
```
|
||||
|
||||
Both delegate to the corresponding `PersistentBitMatrix::append_column` / `PersistentCompactIntMatrix::append_column`. They write a new column file (`col_NNNNNN.pbiv` / `col_NNNNNN.pciv`) and update `meta.json` to increment `n_cols`. `value_of` is called once per slot (0..n).
|
||||
|
||||
### Presence matrix initialisation
|
||||
|
||||
```rust
|
||||
impl Layer<()> {
|
||||
pub fn init_presence_matrix(layer_dir: &Path, n_kmers: usize) -> OLMResult<()>
|
||||
}
|
||||
```
|
||||
|
||||
Called on the first merge of a Presence-mode index. Creates `presence/` with `meta.json {"n": n_kmers, "n_cols": 1}` and `col_000000.pbiv` set entirely to `true`. This retroactively records genome 0 (the original source) as present in every slot, satisfying the column-count invariant before any new-source column is appended.
|
||||
|
||||
### Why the MPHF is never rebuilt
|
||||
|
||||
The MPHF, evidence, and unitigs are built once from the kmer set of a layer and are immutable for the lifetime of that layer. Adding a genome column does not change the kmer set — it only appends a new data column indexed by the same slot numbers. The only disk writes are one new `.pciv`/`.pbiv` file and a single `meta.json` update.
|
||||
|
||||
---
|
||||
|
||||
## Add-layer algorithm
|
||||
|
||||
When adding dataset B to an existing index:
|
||||
|
||||
1. For each partition, probe existing layers for kmers of B routed to that partition.
|
||||
2. Collect kmers absent from all layers → `B \ index`.
|
||||
3. Write `B \ index` to a new `unitigs.bin` via `next_layer_writer()`.
|
||||
4. Call `Layer<D>::build` (or `build_presence`) on the new layer directory.
|
||||
5. Call `push_layer` (or `append_layer`) to register the new layer in `meta.json`.
|
||||
|
||||
Each partition's new layer is built independently; the operation is fully parallel across partitions.
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
| crate | role |
|
||||
|---|---|
|
||||
| `ptr_hash 1.1` | MPHF per layer |
|
||||
| `cacheline-ef 1.1` | compact remap inside ptr_hash |
|
||||
| `epserde 0.8` | zero-copy MPHF serialisation |
|
||||
| `memmap2 0.9` | mmap of evidence and fingerprint files |
|
||||
| `bitvec` | packed b-bit fingerprint storage |
|
||||
| `obiskio` | unitig file writer/reader + `.idx` build |
|
||||
| `obicompactvec` | payload types + aggregation traits |
|
||||
| `rayon 1` | parallel MPHF construction pass |
|
||||
| `serde / serde_json` | `PartitionMeta` serialisation |
|
||||
@@ -0,0 +1,22 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/obilayeredmap.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obilayeredmap/src/mphf_layer.rs` — MphfLayer, LayerEvidence enum (Exact/Approx), find(), find_exact(), find_approx()
|
||||
- `obilayeredmap/src/layer.rs` — Layer<D>, trait LayerData, modes () / PersistentCompactIntMatrix / PersistentBitMatrix, build(), build_evidence(), append_genome_column()
|
||||
- `obilayeredmap/src/map.rs` — LayeredMap<D>, push_layer(), query()
|
||||
- `obilayeredmap/src/evidence.rs` — Evidence, EvidenceWriter, encodage chunk_id:rank
|
||||
- `obilayeredmap/src/fingerprint.rs` — FingerprintVec, FingerprintVecWriter, matches()
|
||||
- `obilayeredmap/src/meta.rs` — LayerMeta, EvidenceKind (Exact / Approx { b, z })
|
||||
|
||||
## Notes
|
||||
|
||||
FORT RISQUE DE DÉRIVE. C'est le fichier le plus affecté par les changements récents :
|
||||
- EvidenceKind (Exact / Approx) est désormais un concept de premier plan — toute la sémantique de query en dépend
|
||||
- `LayerEvidence` enum interne à `MphfLayer` : dispatch transparent find() → find_exact() ou find_approx()
|
||||
- `fingerprint.rs` : module entièrement nouveau (FingerprintVec + FingerprintVecWriter)
|
||||
- `build_evidence()` / `build_exact_evidence()` / `build_approx_evidence()` sont nouveaux
|
||||
- `block_bits` dans les fonctions build : O(1) garanti avec le chemin chaud explicit pour block_bits=0
|
||||
- Séparation open() (accès aléatoire, requiert .idx) vs open_sequential() (itération seule)
|
||||
Pratiquement toute cette page est à réécrire.
|
||||
@@ -0,0 +1,179 @@
|
||||
# obipipeline — parallel pipeline library
|
||||
|
||||
`obipipeline` is a generic, multi-threaded data pipeline crate. It connects a **source**, a chain of **stages**, and a **sink** via crossbeam channels, running each stage with a shared worker pool and a biased scheduler.
|
||||
|
||||
## Core types
|
||||
|
||||
| Type alias | Rust type | Role |
|
||||
|---|---|---|
|
||||
| `SourceFn<D>` | `Box<dyn FnMut() -> Result<D, PipelineError> + Send>` | Called repeatedly; `FnMut` because it holds iterator state |
|
||||
| `SharedFn<D>` | `Arc<dyn Fn(D) -> Result<D, PipelineError> + Send + Sync>` | 1→1 transform shared across workers via `Arc::clone` |
|
||||
| `SharedFlatFn<D>` | `Arc<dyn Fn(D, &Sender<Result<D, _>>, &Sender<isize>) + Send + Sync>` | 1→N transform; pushes items into channel, sends delta |
|
||||
| `SinkFn<D>` | `Box<dyn Fn(D) -> Result<(), PipelineError> + Send>` | Final consumer; returns `Result` so errors propagate back |
|
||||
|
||||
Stages come in two variants:
|
||||
|
||||
```rust
|
||||
pub enum Stage<D> {
|
||||
Transform(SharedFn<D>), // 1→1
|
||||
Flat(SharedFlatFn<D>), // 1→N
|
||||
}
|
||||
```
|
||||
|
||||
`Pipeline<D>` holds one `SourceFn`, a `Vec<Stage>`, and one `SinkFn`.
|
||||
`WorkerPool<D>` wraps a `Pipeline` with `n_workers` and channel `capacity`.
|
||||
|
||||
## WorkerPool
|
||||
|
||||
```rust
|
||||
WorkerPool::new(pipeline: Pipeline<D>, n_workers: usize, capacity: usize) -> Self
|
||||
WorkerPool::run(self)
|
||||
```
|
||||
|
||||
| Parameter | Role |
|
||||
|---|---|
|
||||
| `n_workers` | Number of parallel worker threads. Each worker is generic — it executes whichever transform the scheduler assigns it. |
|
||||
| `capacity` | Bound on every crossbeam channel in the pipeline. Controls memory and back-pressure: a full channel blocks the sender until a slot frees. |
|
||||
|
||||
`run` consumes `self` (all fields are moved into threads). It blocks the calling thread until the pipeline has fully drained — source exhausted and every in-flight item processed by the sink — then joins all threads before returning.
|
||||
|
||||
## Data enum
|
||||
|
||||
All pipeline stages communicate through a single user-defined enum:
|
||||
|
||||
```rust
|
||||
enum MyData {
|
||||
Unsigned(u64),
|
||||
Number(f64),
|
||||
Text(String),
|
||||
}
|
||||
```
|
||||
|
||||
Each variant carries the concrete type for one stage's output. The macros pattern-match on this enum to route values between stages.
|
||||
|
||||
## Macros
|
||||
|
||||
Eight low-level macros build individual stages; one high-level macro (`make_pipeline!`) composes them.
|
||||
|
||||
### Low-level
|
||||
|
||||
```rust
|
||||
make_source!(Enum, iterator, OutputVariant) // iterator yields T
|
||||
make_source_fallible!(Enum, iterator, OutputVariant) // iterator yields Result<T, E>
|
||||
|
||||
make_transform!(Enum, func, InputVariant, OutputVariant) // func: T -> U
|
||||
make_transform_fallible!(Enum, func, InputVariant, OutputVariant) // func: T -> Result<U, E>
|
||||
|
||||
make_flat_transform!(Enum, func, InputVariant, OutputVariant) // func: T -> impl IntoIterator<Item=U>
|
||||
make_flat_transform_fallible!(Enum, func, InputVariant, OutputVariant) // func: T -> Result<impl IntoIterator<Item=U>, E>
|
||||
|
||||
make_sink!(Enum, func, InputVariant) // func: T -> ()
|
||||
make_sink_fallible!(Enum, func, InputVariant) // func: T -> Result<(), E>
|
||||
```
|
||||
|
||||
Each macro wraps the closure in the correct smart pointer (`Box` for source/sink, `Arc` for transforms).
|
||||
|
||||
### make_pipeline! DSL
|
||||
|
||||
```
|
||||
make_pipeline! {
|
||||
DataEnum,
|
||||
source iterator => OutputVariant, // or source? for fallible
|
||||
| func: In => Out, // 1→1 non-fallible transform
|
||||
|? func: In => Out, // 1→1 fallible transform
|
||||
|| func: In => Out, // 1→N non-fallible flat transform
|
||||
||? func: In => Out, // 1→N fallible flat transform
|
||||
sink func @ InputVariant, // or sink? for fallible
|
||||
}
|
||||
```
|
||||
|
||||
`?` marks fallibility on source, individual transforms, or sink independently.
|
||||
Implemented as a **TT muncher**: the internal rule `@build` recurses over transform tokens one at a time, accumulating them into a `vec![]`, then terminates on `sink`/`sink?`.
|
||||
|
||||
### make_pipe! DSL
|
||||
|
||||
`make_pipe!` builds a sourceless/sinkless `Pipe<D, In, Out>` — a reusable, composable stage sequence:
|
||||
|
||||
```
|
||||
make_pipe! {
|
||||
DataEnum : InType => OutType,
|
||||
| func: InVariant => OutVariant,
|
||||
|? func: InVariant => OutVariant,
|
||||
|| func: InVariant => OutVariant,
|
||||
||? func: InVariant => OutVariant,
|
||||
}
|
||||
```
|
||||
|
||||
Two pipes compose with `.then(other)`. Apply to an iterator with `.apply(iter, n_workers, capacity)` to get a `PipeIter<Out>` — an iterator over the pipeline output, backed by a background `WorkerPool`. The scatter step in `obikmer` uses `make_pipe!` and `.apply()` rather than the full `make_pipeline!` / `WorkerPool` pattern.
|
||||
|
||||
## Scheduler architecture
|
||||
|
||||
```
|
||||
Source thread ──► [source_rx] ──► Scheduler ──► [worker_tx] ──► Workers (×N)
|
||||
▲ │
|
||||
[stage_rxs] ────────┘◄──────────────────────────────┘
|
||||
[flat_delta_rx] ──► Scheduler (in_flight adjustment)
|
||||
│
|
||||
[sink_err_rx] ← errors from sink (highest priority)
|
||||
│
|
||||
Sink thread
|
||||
```
|
||||
|
||||
The scheduler is a single thread running a biased `Select` over all input channels. Priority order (highest first):
|
||||
|
||||
```
|
||||
index 0 sink_err_rx abort on sink error
|
||||
index 1 flat_delta_rx adjust in_flight before dispatching
|
||||
index 2..=n+1 stage_rxs[n-1..0] drain last stage first
|
||||
index n+2 source_rx pull new data last
|
||||
```
|
||||
|
||||
This back-pressure-friendly ordering ensures downstream stages are drained before new items enter the pipeline.
|
||||
|
||||
**Workers** are generic: each receives a `WorkerTask` — either `Transform(data, stage_idx)` or `Flat(data, stage_idx)`. For `Transform`, the worker calls `f(data)` and sends the result to `stage_txs[stage_idx]`. For `Flat`, the worker calls `f(data, &push_tx, &delta_tx)`: the closure pushes N items into `push_tx` then sends `N-1` to `delta_tx`. The scheduler uses the delta to adjust `in_flight` without knowing N in advance.
|
||||
|
||||
**Termination** uses an `in_flight: isize` counter and a `flat_workers_active: usize` counter:
|
||||
|
||||
- `in_flight` incremented when an item is dispatched from source to workers
|
||||
- `in_flight` decremented when the item exits the last stage to the sink
|
||||
- `flat_workers_active` incremented when a `Flat` task is dispatched, decremented when the delta arrives
|
||||
- the loop exits only when `source_done && in_flight == 0 && flat_workers_active == 0`
|
||||
|
||||
This guarantees all in-flight items complete (including all N outputs of a flat stage) before `join()`.
|
||||
|
||||
## Error handling
|
||||
|
||||
`PipelineError` has four variants:
|
||||
|
||||
| Variant | Meaning |
|
||||
|---|---|
|
||||
| `EndOfStream` | Source exhausted (normal termination, not sent downstream) |
|
||||
| `TypeMismatch` | Wrong enum variant arrived at a stage |
|
||||
| `StepKindMismatch` | Internal routing error |
|
||||
| `StepError(Box<dyn Error + Send + Sync>)` | Error from user code (wrapped by `make_*_fallible!`) |
|
||||
|
||||
Sink errors flow back to the scheduler via a dedicated `Receiver<PipelineError>` registered at index 0 of the Select — the pipeline stops immediately on the first sink error.
|
||||
|
||||
## Example
|
||||
|
||||
```rust
|
||||
enum PipelineData { Unsigned(u64), Number(f64), Text(String) }
|
||||
|
||||
fn to_f64(x: u64) -> f64 { x as f64 }
|
||||
fn format_num(n: f64) -> String { format!("{}", n) }
|
||||
fn reverse(s: String) -> String { s.chars().rev().collect() }
|
||||
fn hash(s: String) -> u64 { /* djb2 */ }
|
||||
fn print_hash(h: u64) -> Result<(), std::io::Error> { println!("{}", h); Ok(()) }
|
||||
|
||||
let pipeline = make_pipeline! {
|
||||
PipelineData,
|
||||
source 1u64..=10 => Unsigned,
|
||||
| to_f64: Unsigned => Number,
|
||||
| format_num: Number => Text,
|
||||
| reverse: Text => Text,
|
||||
| hash: Text => Unsigned,
|
||||
sink? print_hash @ Unsigned,
|
||||
};
|
||||
|
||||
WorkerPool::new(pipeline, 4, 64).run();
|
||||
```
|
||||
@@ -0,0 +1,13 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/obipipeline.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obipipeline/src/lib.rs` — WorkerPool, Pipeline, macro make_pipeline!
|
||||
- `obipipeline/src/scheduler.rs` — Scheduler avec Select biaisé sur les entrées de canaux
|
||||
|
||||
## Notes
|
||||
|
||||
Document stable (librairie générique, peu de risque de dérive).
|
||||
Vérifier si `obipipeline` est toujours utilisé dans la phase scatter de `obikpartitionner`
|
||||
ou s'il a été remplacé par Rayon dans certains chemins.
|
||||
@@ -0,0 +1,143 @@
|
||||
# `obitaxonomy` — taxonomy concept paths
|
||||
|
||||
`obitaxonomy` is a dependency-free crate that defines a typed representation
|
||||
of hierarchical concept paths (taxonomic or otherwise) stored in genome metadata.
|
||||
|
||||
---
|
||||
|
||||
## Concept path syntax
|
||||
|
||||
A concept path is stored as a metadata value with the prefix `taxonomy:/`:
|
||||
|
||||
```
|
||||
taxonomy:/enterobacteriaceae@family/Escherichia@genus/Escherichia coli@species
|
||||
```
|
||||
|
||||
Structure:
|
||||
|
||||
- The `taxonomy:/` prefix is the type discriminator. Any metadata value starting
|
||||
with it is parsed as a `TaxPath`; all others remain plain strings.
|
||||
- The remainder is one or more `/`-separated segments.
|
||||
- Each segment is `name` or `name@rank`, where `rank` is a label for the
|
||||
taxonomic level (e.g. `family`, `genus`, `species`).
|
||||
- Rank annotations are **optional per segment** and can be mixed freely.
|
||||
- Spaces are allowed in both names and ranks.
|
||||
|
||||
### Reserved character
|
||||
|
||||
`@` is reserved throughout the taxonomy system and may **not** appear in:
|
||||
|
||||
| Context | Constraint |
|
||||
|---------|------------|
|
||||
| Segment name | forbidden |
|
||||
| Rank/class label | forbidden |
|
||||
| Metadata key names | forbidden (used as `key@rank` in predicate syntax) |
|
||||
|
||||
`@` is freely allowed in plain-text metadata values (non-taxonomy).
|
||||
|
||||
### Parse errors
|
||||
|
||||
| Condition | Error |
|
||||
|-----------|-------|
|
||||
| Value does not start with `taxonomy:/` | `MissingPrefix` |
|
||||
| No segments after the prefix | `EmptyPath` |
|
||||
| Segment with empty name (consecutive `/`) | `EmptySegmentName` |
|
||||
| Segment with trailing `@` and no rank (`name@`) | `EmptyRankName` |
|
||||
| Segment with more than one `@` | `AmbiguousRank` |
|
||||
|
||||
---
|
||||
|
||||
## Public API
|
||||
|
||||
### `TaxSegment`
|
||||
|
||||
A single node: a name and an optional rank.
|
||||
|
||||
```rust
|
||||
seg.name() // &str
|
||||
seg.rank() // Option<&str>
|
||||
seg.to_string() // "name" or "name@rank"
|
||||
TaxSegment::parse(s) // Result<TaxSegment, TaxError>
|
||||
```
|
||||
|
||||
### `TaxPath`
|
||||
|
||||
```rust
|
||||
TaxPath::parse(s) // Result<TaxPath, TaxError>
|
||||
path.segments() // &[TaxSegment]
|
||||
path.depth() // usize — number of segments
|
||||
path.is_ancestor_of(&other) // bool — prefix match by name, ranks ignored
|
||||
path.name_at_rank("genus") // Option<&str>
|
||||
path.to_string() // reconstructs "taxonomy:/…"
|
||||
```
|
||||
|
||||
`is_ancestor_of` compares segment **names** only — rank annotations are
|
||||
informational and do not affect the ancestry relation.
|
||||
|
||||
```rust
|
||||
let a: TaxPath = "taxonomy:/Enterobacteriaceae@family/Escherichia@genus".parse()?;
|
||||
let b: TaxPath = "taxonomy:/Enterobacteriaceae@family/Escherichia@genus/Escherichia coli@species".parse()?;
|
||||
|
||||
assert!(a.is_ancestor_of(&b)); // true
|
||||
assert!(b.is_ancestor_of(&a)); // false
|
||||
assert!(a.is_ancestor_of(&a)); // true (equal ⇒ ancestor)
|
||||
|
||||
assert_eq!(b.name_at_rank("species"), Some("Escherichia coli"));
|
||||
assert_eq!(b.name_at_rank("genus"), Some("Escherichia"));
|
||||
assert_eq!(b.name_at_rank("order"), None);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integration with `GenomeInfo`
|
||||
|
||||
At index load time, every metadata value is inspected once:
|
||||
|
||||
- Starts with `taxonomy:/` → parsed into `TaxPath`, stored in `genome.taxonomy`.
|
||||
- Otherwise → kept as-is in `genome.meta`.
|
||||
|
||||
```rust
|
||||
struct GenomeInfo {
|
||||
label: String,
|
||||
meta: HashMap<String, String>, // plain text metadata
|
||||
taxonomy: HashMap<String, TaxPath>, // parsed taxonomy metadata
|
||||
}
|
||||
```
|
||||
|
||||
The raw string is not duplicated. `TaxPath::to_string()` reconstructs the
|
||||
original value losslessly for serialisation.
|
||||
|
||||
---
|
||||
|
||||
## Predicate operators (in `filter` / `select`)
|
||||
|
||||
Path predicates use the `~` / `!~` operators. The **stored value** always starts
|
||||
with `/` (rooted path); the **query pattern** does not need to.
|
||||
|
||||
### Path pattern syntax
|
||||
|
||||
| Pattern | Semantics |
|
||||
|---------|-----------|
|
||||
| `A/B` | contiguous sub-path A then B, anywhere in the value |
|
||||
| `/A/B` | value starts with A then B (start-anchored) |
|
||||
| `A/B$` | value ends with A then B (end-anchored) |
|
||||
| `/A/B$` | value is exactly A then B (fully anchored) |
|
||||
| `A@x/B` | A with class `x` followed by B with any class |
|
||||
| `A@x/B@y` | A with class `x` followed by B with class `y` |
|
||||
|
||||
A segment pattern without `@` matches the segment name regardless of its stored class.
|
||||
|
||||
### Rank-aware queries
|
||||
|
||||
```
|
||||
key@rank=value
|
||||
```
|
||||
|
||||
| Predicate form | Semantics |
|
||||
|----------------|-----------|
|
||||
| `key@rank=value` | genome's `key` has `value` at rank `rank` |
|
||||
| `key@rank!=value` | does not |
|
||||
| `key@rank=v1\|v2` | value at `rank` is `v1` or `v2` |
|
||||
|
||||
`~` combined with `@rank` on the key (e.g. `key@genus~pattern`) is not defined
|
||||
and is rejected at parse time.
|
||||
@@ -0,0 +1,270 @@
|
||||
# PersistentBitVec and PersistentBitMatrix
|
||||
|
||||
## Purpose
|
||||
|
||||
`PersistentBitVec` stores a dense bit vector (presence/absence per slot) backed by a single mmap'd file. It is the binary counterpart of `PersistentCompactIntVec` and shares the same lifecycle pattern (builder → close → reader). All bulk operations work on u64 words rather than bytes, giving 8× fewer iterations and enabling the compiler to emit POPCNT and SIMD instructions.
|
||||
|
||||
Typical use: converting k-mer count vectors to presence/absence vectors (with optional threshold), then computing set-theoretic distances (Jaccard) or edit distances (Hamming) between samples.
|
||||
|
||||
`PersistentBitMatrix` wraps multiple `PersistentBitVec` columns in a directory, exposing a column-major binary matrix with row-access API. A single-column bit matrix is a vector at the API level.
|
||||
|
||||
---
|
||||
|
||||
## PersistentBitVec — single-column file
|
||||
|
||||
### File format
|
||||
|
||||
Single `.pbiv` file.
|
||||
|
||||
```
|
||||
offset 0:
|
||||
magic: [u8; 4] = b"PBIV"
|
||||
_pad: [u8; 4] = 0 alignment padding
|
||||
n: u64 number of bits
|
||||
|
||||
offset 16:
|
||||
data: [u64; ⌈n/64⌉] bit words, LSB-first, zero-padded
|
||||
```
|
||||
|
||||
**Header is 16 bytes**, so data starts at an offset divisible by 8. Since `mmap` returns page-aligned memory (≥ 4096-byte aligned), the data slice is u64-aligned, enabling a zero-copy `&[u8] → &[u64]` reinterpretation.
|
||||
|
||||
**Bit layout**: bit `i` is in `data[i >> 6]` at bit position `i & 63` (LSB-first). Bits `[n, ⌈n/64⌉×64)` are **always zero** (padding). This invariant is maintained by all write operations and must be restored by `not()` after flipping.
|
||||
|
||||
**Total file size**: `16 + ⌈n/64⌉ × 8` bytes.
|
||||
|
||||
### Lifecycle
|
||||
|
||||
#### Builder (`PersistentBitVecBuilder`)
|
||||
|
||||
```rust
|
||||
struct PersistentBitVecBuilder {
|
||||
mmap: MmapMut,
|
||||
n: usize,
|
||||
}
|
||||
```
|
||||
|
||||
The file and mmap are created immediately at construction. The header is written once at `new()` or copied from the source at `build_from*()`. `close()` is a single flush — there is no tail to append, unlike `PersistentCompactIntVec`.
|
||||
|
||||
**`new(n: usize, path: &Path) -> io::Result<Self>`**
|
||||
|
||||
Creates the file, writes the header, zero-extends to `16 + ⌈n/64⌉×8` bytes, mmaps immediately. All bits default to 0.
|
||||
|
||||
**`build_from(source: &PersistentBitVec, path: &Path) -> io::Result<Self>`**
|
||||
|
||||
OS-level file copy (no per-bit iteration), then mmap. Initialisation cost: O(file_size).
|
||||
|
||||
**`build_from_counts(source: &PersistentCompactIntVec, threshold: u32, path: &Path) -> io::Result<Self>`**
|
||||
|
||||
Creates a new file, iterates `source` with its merge-scan iterator (O(n)), and writes bits directly into u64 words:
|
||||
|
||||
```rust
|
||||
// bit i = 1 iff source[i] >= threshold
|
||||
words[slot >> 6] |= 1u64 << (slot & 63);
|
||||
```
|
||||
|
||||
Handles overflow values (≥ 255) transparently — the count iterator returns the true u32 value regardless.
|
||||
|
||||
**`build_from_presence(source: &PersistentCompactIntVec, path: &Path) -> io::Result<Self>`**
|
||||
|
||||
Shorthand for `build_from_counts(source, 1, path)`.
|
||||
|
||||
**Bit-level access**
|
||||
|
||||
```rust
|
||||
fn get(&self, slot: usize) -> bool
|
||||
fn set(&mut self, slot: usize, value: bool)
|
||||
```
|
||||
|
||||
Byte-level mmap access: `mmap[16 + slot/8]`, bit `slot % 8`. O(1).
|
||||
|
||||
**Word-level bulk operations**
|
||||
|
||||
All operate on `⌈n/64⌉` u64 words. O(n/64) per call.
|
||||
|
||||
```rust
|
||||
builder.and(&other); // self[i] &= other[i] for all i
|
||||
builder.or(&other); // self[i] |= other[i]
|
||||
builder.xor(&other); // self[i] ^= other[i]
|
||||
builder.not(); // self[i] = !self[i], then re-zero padding bits
|
||||
```
|
||||
|
||||
`and`/`or`/`xor` read `other`'s word slice directly (no allocation). `not()` flips all words then masks the last word's padding bits to restore the invariant.
|
||||
|
||||
**`close(self) -> io::Result<()>`**
|
||||
|
||||
Flushes the mmap. The header was written at construction and is never rewritten. O(1) in Rust code.
|
||||
|
||||
#### Reader (`PersistentBitVec`)
|
||||
|
||||
```rust
|
||||
struct PersistentBitVec {
|
||||
mmap: Mmap,
|
||||
n: usize,
|
||||
path: PathBuf,
|
||||
}
|
||||
```
|
||||
|
||||
**`open(path: &Path) -> io::Result<Self>`**
|
||||
|
||||
Mmaps the file, validates magic, reads `n` from bytes `[8..16]`. O(1).
|
||||
|
||||
**`get(slot: usize) -> bool`**
|
||||
|
||||
Byte-level read from `mmap[16 + slot/8]`. O(1).
|
||||
|
||||
**`iter() -> BitIter<'_>`**
|
||||
|
||||
Sequential scan, byte by byte, yielding `bool` values in slot order. Implements `ExactSizeIterator`. O(n).
|
||||
|
||||
**Aggregates**
|
||||
|
||||
```rust
|
||||
fn count_ones(&self) -> u64 // popcount over all words; padding bits are 0
|
||||
fn count_zeros(&self) -> u64 // n - count_ones()
|
||||
```
|
||||
|
||||
`count_ones` iterates `⌈n/64⌉` words and calls `u64::count_ones()` (maps to `POPCNT`). O(n/64).
|
||||
|
||||
**Distance methods**
|
||||
|
||||
Both operate word by word. O(n/64).
|
||||
|
||||
| Method | Formula | Notes |
|
||||
|---|---|---|
|
||||
| `jaccard_dist(&other) -> f64` | `1 − \|A∩B\| / \|A∪B\|` | `(a&b).count_ones()`, `(a\|b).count_ones()` per word |
|
||||
| `hamming_dist(&other) -> u64` | number of differing bits | `(a^b).count_ones()` per word |
|
||||
|
||||
Edge case (both all-zero → union = 0): `jaccard_dist` returns 0.0.
|
||||
|
||||
### Implementation notes
|
||||
|
||||
#### u64 word view
|
||||
|
||||
The unsafe cast from `&[u8]` to `&[u64]` is sound because:
|
||||
|
||||
1. `mmap` base is page-aligned (≥ 4096-byte boundary).
|
||||
2. Data offset = 16, and `16 % 8 == 0` → the data pointer is 8-byte aligned.
|
||||
3. Data length = `⌈n/64⌉ × 8` bytes — always a multiple of 8.
|
||||
|
||||
This gives zero-copy word-level access with no intermediate allocation.
|
||||
|
||||
#### Padding invariant
|
||||
|
||||
Writing `not()` without masking the last word would corrupt `count_ones()`, `hamming_dist()`, and `jaccard_dist()`. The mask applied after flipping is `(1u64 << (n % 64)) - 1` (no-op if `n % 64 == 0`). All other operations (`and`, `or`, `xor`) preserve existing zero padding since they can only clear or preserve bits already set by `not()`.
|
||||
|
||||
### Complexity
|
||||
|
||||
| Operation | Time | Notes |
|
||||
|---|---|---|
|
||||
| `new` / `open` | O(1) | mmap setup + header parse |
|
||||
| `get` / `set` (builder or reader) | O(1) | byte-level mmap |
|
||||
| `iter()` | O(n) | byte-by-byte scan |
|
||||
| `count_ones` / `count_zeros` | O(n/64) | POPCNT per u64 word |
|
||||
| `and` / `or` / `xor` / `not` | O(n/64) | word-level bitwise ops |
|
||||
| `jaccard_dist` / `hamming_dist` | O(n/64) | word AND/OR/XOR + POPCNT |
|
||||
| `build_from` | O(file_size) | OS copy |
|
||||
| `build_from_counts` / `build_from_presence` | O(n) | count iter + word fill |
|
||||
| `close` | O(1) | flush only |
|
||||
|
||||
---
|
||||
|
||||
## PersistentBitMatrix — column-major directory
|
||||
|
||||
### Design
|
||||
|
||||
A directory containing `meta.json` and N column files `col_000000.pbiv`, `col_000001.pbiv`, …, each a `PersistentBitVec`. Used for presence/absence matrices: one column per genome, one bit per MPHF slot.
|
||||
|
||||
```
|
||||
presence/
|
||||
meta.json {"n": <n_slots>, "n_cols": <G>}
|
||||
col_000000.pbiv genome 0
|
||||
col_000001.pbiv genome 1
|
||||
...
|
||||
```
|
||||
|
||||
Column-major layout makes per-genome set operations (Jaccard, Hamming, AND/OR) cache-friendly — each genome is a contiguous file. Row access (which genomes contain a given kmer) requires one O(1) read per column.
|
||||
|
||||
### Builder (`PersistentBitMatrixBuilder`)
|
||||
|
||||
```rust
|
||||
struct PersistentBitMatrixBuilder {
|
||||
dir: PathBuf,
|
||||
n: usize,
|
||||
n_cols: usize,
|
||||
}
|
||||
```
|
||||
|
||||
**`new(n: usize, dir: &Path) -> io::Result<Self>`**
|
||||
|
||||
Creates the directory (including parents).
|
||||
|
||||
**`add_col(&mut self) -> io::Result<PersistentBitVecBuilder>`**
|
||||
|
||||
Creates `col_NNNNNN.pbiv` for the next column and returns its builder. The caller fills the column and calls `builder.close()` before calling `add_col` again.
|
||||
|
||||
**`close(self) -> io::Result<()>`**
|
||||
|
||||
Writes `meta.json` with the final `n` and `n_cols`.
|
||||
|
||||
### Reader (`PersistentBitMatrix`)
|
||||
|
||||
```rust
|
||||
struct PersistentBitMatrix {
|
||||
cols: Vec<PersistentBitVec>,
|
||||
n: usize,
|
||||
}
|
||||
```
|
||||
|
||||
**`open(dir: &Path) -> io::Result<Self>`**
|
||||
|
||||
Reads `meta.json`, opens all `col_NNNNNN.pbiv` files.
|
||||
|
||||
**`row(slot: usize) -> Box<[bool]>`**
|
||||
|
||||
Returns the presence vector: `[col_0[slot], col_1[slot], …, col_{G-1}[slot]]`. One byte read per column. O(G).
|
||||
|
||||
**`col(c: usize) -> &PersistentBitVec`**
|
||||
|
||||
Direct access to a single column for column-oriented operations.
|
||||
|
||||
### LayerData implementation
|
||||
|
||||
```rust
|
||||
impl LayerData for PersistentBitMatrix {
|
||||
type Item = Box<[bool]>;
|
||||
fn open(layer_dir: &Path) -> OLMResult<Self> { /* opens layer_dir/presence/ */ }
|
||||
fn read(&self, slot: usize) -> Box<[bool]> { self.row(slot) }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Aggregation traits — `obicompactvec::traits`
|
||||
|
||||
`PersistentBitMatrix` implements two aggregation traits used by `LayeredStore<S>` for cross-layer and cross-partition distance computations.
|
||||
|
||||
### ColumnWeights
|
||||
|
||||
```rust
|
||||
impl ColumnWeights for PersistentBitMatrix {
|
||||
fn col_weights(&self) -> Array1<u64> // = self.count_ones()
|
||||
}
|
||||
```
|
||||
|
||||
`col_weights()[c]` = number of set bits in column `c` across all slots.
|
||||
|
||||
### BitPartials
|
||||
|
||||
```rust
|
||||
impl BitPartials for PersistentBitMatrix {
|
||||
// Self-contained partials (additive across layers)
|
||||
fn partial_jaccard(&self) -> (Array2<u64>, Array2<u64>) // (inter, union)
|
||||
fn partial_hamming(&self) -> Array2<u64> // differing bits
|
||||
|
||||
// Provided finalisations
|
||||
fn jaccard_dist_matrix(&self) -> Array2<f64>
|
||||
fn hamming_dist_matrix(&self) -> Array2<u64>
|
||||
}
|
||||
```
|
||||
|
||||
`partial_jaccard` returns `(inter, union)` as a pair because `union` is not reconstructible from per-column `count_ones()` — it depends on both columns simultaneously. Both components are additively decomposable across `(partition, layer)` pairs; the final `jaccard_dist_matrix()` is computed from their element-wise sums.
|
||||
@@ -0,0 +1,13 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/persistent_bit_vec.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obicompactvec/src/bitvec.rs` — PersistentBitVec, opérations mot u64, invariant de padding
|
||||
- `obicompactvec/src/bitmatrix.rs` — PersistentBitMatrix, wrapper colonne-major, append_column
|
||||
- `obicompactvec/src/bitmatrix.rs` — PersistentBitMatrixBuilder
|
||||
|
||||
## Notes
|
||||
|
||||
Document d'implémentation stable. Vérifier que `PersistentBitMatrixBuilder` et `append_column`
|
||||
sont couverts (utilisés dans `Layer::<PersistentBitMatrix>::build_presence` et `append_genome_column`).
|
||||
@@ -0,0 +1,308 @@
|
||||
# PersistentCompactIntVec and PersistentCompactIntMatrix
|
||||
|
||||
## Purpose
|
||||
|
||||
`PersistentCompactIntVec` stores a dense array of non-negative integers indexed by MPHF slot where the vast majority of values are small (0–254) and large values are rare. It is designed for mmap-compatible random and sequential access with minimal memory footprint and optimal cache behaviour.
|
||||
|
||||
Motivation from observed count distributions in genomics data: 99.9% of k-mer counts fit in a u8; overflow (count ≥ 255) affects ~0.07% of distinct k-mers but can reach values above 10⁶ (chloroplast, ribosomal repeats).
|
||||
|
||||
`PersistentCompactIntMatrix` wraps multiple `PersistentCompactIntVec` columns in a directory, exposing a column-major matrix with row-access API. A vector is a matrix with 1 column.
|
||||
|
||||
---
|
||||
|
||||
## PersistentCompactIntVec — single-column file
|
||||
|
||||
### Design
|
||||
|
||||
Two-tier structure:
|
||||
|
||||
1. **Primary array** — `[u8; n]`, stored at offset 40 in the PCIV file and mmap'd. Values 0–254 are stored directly. Value **255 is a sentinel** meaning "look in overflow".
|
||||
2. **Overflow section** — sorted list of `(slot: u64, value: u32)` pairs for all slots where the true value ≥ 255, with a **sparse L1-fitting index** for fast lookup.
|
||||
|
||||
```
|
||||
primary[slot] < 255 → return primary[slot]
|
||||
primary[slot] == 255 → binary search in overflow
|
||||
```
|
||||
|
||||
### File format
|
||||
|
||||
Single `.pciv` file. Write order: header placeholder → primary → overflow + index → header overwrite at offset 0.
|
||||
|
||||
```
|
||||
offset 0:
|
||||
magic: [u8; 4] = b"PCIV"
|
||||
_pad: [u8; 4] = 0
|
||||
n: u64 number of slots
|
||||
n_overflow: u64 number of overflow entries
|
||||
n_index: u64 number of sparse index entries
|
||||
step: u64 sparse index step (0 = no index)
|
||||
|
||||
offset 40:
|
||||
primary: [u8; n] one byte per slot, 255 = overflow sentinel
|
||||
|
||||
offset 40 + n:
|
||||
data: [(slot: u64, value: u32); n_overflow] 12 bytes each, sorted by slot
|
||||
|
||||
offset 40 + n + n_overflow × 12:
|
||||
index: [(slot: u64, pos: u64); n_index] 16 bytes each, sparse index
|
||||
```
|
||||
|
||||
The index entries point into `data`: `index[i] = (slot of data[i×step], i×step)`.
|
||||
|
||||
All integer fields are little-endian. Slot indices are stored as `u64` in the file; they are `usize` in Rust code.
|
||||
|
||||
### Lifecycle
|
||||
|
||||
#### Builder (`PersistentCompactIntVecBuilder`)
|
||||
|
||||
Used during construction. The primary section is **mmap'd immediately** at construction time (both for `new` and `build_from`), so the file exists and is addressable from the start. The overflow is held in a `HashMap<usize, u32>` in RAM.
|
||||
|
||||
```rust
|
||||
struct PersistentCompactIntVecBuilder {
|
||||
path: PathBuf,
|
||||
mmap: MmapMut, // primary section live in the file from the start
|
||||
n: usize,
|
||||
overflow: HashMap<usize, u32>, // values ≥ 255
|
||||
}
|
||||
```
|
||||
|
||||
**`new(n: usize, path: &Path) -> io::Result<Self>`**
|
||||
|
||||
Creates the file, pre-allocates `HEADER_SIZE + n` zero bytes, mmaps it. The primary is zero-initialised (all slots = 0). Returns immediately ready for `set` / `get`.
|
||||
|
||||
**`build_from(source: &PersistentCompactIntVec, path: &Path) -> io::Result<Self>`**
|
||||
|
||||
Copies the source PCIV file to `path` (OS-level copy — no per-slot iteration), mmaps the copy, then loads the overflow section into a `HashMap`. Initialisation cost: O(file copy) + O(n_overflow), not O(n).
|
||||
|
||||
At `close()`, the primary section is **not rewritten**: it is already in the file via mmap. Only the overflow data, the sparse index, and the header are updated.
|
||||
|
||||
**`set(slot: usize, value: u32)` / `get(slot: usize) -> u32`**
|
||||
|
||||
Direct mmap byte access for the primary; HashMap for the overflow. Both O(1). Mutations can move a slot between tiers freely (downward mutation removes the HashMap entry; upward mutation adds it).
|
||||
|
||||
**Element-wise operations — `min`, `max`, `add`, `diff`**
|
||||
|
||||
Each takes a `&PersistentCompactIntVec` of equal length and updates `self` in place via `set`:
|
||||
|
||||
```rust
|
||||
builder.min(&other); // self[i] = min(self[i], other[i])
|
||||
builder.max(&other); // self[i] = max(self[i], other[i])
|
||||
builder.add(&other); // self[i] = self[i].checked_add(other[i]) (panics on u32 overflow)
|
||||
builder.diff(&other); // self[i] = self[i].saturating_sub(other[i])
|
||||
```
|
||||
|
||||
All iterate `other` with `other.iter()` (merge-scan, O(n_other)).
|
||||
|
||||
**`close(self) -> io::Result<()>`**
|
||||
|
||||
1. Flush and drop the mmap (primary changes are now on disk).
|
||||
2. Sort the overflow HashMap into `Vec<(usize, u32)>`.
|
||||
3. Truncate the file to `HEADER_SIZE + n` (removes old data+index if `build_from` was used).
|
||||
4. Append sorted overflow data, then sparse index.
|
||||
5. Seek to offset 0, overwrite the header with final values.
|
||||
|
||||
#### Reader (`PersistentCompactIntVec`)
|
||||
|
||||
Used at query time. The whole file is mmap'd; only the sparse index is copied into a `Vec` at open time (≤ 32 KB, L1-resident).
|
||||
|
||||
```rust
|
||||
struct PersistentCompactIntVec {
|
||||
mmap: Mmap,
|
||||
n: usize,
|
||||
n_overflow: usize,
|
||||
step: usize,
|
||||
index: Vec<(usize, usize)>, // (slot, pos) — L1-resident
|
||||
primary_offset: usize, // = 40 (HEADER_SIZE)
|
||||
data_offset: usize, // = 40 + n
|
||||
path: PathBuf,
|
||||
}
|
||||
```
|
||||
|
||||
**`open(path: &Path) -> io::Result<Self>`**
|
||||
|
||||
Mmaps the file, parses the 40-byte header, copies the sparse index entries into a `Vec`. The primary and data sections stay mmap'd.
|
||||
|
||||
**`get(slot: usize) -> u32` — random access**
|
||||
|
||||
```
|
||||
primary[slot] < 255 → return it directly
|
||||
|
||||
step == 0:
|
||||
binary_search(data[0..n_overflow], slot)
|
||||
|
||||
step > 0:
|
||||
i = upper_bound(index[..].slot, slot) − 1 // in L1-resident Vec
|
||||
binary_search(data[index[i].pos .. index[i+1].pos], slot)
|
||||
```
|
||||
|
||||
**`iter() -> Iter<'_>` — sequential scan, O(n)**
|
||||
|
||||
Merge-scan: reads primary bytes in order; on sentinel 255, advances a sequential pointer into the sorted data section rather than doing a binary search. This gives O(n + n_overflow) with no random access into the data section.
|
||||
|
||||
`Iter` implements `ExactSizeIterator`. `&PersistentCompactIntVec` implements `IntoIterator`.
|
||||
|
||||
**Aggregate**
|
||||
|
||||
```rust
|
||||
fn sum(&self) -> u64 // Σ self[i] as u64, via iter()
|
||||
```
|
||||
|
||||
**Distance methods**
|
||||
|
||||
All take `&other` of equal length, iterate both with `zip(self.iter(), other.iter())`, and return `f64`.
|
||||
|
||||
| Method | Formula |
|
||||
|---|---|
|
||||
| `bray_dist` | `1 − 2·Σmin(aᵢ,bᵢ) / (Σaᵢ + Σbᵢ)` |
|
||||
| `relfreq_bray_dist` | Bray-Curtis on relative frequencies: `1 − Σmin(pᵢ,qᵢ)` where `pᵢ = aᵢ/Σa` |
|
||||
| `euclidean_dist` | `√Σ(aᵢ − bᵢ)²` |
|
||||
| `relfreq_euclidean_dist` | Euclidean on relative frequencies |
|
||||
| `hellinger_euclidean_dist` | `√Σ(√pᵢ − √qᵢ)²` — Euclidean on sqrt(relfreq) |
|
||||
| `hellinger_dist` | `hellinger_euclidean_dist / √2` — standard Hellinger distance ∈ [0, 1] |
|
||||
| `threshold_jaccard_dist(&other, threshold: u32)` | `1 − \|A∩B\| / \|A∪B\|` where presence iff count ≥ threshold |
|
||||
| `jaccard_dist` | `threshold_jaccard_dist(&other, 1)` |
|
||||
|
||||
Edge cases (both vectors all-zero, or union empty for Jaccard): distance = 0.0.
|
||||
|
||||
### Step computation
|
||||
|
||||
Chosen at `close()` once `n_overflow` is known:
|
||||
|
||||
```
|
||||
L1_INDEX_ENTRIES = 2048
|
||||
|
||||
step = 0 if n_overflow ≤ 2048
|
||||
step = ⌈n_overflow / 2048⌉ otherwise
|
||||
```
|
||||
|
||||
### Complexity
|
||||
|
||||
| Operation | Time | Notes |
|
||||
|---|---|---|
|
||||
| `set` / `get` (builder) | O(1) | mmap byte + HashMap |
|
||||
| `get` (reader, no overflow) | O(1) | single mmap byte |
|
||||
| `get` (reader, with index) | O(log step) | ≤ 2 memory regions |
|
||||
| `get` (reader, no index) | O(log n_overflow) | data fits in a few cache lines |
|
||||
| `iter()` full scan | O(n + n_overflow) | merge-scan, no binary search |
|
||||
| `sum`, distances | O(n) | via `iter()` / `zip(iter(), iter())` |
|
||||
| `min` / `max` / `add` / `diff` | O(n) | via `other.iter()` + builder `set` |
|
||||
| `close` | O(n_overflow log n_overflow) | sort + sequential write |
|
||||
| `open` | O(n_index) | index copy into Vec |
|
||||
| `build_from` | O(file_size) + O(n_overflow) | OS copy + HashMap load |
|
||||
|
||||
---
|
||||
|
||||
## PersistentCompactIntMatrix — column-major directory
|
||||
|
||||
### Design
|
||||
|
||||
A directory containing `meta.json` and N column files `col_000000.pciv`, `col_000001.pciv`, …, each a `PersistentCompactIntVec`. This is the type used by `LayerData` — a single-column matrix is functionally equivalent to a vector but shares the same interface as multi-column matrices.
|
||||
|
||||
```
|
||||
counts/
|
||||
meta.json {"n": <n_slots>, "n_cols": <N>}
|
||||
col_000000.pciv
|
||||
col_000001.pciv
|
||||
...
|
||||
```
|
||||
|
||||
### Builder (`PersistentCompactIntMatrixBuilder`)
|
||||
|
||||
```rust
|
||||
struct PersistentCompactIntMatrixBuilder {
|
||||
dir: PathBuf,
|
||||
n: usize,
|
||||
n_cols: usize,
|
||||
}
|
||||
```
|
||||
|
||||
**`new(n: usize, dir: &Path) -> io::Result<Self>`**
|
||||
|
||||
Creates the directory (including parents). Does not write `meta.json` yet.
|
||||
|
||||
**`add_col(&mut self) -> io::Result<PersistentCompactIntVecBuilder>`**
|
||||
|
||||
Creates `col_NNNNNN.pciv` for the next column and returns its builder. The caller fills the column and calls `builder.close()` before calling `add_col` again.
|
||||
|
||||
**`close(self) -> io::Result<()>`**
|
||||
|
||||
Writes `meta.json` with the final `n` and `n_cols`. Must be called after all column builders are closed.
|
||||
|
||||
### Reader (`PersistentCompactIntMatrix`)
|
||||
|
||||
```rust
|
||||
struct PersistentCompactIntMatrix {
|
||||
cols: Vec<PersistentCompactIntVec>,
|
||||
n: usize,
|
||||
}
|
||||
```
|
||||
|
||||
**`open(dir: &Path) -> io::Result<Self>`**
|
||||
|
||||
Reads `meta.json`, opens all `col_NNNNNN.pciv` files.
|
||||
|
||||
**`row(slot: usize) -> Box<[u32]>`**
|
||||
|
||||
Returns the full row: `[col_0[slot], col_1[slot], …, col_{N-1}[slot]]`. One mmap access per column. O(N).
|
||||
|
||||
**`col(c: usize) -> &PersistentCompactIntVec`**
|
||||
|
||||
Direct access to a single column for column-oriented operations (distance computations, iteration).
|
||||
|
||||
### LayerData implementation
|
||||
|
||||
```rust
|
||||
impl LayerData for PersistentCompactIntMatrix {
|
||||
type Item = Box<[u32]>;
|
||||
fn open(layer_dir: &Path) -> OLMResult<Self> { /* opens layer_dir/counts/ */ }
|
||||
fn read(&self, slot: usize) -> Box<[u32]> { self.row(slot) }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Aggregation traits — `obicompactvec::traits`
|
||||
|
||||
`PersistentCompactIntMatrix` implements two aggregation traits used by `LayeredStore<S>` for cross-layer and cross-partition distance computations.
|
||||
|
||||
### ColumnWeights
|
||||
|
||||
```rust
|
||||
impl ColumnWeights for PersistentCompactIntMatrix {
|
||||
fn col_weights(&self) -> Array1<u64> // = self.sum()
|
||||
}
|
||||
```
|
||||
|
||||
`col_weights()[c]` = sum of all values in column `c` across all slots.
|
||||
|
||||
### CountPartials
|
||||
|
||||
```rust
|
||||
impl CountPartials for PersistentCompactIntMatrix {
|
||||
// Self-contained partials (additive across layers, no external parameter)
|
||||
fn partial_bray(&self) -> Array2<u64>
|
||||
fn partial_euclidean(&self) -> Array2<f64>
|
||||
fn partial_threshold_jaccard(&self, threshold: u32) -> (Array2<u64>, Array2<u64>)
|
||||
|
||||
// Normalised partials (require global col_weights across all layers/partitions)
|
||||
fn partial_relfreq_bray(&self, global: &Array1<u64>) -> Array2<f64>
|
||||
fn partial_relfreq_euclidean(&self, global: &Array1<u64>) -> Array2<f64>
|
||||
fn partial_hellinger(&self, global: &Array1<u64>) -> Array2<f64>
|
||||
|
||||
// Provided finalisations (default implementations on the trait)
|
||||
fn bray_dist_matrix(&self) -> Array2<f64>
|
||||
fn euclidean_dist_matrix(&self) -> Array2<f64>
|
||||
fn threshold_jaccard_dist_matrix(&self, threshold: u32) -> Array2<f64>
|
||||
fn relfreq_bray_dist_matrix(&self) -> Array2<f64>
|
||||
fn relfreq_euclidean_dist_matrix(&self) -> Array2<f64>
|
||||
fn hellinger_dist_matrix(&self) -> Array2<f64>
|
||||
}
|
||||
```
|
||||
|
||||
**Self-contained partials** are additively decomposable: summing `partial_bray()` across all `(partition, layer)` pairs and finalising gives the same result as computing on the combined data.
|
||||
|
||||
**Normalised partials** require the global column weights (sum across all layers and all partitions). The `global` parameter must reflect the complete index, not a per-layer sum. The provided `relfreq_bray_dist_matrix()` etc. call `col_weights()` first (pass 1) then the normalised partial (pass 2); when called on a `LayeredStore<LayeredStore<…>>` these two-pass calls cascade automatically through the blanket impls.
|
||||
|
||||
**`partial_bray` returns `Array2<u64>`** (sum_min only, not a tuple). The denominator is always reconstructible as `col_weights()[i] + col_weights()[j]`.
|
||||
|
||||
**`partial_threshold_jaccard` returns `(inter, union)`** as a pair because `union[i,j]` is not reconstructible from per-column statistics — it depends on both columns simultaneously.
|
||||
@@ -0,0 +1,14 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/persistent_compact_int_vec.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obicompactvec/src/builder.rs` — PersistentCompactIntVecBuilder, cycle de vie
|
||||
- `obicompactvec/src/reader.rs` — PersistentCompactIntVec, accès aléatoire et séquentiel
|
||||
- `obicompactvec/src/intmatrix.rs` — PersistentCompactIntMatrix, wrapper colonne-major, append_column
|
||||
- `obicompactvec/src/format.rs` — format de fichier (magic PCIV, header, primary u8, overflow, index)
|
||||
|
||||
## Notes
|
||||
|
||||
Document d'implémentation stable. Vérifier que `append_column` (utilisé dans merge et reindex)
|
||||
est décrit. Vérifier que `PersistentCompactIntMatrixBuilder` est couvert (utilisé dans `layer.rs`).
|
||||
@@ -0,0 +1,228 @@
|
||||
# Construction pipeline
|
||||
|
||||
All phases after scatter are embarrassingly parallel across partitions.
|
||||
|
||||
## Phase 0 — Parameter estimation
|
||||
|
||||
The construction parameters p, n, and min_count depend on the kmer frequency spectrum of the dataset. Estimating this spectrum before construction avoids costly re-partitioning if p is badly chosen.
|
||||
|
||||
Two approaches are supported:
|
||||
|
||||
- **External estimation (preferred):** run [NT-CARD](https://github.com/bcgsc/ntCard) on the input files and pass its histogram output to `obikmer build`. NT-CARD produces a kmer frequency histogram in a single streaming pass using ntHash and a Flajolet-Martin-style estimator; obikmer reads this file and derives p, n, and min_count automatically.
|
||||
- **Internal estimation (future):** an `obikmer estimate` subcommand for users who prefer a single-tool workflow. The implementation would combine two components: (1) **ntHash**, a rolling hash that updates the kmer hash in O(1) per nucleotide by incrementally adding the incoming base and removing the outgoing one — Rust crates exist; (2) a **Flajolet-Martin-style streaming estimator** that maintains a small table of minimum hash values and infers the frequency histogram from their statistical distribution, as described in the NT-CARD paper [@Mohamadi2017-ok].
|
||||
|
||||
The histogram gives:
|
||||
|
||||
- **F0** (number of distinct kmers) → sets p (target ~10M kmers/partition → p = ⌈log₂(F0 / 10M)⌉)
|
||||
- **frequency distribution** → sets n (choose n so that fewer than 1% of kmers overflow)
|
||||
- **error valley** → suggests min_count (typically the local minimum between the error peak and the coverage peak)
|
||||
|
||||
## Phase 1 — Scatter
|
||||
|
||||
Single streaming pass over raw input files (FASTA/FASTQ, gzip). FASTQ quality scores are ignored.
|
||||
|
||||
Input files are read via `open_nuc_stream`, which opens and decompresses the file, auto-detects the format (FASTA / FASTQ / GenBank), and yields a sequence of `NucPage` buffers. Each `NucPage` is a flat 64 KB buffer of normalised bytes (`ACGT` + `\x00` separators), carrying a k−1 byte overlap from the preceding page so that no k-mer is lost at page boundaries. Per-record identity (sequence id, raw bytes) is not preserved; this is intentional — the scatter phase only needs normalised bases to produce superkmers.
|
||||
|
||||
For each read fragment within a page:
|
||||
|
||||
1. **Ambiguous base filter**: cut at any non-ACGT base; discard fragments shorter than k.
|
||||
2. **Entropy filter**: scan each fragment with a sliding window of size k. When the kmer $K_i = S[i \mathinner{..} i+k-1]$ ended by nucleotide $S[j]$ (with $j = i+k-1$) has entropy below threshold $\theta$, emit the current segment and start a new one (see algorithm below). $K_i$ belongs to neither segment, and no valid kmer is lost.
|
||||
3. **Length filter**: discard any segment shorter than k produced by step 2.
|
||||
4. **Super-kmer extraction**: for each clean segment, slide a minimizer window and group consecutive kmers sharing the same canonical minimizer; canonise each super-kmer by lexicographic comparison with its reverse complement (early exit).
|
||||
5. **Partition routing**: `hash(canonical_minimizer) → PART` → append super-kmer to `partition/superkmers.bin.gz`.
|
||||
|
||||
**Segmentation behavior:**
|
||||
|
||||
When $K_i$ (ended by $S[j]$, $j = i+k-1$) fails the entropy threshold:
|
||||
|
||||
- Current segment $S[\textit{seg_start} \mathinner{..} j-1]$ is emitted (last valid kmer = $K_{i-1}$)
|
||||
- New segment starts at $S[i+1]$ (first new kmer = $K_{i+1}$)
|
||||
- $K_i$ is excluded: current segment lacks $S[j]$, new segment lacks $S[i]$
|
||||
- Overlap = $S[i+1 \mathinner{..} j-1]$ = $k-2$ nucleotides
|
||||
|
||||
!!! abstract "Algorithm — Entropy filter: sliding window segmentation"
|
||||
```text
|
||||
procedure EntropyFilter(S, N, k, θ):
|
||||
seg_start ← 0
|
||||
window ← []
|
||||
for j ← 0 to N−1:
|
||||
window.push(S[j])
|
||||
if |window| < k: continue
|
||||
i ← j − k + 1
|
||||
if entropy(window) ≤ θ:
|
||||
emit S[seg_start .. j−1]
|
||||
seg_start ← i + 1
|
||||
window ← S[i+1 .. j]
|
||||
else:
|
||||
window.pop_front()
|
||||
emit S[seg_start .. N−1]
|
||||
```
|
||||
|
||||
Writes are sequential and append-only — IO-friendly. Gzip applied at write time. Data volume ≈ raw genome size (2 bits/nt compaction offsets header overhead).
|
||||
|
||||
## Phase 2 — Dereplication
|
||||
|
||||
Performed independently per partition. Identical super-kmers are consolidated and their COUNT accumulated — analogous to amplicon dereplication in metabarcoding. Uses external bucket sort to stay within RAM bounds:
|
||||
|
||||
**Pass 1** (streaming): hash the nucleotide payload of each super-kmer, route to one of B bucket files:
|
||||
```
|
||||
hash(sequence) % B → bucket_i.bin
|
||||
```
|
||||
B ≈ 100 is tunable; RAM needed ≈ partition_size / B.
|
||||
|
||||
**Pass 2**: for each bucket, load into an in-memory `HashMap<sequence, COUNT>`, dereplicate by summing COUNT values, write consolidated super-kmers.
|
||||
|
||||
After dereplication: at Nx coverage the partition shrinks by ~x (errors aside). The COUNT field in each super-kmer header = number of times that exact super-kmer sequence was observed across all input reads.
|
||||
|
||||
**Important:** super-kmer COUNT ≠ individual kmer count. A kmer can appear in multiple distinct super-kmers (same partition, different flanking context); its true count = sum of COUNT of all super-kmers containing it. A super-kmer with COUNT=1 may contain only high-abundance kmers, each appearing in many other super-kmers. Abundance filtering therefore cannot be applied at this phase.
|
||||
|
||||
## Phase 3 — Per-kmer count aggregation and quorum filtering
|
||||
|
||||
For each dereplicated super-kmer, enumerate its kmers and accumulate counts:
|
||||
|
||||
```
|
||||
for each super-kmer (sequence, COUNT):
|
||||
for each kmer in sequence:
|
||||
kmer_counts[canonical(kmer)] += COUNT
|
||||
```
|
||||
|
||||
Implemented as a three-step pipeline in `count_partition()`:
|
||||
|
||||
1. **External sort** (`kmer_sort::sort_unique_kmers`): read dereplicated superkmers, extract canonical kmer raw `u64` values, sort in RAM-bounded chunks (adaptive: 40% of available RAM ÷ n_threads, min 1 M kmers/chunk), k-way merge with inline dedup → `sorted_unique.bin`. f0 is now known exactly.
|
||||
2. **Provisional MPHF** (ptr_hash): built from `sorted_unique.bin` via `new_from_par_iter(f0, ...)`. Stored to `mphf1.bin`; `sorted_unique.bin` deleted immediately.
|
||||
3. **Accumulation pass**: re-read dereplicated superkmers; for each kmer, `slot = mphf.index(kmer.raw())`, increment `counts1[slot]` by the superkmer COUNT. Stored in a `PersistentCompactIntVec` (`counts1.bin`).
|
||||
|
||||
At the end of this phase, each distinct canonical kmer has its exact total count, and the frequency spectrum (`spectrums/{label}.json`) is written to the index root.
|
||||
|
||||
No pre-filter on super-kmer COUNT is possible at phase 2: a super-kmer with COUNT=1 may contain only high-abundance kmers, each present in many other super-kmers across the partition.
|
||||
|
||||
## Phase 4 — Super-kmer compaction
|
||||
|
||||
The valid kmer set from phase 3 is used as a mask to rewrite the super-kmer files:
|
||||
|
||||
```
|
||||
for each dereplicated super-kmer:
|
||||
scan kmer by kmer
|
||||
kmer not in valid set → break point (terminates current super-kmer)
|
||||
kmer in valid set → extend current super-kmer
|
||||
```
|
||||
|
||||
Three cases per super-kmer:
|
||||
|
||||
- **All kmers valid** → copied as-is
|
||||
- **No kmer valid** → discarded
|
||||
- **Mixed** → split into sub-super-kmers at invalid boundaries; each sub-super-kmer inherits the original COUNT
|
||||
|
||||
After splitting, re-apply dereplication (bucket sort, phase 2 method) — splitting can produce new identical super-kmers. This re-dereplication is cheap: the volume is already greatly reduced.
|
||||
|
||||
Output: a clean super-kmer file where every kmer passes quorum. This file feeds phase 5.
|
||||
|
||||
## Phase 5 — Local de Bruijn graph and unitig construction
|
||||
|
||||
Within each partition, build a **local de Bruijn graph** from the valid kmer set and compute its unitigs. All operations are local to the partition — no cross-partition communication.
|
||||
|
||||
```
|
||||
valid kmers → HashSet<u64>
|
||||
|
||||
for each kmer K:
|
||||
out_degree = |{K[1:]+b | b ∈ {A,C,G,T}} ∩ HashSet|
|
||||
in_degree = |{b+K[:-1] | b ∈ {A,C,G,T}} ∩ HashSet|
|
||||
|
||||
internal node ↔ in_degree=1 AND out_degree=1
|
||||
branching / dead-end → unitig start or end
|
||||
```
|
||||
|
||||
Traverse non-branching paths to assemble unitigs. Kmers whose neighbours fall in other partitions appear as dead ends locally — they terminate the unitig. The result: **each kmer appears in exactly one unitig** within the partition.
|
||||
|
||||
The partition size (controlled by p) must be calibrated so that the HashSet fits in RAM during this phase.
|
||||
|
||||
Output: `unitigs.bin` — the permanent evidence structure for the partition. Each kmer in the partition appears at exactly one (unitig_id, offset) location.
|
||||
|
||||
**Scope of local unitigs:** these are unitigs of the partition's local de Bruijn graph, not global unitigs. A kmer whose k-1 successor or predecessor falls in another partition appears as a dead end locally and terminates the unitig. This does not affect correctness of verification but means partition-local unitigs cannot be directly reused for global assembly.
|
||||
|
||||
## Phase 6 — MPHF construction and index finalisation
|
||||
|
||||
`build_index_layer` is called per partition (in parallel via `build_layers`) with the following parameters sourced from `IndexConfig`:
|
||||
|
||||
- `block_bits` — from `IndexConfig::block_bits`; controls the `.idx` block size (2^block_bits unitig chunks per block) for exact evidence
|
||||
- `evidence` — `EvidenceKind::Exact` or `EvidenceKind::Approx { b, z }`; propagated unchanged from `IndexConfig::evidence`
|
||||
- `min_ab` / `max_ab` — abundance bounds applied before graph construction
|
||||
- `with_counts` — whether to store kmer counts alongside set membership
|
||||
|
||||
**Abundance filtering:** when `min_ab > 1` or `max_ab.is_some()`, the provisional `mphf1.bin` and `counts1.bin` produced in phase 3 are memory-mapped. Each canonical kmer is accepted only if its count in `counts1` satisfies the bounds. If either file is absent, filtering is skipped (all kmers accepted).
|
||||
|
||||
```
|
||||
for each kmer in dereplicated super-kmer:
|
||||
ab = counts1[mphf1.index(kmer.raw())]
|
||||
if ab < min_ab || ab > max_ab: skip
|
||||
graph.push(kmer)
|
||||
```
|
||||
|
||||
**Graph build and unitig write:**
|
||||
|
||||
The surviving kmers are fed into `GraphDeBruijn`, which computes degrees and yields unitigs. Unitigs are written to `layer_0/unitigs.bin` via a `UnitigFileWriter`.
|
||||
|
||||
**MPHF and evidence build:**
|
||||
|
||||
`Layer::build` (membership-only) or `Layer::<PersistentCompactIntMatrix>::build` (with counts) is called next. Internally, `MphfLayer::build` performs two passes:
|
||||
|
||||
1. **Pass 1 (parallel):** build `unitigs.bin.idx` (block size = 2^`block_bits`) then construct the MPHF from all canonical kmers in `unitigs.bin`; store to `mphf.bin`.
|
||||
2. **Pass 2 (sequential):** for each kmer in `unitigs.bin`, compute its slot and write `evidence.bin` (`chunk_id: 25 bits | rank: 7 bits` packed into a `u32`); also invoke the payload callback (`fill_slot`) to populate `counts/` if `with_counts`.
|
||||
|
||||
After `Layer::build` completes, `layer_meta.json` records `EvidenceKind::Exact`.
|
||||
|
||||
**Approximate evidence override:**
|
||||
|
||||
If `evidence` is `EvidenceKind::Approx { b, z }`, `build_approx_evidence` is called immediately after `Layer::build`. It overwrites the exact evidence bundle with `fingerprint.bin` (b-bit hash per slot) and rewrites `layer_meta.json` with `EvidenceKind::Approx { b, z }`. No `.idx` file is needed at query time in this mode.
|
||||
|
||||
```
|
||||
// Exact path → evidence.bin + unitigs.bin.idx + layer_meta.json(Exact)
|
||||
// Approx path → fingerprint.bin + layer_meta.json(Approx{b,z})
|
||||
// (evidence.bin left on disk but not used)
|
||||
```
|
||||
|
||||
**Partition metadata:**
|
||||
|
||||
After all layer files are written, `PartitionMeta { n_layers: 1 }` is serialised to `index/meta.json` inside the partition directory. This file is required by `LayeredMap::open` for subsequent merge operations.
|
||||
|
||||
**File layout per partition after phase 6:**
|
||||
|
||||
```
|
||||
part_XXXXX/
|
||||
index/
|
||||
meta.json ← PartitionMeta { n_layers: 1 }
|
||||
layer_0/
|
||||
unitigs.bin ← permanent evidence (all modes)
|
||||
unitigs.bin.idx ← block index (exact mode only)
|
||||
mphf.bin ← MPHF
|
||||
evidence.bin ← exact evidence (exact mode)
|
||||
fingerprint.bin ← b-bit fingerprints (approx mode)
|
||||
layer_meta.json ← EvidenceKind tag
|
||||
counts/ ← PersistentCompactIntMatrix (with_counts only)
|
||||
```
|
||||
|
||||
**Cleanup:** unless `--keep-intermediate` is set, `remove_build_artifacts` deletes `dereplicated.skmer.zst`, `mphf1.bin`, and `counts1.bin` after all partitions are indexed.
|
||||
|
||||
See [obilayeredmap](obilayeredmap.md) and [MPHF selection](mphf.md) for data structure details.
|
||||
|
||||
**Query path (exact evidence):**
|
||||
|
||||
```
|
||||
query kmer q
|
||||
→ canonical_minimizer(q) → hash → PART → part_XXXXX/
|
||||
→ MPHF(q) → slot s
|
||||
→ evidence[s] = (chunk_id, rank)
|
||||
→ read k nucleotides at rank in unitigs[chunk_id] → compare with q
|
||||
→ match : return payload[s] ← exact hit
|
||||
→ no match: kmer absent ← MPHF collision on absent kmer
|
||||
```
|
||||
|
||||
**Query path (approximate evidence):**
|
||||
|
||||
```
|
||||
query kmer q
|
||||
→ MPHF(q) → slot s
|
||||
→ fingerprint[s] matches seq_hash(q)?
|
||||
→ yes : probable hit (FP rate = 1/2^b per kmer, 1/2^(b·z) per z-window)
|
||||
→ no : kmer absent
|
||||
```
|
||||
@@ -0,0 +1,19 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/pipeline.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikpartitionner/src/partition.rs` — estimation des paramètres (phase 0)
|
||||
- `obiskbuilder/src/iter.rs` — scatter : filtre entropie, extraction superkmers, routage partition (phase 1)
|
||||
- `obikpartitionner/src/filter.rs` — déduplication bucket-sort (phase 2)
|
||||
- `obikpartitionner/src/kmer_sort.rs` — tri externe + agrégation de comptages (phase 3)
|
||||
- `obidebruinj/src/debruijn.rs` — graphe De Bruijn, extraction des unitigs (phase 5)
|
||||
- `obikpartitionner/src/index_layer.rs` — construction MPHF + évidence (phase 6), paramètre `block_bits`
|
||||
- `obikindex/src/index.rs` — `build_layers()`, `dereplicate_and_count()`
|
||||
|
||||
## Notes
|
||||
|
||||
RISQUE DE DÉRIVE modéré. Vérifier :
|
||||
- Phase 6 : la doc mentionne-t-elle le filtre d'abondance (`min_ab`, `max_ab`) ?
|
||||
- Phase 6 : `block_bits` passé à `build_index_layer` depuis `IndexConfig`
|
||||
- Phase 6 : dispatch exact/approx selon `EvidenceKind` dans `build_index_layer`
|
||||
@@ -0,0 +1,234 @@
|
||||
# `select` — column projection and aggregation
|
||||
|
||||
`select` transforms an index by operating on its **genome columns**: projecting a
|
||||
subset of columns, aggregating groups of genomes into synthetic columns, or both.
|
||||
It is the column-axis counterpart of `filter` (row-axis operations).
|
||||
|
||||
Following relational algebra conventions:
|
||||
|
||||
| Command | Relational operation | Axis |
|
||||
|----------|---------------------|----------|
|
||||
| `filter` | σ — selection | rows (k-mers) |
|
||||
| `select` | π — projection | columns (genomes) |
|
||||
|
||||
The two commands compose naturally: run `filter` first to restrict the kmer set,
|
||||
then `select` to reshape the genome columns.
|
||||
|
||||
`select` never changes the kmer set. The MPHF and `unitigs.bin` of each layer
|
||||
are preserved unchanged; only the data matrices are rewritten.
|
||||
|
||||
---
|
||||
|
||||
## Synopsis
|
||||
|
||||
```sh
|
||||
obikmer select <input-index>
|
||||
{ --output <dir> | --in-place }
|
||||
[--group <name>:<pred> ...]
|
||||
[--group-op <name>:<op> ...]
|
||||
[--aggregate-by <key> ]
|
||||
[--aggregate-op <op> ]
|
||||
[--select <col1,col2,...> ]
|
||||
[--presence-threshold <N> ]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Output destination
|
||||
|
||||
Exactly one of `--output` or `--in-place` must be specified.
|
||||
|
||||
**`--output <dir>`** — writes a new index to `<dir>`. The source index is
|
||||
unchanged. The MPHF and unitig files are copied; only the data matrices are
|
||||
rewritten with the new column layout.
|
||||
|
||||
**`--in-place`** — rewrites the data matrices of the source index directly.
|
||||
Removed or replaced columns are lost. The operation writes to temporary files
|
||||
first, then renames atomically, so an interrupted run leaves the index intact.
|
||||
|
||||
---
|
||||
|
||||
## Defining output columns
|
||||
|
||||
### Named groups — `--group`
|
||||
|
||||
```
|
||||
--group <name>:<pred>
|
||||
```
|
||||
|
||||
Defines a named group of genomes using the same predicate syntax as `filter`.
|
||||
Repeatable; a genome can belong to several groups.
|
||||
|
||||
```sh
|
||||
--group "pub:species=Betula_pubescens"
|
||||
--group "nan:species=Betula_nana"
|
||||
```
|
||||
|
||||
### Per-group operator — `--group-op`
|
||||
|
||||
```
|
||||
--group-op <name>:<op>
|
||||
```
|
||||
|
||||
Assigns an aggregation operator to a named group. Optional; if absent, the
|
||||
default operator applies (see below).
|
||||
|
||||
```sh
|
||||
--group-op "pub:any"
|
||||
--group-op "nan:all"
|
||||
```
|
||||
|
||||
### Shorthand — `--aggregate-by` / `--aggregate-op`
|
||||
|
||||
`--aggregate-by <key>` automatically creates one group per unique value of the
|
||||
metadata key `<key>`. Equivalent to one `--group <val>:<key>=<val>` per distinct
|
||||
value. `--aggregate-op <op>` sets the operator for all auto-generated groups.
|
||||
|
||||
`--aggregate-by` and `--group` are mutually exclusive.
|
||||
|
||||
### Column selection and ordering — `--select`
|
||||
|
||||
```
|
||||
--select col1,col2,...
|
||||
```
|
||||
|
||||
Lists the output columns in order. Each element is either a group name (defined
|
||||
by `--group` or generated by `--aggregate-by`) or a genome label from the source
|
||||
index (pass-through, no aggregation).
|
||||
|
||||
**Default when `--select` is absent:**
|
||||
all defined groups in declaration order (for `--group`), or all generated groups
|
||||
in metadata-value order (for `--aggregate-by`). Individual genomes not in any
|
||||
group are excluded unless named explicitly.
|
||||
|
||||
**When neither `--group` nor `--aggregate-by` is specified:**
|
||||
`--select` can still reference genome labels for pure column projection (no
|
||||
aggregation). If `--select` is also absent, all genomes are output unchanged
|
||||
(identity transform — useful combined with row filtering via a prior `filter`
|
||||
run).
|
||||
|
||||
---
|
||||
|
||||
## Aggregation operators
|
||||
|
||||
| Operator | Input | Output | Semantics |
|
||||
|----------|-------------|----------|-----------|
|
||||
| `any` | pres / count | presence | 1 if ≥ 1 genome in group carries the k-mer |
|
||||
| `all` | pres / count | presence | 1 if every genome in group carries the k-mer |
|
||||
| `none` | pres / count | presence | 1 if no genome in group carries the k-mer |
|
||||
| `sum` | count | count | sum of counts across the group |
|
||||
| `min` | count | count | minimum count |
|
||||
| `max` | count | count | maximum count |
|
||||
|
||||
**Default operator:**
|
||||
- Presence index: `any`
|
||||
- Count index: `sum`
|
||||
|
||||
Logical operators (`any`/`all`/`none`) on a count index use
|
||||
`--presence-threshold N` (default 0): a genome "carries" the k-mer if its count
|
||||
is > N.
|
||||
|
||||
**Output index type:**
|
||||
- If the source is a presence index, the output is always a presence index.
|
||||
- If the source is a count index and every output column uses a logical operator
|
||||
or is a pass-through from a presence source, the output is a presence index.
|
||||
- Otherwise (at least one arithmetic operator on a count source), the output is
|
||||
a count index.
|
||||
|
||||
---
|
||||
|
||||
## Behaviour for edge cases
|
||||
|
||||
| Situation | Behaviour |
|
||||
|-----------|-----------|
|
||||
| Genome missing the metadata key in `--aggregate-by` | genome ignored (no `NA` group) |
|
||||
| Genome in multiple groups | contributes independently to each |
|
||||
| `--group-op` references undefined group | error |
|
||||
| `--select` element is neither group name nor genome label | error |
|
||||
| `--output` and `--in-place` both specified | error |
|
||||
| Neither `--output` nor `--in-place` | error |
|
||||
| Group with zero matching genomes | column is all-zeros (or all-ones for `none`) |
|
||||
|
||||
---
|
||||
|
||||
## Examples
|
||||
|
||||
### Aggregate by metadata group, default operators
|
||||
|
||||
```sh
|
||||
obikmer select myindex --output out --aggregate-by group
|
||||
# one column per unique value of "group"; presence→any, count→sum
|
||||
```
|
||||
|
||||
### Named groups with different operators
|
||||
|
||||
```sh
|
||||
obikmer select myindex --output out \
|
||||
--group "pub:species=Betula_pubescens" \
|
||||
--group "nan:species=Betula_nana" \
|
||||
--group-op "pub:any" \
|
||||
--group-op "nan:all" \
|
||||
--select "pub,nan"
|
||||
```
|
||||
|
||||
### Mix aggregated group and individual genome
|
||||
|
||||
```sh
|
||||
obikmer select myindex --output out \
|
||||
--group "A:group=A" \
|
||||
--select "A,Betula_nana--IGA-24-39"
|
||||
```
|
||||
|
||||
### Pure column projection (no aggregation)
|
||||
|
||||
```sh
|
||||
obikmer select myindex --output out \
|
||||
--select "Betula_nana--TROM-V-149986,Betula_nana--AG-P04-25-01"
|
||||
```
|
||||
|
||||
### In-place: keep only group A
|
||||
|
||||
```sh
|
||||
obikmer select myindex --in-place --group "A:group=A" --select "A"
|
||||
```
|
||||
|
||||
### Compose with filter
|
||||
|
||||
```sh
|
||||
# Step 1: keep only B. nana-specific k-mers
|
||||
obikmer filter myindex --output filtered \
|
||||
--ingroup "species=Betula_nana" --outgroup "*"
|
||||
|
||||
# Step 2: aggregate genome columns by collection site
|
||||
obikmer select filtered --output final --aggregate-by site
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation notes
|
||||
|
||||
`select` does not rebuild the MPHF. The 256 partitions are processed in parallel
|
||||
(rayon), each writing its output independently; results require no synchronisation
|
||||
because every partition owns a distinct set of files.
|
||||
|
||||
For each layer in each partition:
|
||||
|
||||
1. The slot count `n` is read by opening the source data matrix.
|
||||
2. A new data matrix is built with M columns (M = number of output columns).
|
||||
3. For each slot `s` in `0..n`:
|
||||
- `old_row = matrix.fill_row(s)` — reads the original `N`-column row without allocating.
|
||||
- For each output column `j`:
|
||||
- `new_row[j] = aggregate(op, old_row[group_indices])`.
|
||||
- Pass-through columns are represented as single-element groups with the
|
||||
default operator (`any` for presence, `sum` for count) — same code path.
|
||||
- The new row is written slot by slot into each column builder.
|
||||
4. All plain files in the source layer directory (`mphf.bin`, `unitigs.bin`,
|
||||
evidence files, `layer_meta.json`) are copied verbatim; only the `presence/`
|
||||
or `counts/` subdirectory is rewritten.
|
||||
5. `index.meta` is rewritten with the new genome list and updated `with_counts`.
|
||||
|
||||
**`--in-place` write strategy:** new data is written to a temporary sibling
|
||||
directory (`presence_new/` or `counts_new/`); on success the old directory is
|
||||
removed and the temporary one is renamed into place. An interrupted run leaves
|
||||
at most one stale `*_new/` directory; the original data is intact until the
|
||||
rename step.
|
||||
@@ -0,0 +1,136 @@
|
||||
# On-disk index layout
|
||||
|
||||
## Directory tree
|
||||
|
||||
```
|
||||
<index_root>/
|
||||
index.meta ← JSON: IndexMeta
|
||||
scatter.done ← sentinel: scatter phase complete
|
||||
count.done ← sentinel: dereplicate + count complete
|
||||
index.done ← sentinel: MPHF index fully built
|
||||
spectrums/
|
||||
<label>.json ← kmer frequency spectrum per genome
|
||||
partitions/
|
||||
part_00000/ ← one dir per partition (zero-padded 5 digits, 0..2^n_bits−1)
|
||||
index/
|
||||
meta.json ← PartitionMeta { n_layers }
|
||||
layer_0/
|
||||
unitigs.bin ← binary unitig sequences (2-bit packed)
|
||||
unitigs.bin.idx ← block-sampled offset index (exact evidence only)
|
||||
mphf.bin ← serialised PtrHash MPHF
|
||||
layer_meta.json ← LayerMeta { evidence: EvidenceKind }
|
||||
evidence.bin ← chunk_id:rank per MPHF slot (Exact only)
|
||||
fingerprint.bin ← b-bit fingerprints per MPHF slot (Approx only)
|
||||
counts/ ← PersistentCompactIntMatrix (if with_counts=true)
|
||||
presence/ ← PersistentBitMatrix (if presence mode, merge)
|
||||
layer_1/ ← added by merge; same structure as layer_0
|
||||
layer_2/ …
|
||||
part_00001/ …
|
||||
```
|
||||
|
||||
## State machine (sentinels)
|
||||
|
||||
The sentinels are touched atomically at the end of each pipeline stage.
|
||||
A partial run (e.g. scatter interrupted) leaves no sentinel; the state is
|
||||
detected as the lowest sentinel present.
|
||||
|
||||
| State | Sentinel present | Meaning |
|
||||
|---|---|---|
|
||||
| `Empty` | — | `index.meta` exists; scatter not started or interrupted |
|
||||
| `Scattered` | `scatter.done` | All super-kmers routed to partition files |
|
||||
| `Counted` | `count.done` | Partitions dereplicated; `spectrums/` written |
|
||||
| `Indexed` | `index.done` | All MPHF layers built; index ready for queries |
|
||||
|
||||
## index.meta (IndexMeta)
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"config": {
|
||||
"kmer_size": 31,
|
||||
"minimizer_size": 11,
|
||||
"n_bits": 8,
|
||||
"with_counts": false,
|
||||
"evidence": "Exact",
|
||||
"block_bits": 0
|
||||
},
|
||||
"genomes": [
|
||||
{ "label": "genome_A", "meta": { "species": "Homo sapiens" } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`n_bits` determines the partition count: `2^n_bits` directories under `partitions/`.
|
||||
|
||||
`evidence` is either the string `"Exact"` or `{"Approx": {"b": 8, "z": 1}}`.
|
||||
|
||||
`block_bits` controls the `.idx` granularity: one offset entry every `2^block_bits`
|
||||
chunks. `block_bits=0` stores one entry per chunk (O(1) random access, largest `.idx`).
|
||||
|
||||
`GenomeInfo.meta` is a free-form string→string map for categorical metadata (e.g.
|
||||
taxonomy, sample origin). It is optional; defaults to empty.
|
||||
|
||||
## Layer files
|
||||
|
||||
### unitigs.bin
|
||||
|
||||
2-bit packed binary unitig sequences. Each record: 1 byte `seql_minus_k`
|
||||
(nucleotide length − k), followed by `ceil((seql_minus_k + k) / 4)` bytes of
|
||||
packed sequence. Long unitigs are transparently split into overlapping chunks
|
||||
(k−1 nucleotide overlap) so no k-mer crosses a chunk boundary.
|
||||
|
||||
### unitigs.bin.idx (Exact only)
|
||||
|
||||
Magic `UIX3`, little-endian header: `block_bits` (u32), `n_unitigs` (u32),
|
||||
`n_kmers` (u64), then `ceil(n_unitigs / 2^block_bits) + 1` byte-offset entries
|
||||
(u32 each, last entry is a sentinel past-end offset). Absent for Approx layers.
|
||||
|
||||
### mphf.bin
|
||||
|
||||
PtrHash MPHF serialised with epserde. Maps canonical kmer (u64, left-aligned
|
||||
2-bit) to a slot index in `[0, n_kmers)`.
|
||||
|
||||
### layer_meta.json (LayerMeta)
|
||||
|
||||
```json
|
||||
{ "evidence": { "type": "exact" } }
|
||||
```
|
||||
or
|
||||
```json
|
||||
{ "evidence": { "type": "approx", "b": 8, "z": 1 } }
|
||||
```
|
||||
|
||||
### evidence.bin (Exact)
|
||||
|
||||
One `(chunk_id: u32, rank: u8)` record per MPHF slot, packed. Used to verify
|
||||
that the kmer mapped to a slot is actually present: `unitigs.bin[chunk_id][rank]`
|
||||
is re-read and compared against the query.
|
||||
|
||||
### fingerprint.bin (Approx)
|
||||
|
||||
`b`-bit fingerprint per MPHF slot derived from the kmer's sequence hash.
|
||||
False-positive rate per query ≈ `1/2^b`. With Findere parameter `z ≥ 2`,
|
||||
`z` consecutive k-mers must all match, reducing the effective FP rate to
|
||||
approximately `W / 2^(b·z)` per read of length `L`
|
||||
(where `W = L − k − z + 2`).
|
||||
|
||||
### counts/ (PersistentCompactIntMatrix)
|
||||
|
||||
Present when `with_counts=true`. One column per genome; each row holds the
|
||||
per-genome k-mer count for the corresponding MPHF slot. Appended column-by-column
|
||||
during indexing and merge.
|
||||
|
||||
### presence/ (PersistentBitMatrix)
|
||||
|
||||
Present when the layer was built in presence/absence mode (merge path).
|
||||
One bit per genome per MPHF slot. Written during merge; never present on a
|
||||
freshly indexed single-genome layer.
|
||||
|
||||
## meta.json (PartitionMeta)
|
||||
|
||||
```json
|
||||
{ "n_layers": 2 }
|
||||
```
|
||||
|
||||
Records how many `layer_N/` directories exist under `index/`. Incremented by
|
||||
each merge that adds a layer.
|
||||
@@ -0,0 +1,18 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/storage.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikindex/src/meta.rs` — IndexMeta, IndexConfig (version, config, genomes)
|
||||
- `obikindex/src/index.rs` — layout sur disque : partitions/, index.meta
|
||||
- `obilayeredmap/src/meta.rs` — LayerMeta (evidence kind), PartitionMeta (n_layers)
|
||||
- `obiskio/src/unitig_index.rs` — fichiers unitigs.bin + unitigs.bin.idx
|
||||
|
||||
## Notes
|
||||
|
||||
FORT RISQUE DE DÉRIVE. Nombreux champs ajoutés :
|
||||
- `IndexConfig` : champs `evidence` (EvidenceKind) et `block_bits` ajoutés
|
||||
- Nouveau fichier `fingerprint.bin` pour l'évidence approximative
|
||||
- `LayerMeta` / `layer_meta.json` introduit pour stocker EvidenceKind par layer
|
||||
- Structure du répertoire layer : `evidence.bin` vs `fingerprint.bin` selon le mode
|
||||
Mettre à jour le schéma de layout sur disque en conséquence.
|
||||
@@ -0,0 +1,171 @@
|
||||
# SuperKmer — implementation
|
||||
|
||||
## Memory layout
|
||||
|
||||
`SuperKmer` holds two separate fields:
|
||||
|
||||
```rust
|
||||
pub struct SuperKmer {
|
||||
pub(crate) count: u32,
|
||||
pub(crate) inner: PackedSeq,
|
||||
}
|
||||
```
|
||||
|
||||
`PackedSeq` stores a 2-bit packed DNA sequence as a heap-allocated `Box<[u8]>` plus a `tail: u8` field:
|
||||
|
||||
| Field | Type | Role |
|
||||
|-------|------|------|
|
||||
| `tail` | `u8` | Number of valid nucleotides in the last byte: 0 encodes 4, 1–3 are identity |
|
||||
| `seq` | `Box<[u8]>` | 2-bit packed bytes, nucleotide 0 at bits 7–6 of `seq[0]` |
|
||||
|
||||
Nucleotide length is recovered without storing it explicitly:
|
||||
|
||||
```text
|
||||
seql = (seq.len() - 1) * 4 + tail_count(tail)
|
||||
```
|
||||
|
||||
There is no packed header word — `count` and the sequence live in separate fields.
|
||||
|
||||
The on-disk binary format (produced by `write_to_binary`) is:
|
||||
|
||||
```text
|
||||
[varint(count)] [u8: seql − k] [packed bytes…]
|
||||
```
|
||||
|
||||
`seql − k` fits in a `u8` when `n_kmers = seql − k + 1 ≤ MAX_KMERS_PER_CHUNK (= 256)`. If a super-kmer exceeds 256 kmers, `write_to_binary` splits it into overlapping chunks (k−1 nucleotide overlap, same count per chunk), each a self-contained record readable by `read_from_binary`.
|
||||
|
||||
The public accessors operate on the struct fields directly:
|
||||
|
||||
```rust
|
||||
fn seql(&self) -> usize { self.inner.seql() }
|
||||
fn count(&self) -> u32 { self.count }
|
||||
fn increment(&mut self) { self.count += 1; }
|
||||
fn add(&mut self, n: u32) { self.count += n; }
|
||||
fn set_count(&mut self, n: u32) { self.count = n; }
|
||||
```
|
||||
|
||||
## ASCII encoding and decoding
|
||||
|
||||
Two lookup tables handle ASCII ↔ 2-bit conversion:
|
||||
|
||||
- **`ENC: [u8; 32]`** — indexed by `b & 0x1F` (lower 5 bits of the ASCII byte). Maps A/a→0, C/c→1, G/g→2, T/t and U/u→3; ambiguous bases and unknowns silently map to 0 (A). 32 entries, fits entirely in L1 cache. Upper- and lowercase are handled identically.
|
||||
- **`DEC4: [u32; 256]`** — maps a packed byte (4 nucleotides) to 4 ASCII characters packed as a big-endian `u32`. 1 KB total, fits in L1 cache. One lookup per output byte yields 4 decoded characters.
|
||||
|
||||
Encoding 4 nucleotides into one byte:
|
||||
|
||||
```rust
|
||||
byte = ENC[c0 & 0x1F] << 6 | ENC[c1 & 0x1F] << 4 | ENC[c2 & 0x1F] << 2 | ENC[c3 & 0x1F]
|
||||
```
|
||||
|
||||
Decoding one byte into 4 ASCII characters:
|
||||
|
||||
```rust
|
||||
DEC4[byte].to_be_bytes() // [nuc0, nuc1, nuc2, nuc3] in ASCII
|
||||
```
|
||||
|
||||
## Reverse complement
|
||||
|
||||
The reverse complement is computed **in place** with zero allocation in two steps.
|
||||
|
||||
**Step 1 — byte swap with `REVCOMP4`.** A 256-byte lookup table `REVCOMP4` maps each byte (4 nucleotides) to its reverse complement. Bytes are swapped from the outside in, applying `REVCOMP4` to each:
|
||||
|
||||
```rust
|
||||
const fn revcomp4(x: u8) -> u8 {
|
||||
let x = !x; // complement all bases
|
||||
let x = (x >> 4) | (x << 4); // swap nibbles
|
||||
let x = ((x >> 2) & 0x33) | ((x & 0x33) << 2); // swap 2-bit groups
|
||||
x
|
||||
}
|
||||
```
|
||||
|
||||
`REVCOMP4` is 256 bytes (fits in L1 cache), computed at compile time. No endianness dependency — all operations are pure arithmetic on byte values.
|
||||
|
||||
**Step 2 — realignment.** After step 1, `padding = n × 8 − seql × 2` spurious bits (complements of the original padding A's) appear at the start of the array. They are flushed left using `BitSlice<u8, Msb0>::rotate_left(padding)` from the `bitvec` crate, which is SIMD-accelerated. The trailing `padding` bits are then zeroed:
|
||||
|
||||
```rust
|
||||
let seql = self.seql();
|
||||
shift = n * 8 - seql * 2 // number of padding bits
|
||||
bits.rotate_left(shift)
|
||||
bits[len - shift..].fill(false)
|
||||
```
|
||||
|
||||
`Msb0` ordering makes the bit layout hardware-independent.
|
||||
|
||||
!!! abstract "Algorithm — Super-kmer canonisation"
|
||||
```text
|
||||
procedure SuperKmerCanonical(seq, SEQL):
|
||||
for i ← 0 to SEQL − 1:
|
||||
fwd ← nucleotide(seq, i)
|
||||
rev ← complement(nucleotide(seq, SEQL − 1 − i))
|
||||
if fwd < rev: return seq -- forward is canonical
|
||||
if fwd > rev: return SuperKmerRevcomp(seq, SEQL) -- revcomp is canonical
|
||||
return seq -- palindrome: either orientation valid
|
||||
```
|
||||
|
||||
## Minimizer sliding window
|
||||
|
||||
Super-kmers are built by `SuperKmerIter` (crate `obiskbuilder`), which tracks the current minimizer with a **monotonic deque** (`Ring<MmerItem, 32>`) inside `RollingStat`, a rolling-window entropy and minimizer tracker.
|
||||
|
||||
Each deque entry stores:
|
||||
|
||||
| Field | Type | Purpose |
|
||||
|------------|-------|----------------------------------------------|
|
||||
| `position` | usize | 0-based start of this m-mer in the segment |
|
||||
| `canonical`| u64 | right-aligned canonical m-mer value (lex-min of fwd and rc); used as partition key |
|
||||
| `hash` | u64 | `hash_kmer(canonical << (64 − 2m))` — ordering key for random minimizer selection |
|
||||
|
||||
The hash uses the seeded splitmix64 finalizer (`mix64(raw ^ 0x9e3779b97f4a7c15)`), the same function as `kmer::hash_kmer`.
|
||||
|
||||
On each new nucleotide, once the window is full, the deque is updated:
|
||||
|
||||
!!! abstract "Algorithm — minimizer deque update"
|
||||
```text
|
||||
procedure UpdateMinimizer(deque, position, canonical, hash, k, received):
|
||||
-- pop dominated entries from the back
|
||||
while deque.back.hash ≥ hash:
|
||||
deque.pop_back()
|
||||
deque.push_back({position, canonical, hash})
|
||||
|
||||
-- evict expired entries from the front
|
||||
while deque.front.position + k < received:
|
||||
deque.pop_front()
|
||||
```
|
||||
|
||||
The front of the deque is always the current minimizer. Because the deque is maintained in strictly increasing hash order, each entry is popped at most once — O(1) amortized per nucleotide.
|
||||
|
||||
A super-kmer boundary is emitted when the minimizer changes: `current_minimizer != prev_minimizer`. `SuperKmerIter` also emits a boundary when:
|
||||
|
||||
- entropy of the current k-mer falls at or below the threshold θ (cursor retreated by k−1)
|
||||
- super-kmer length reaches 256 nucleotides (cursor retreated by k)
|
||||
|
||||
## Kmer extraction
|
||||
|
||||
A k-mer is extracted from a super-kmer with `SuperKmer::kmer(i)`, which delegates to `PackedSeq::extract::<KLen>(i)` and returns a `Kmer` — a left-aligned `u64` newtype (see [Kmer implementation](kmer.md)):
|
||||
|
||||
```rust
|
||||
pub fn kmer(&self, i: usize) -> Result<Kmer, KmerError>
|
||||
```
|
||||
|
||||
The bit slice `seq[i*2 .. (i+k)*2]` (Msb0 order) is loaded as a `u64` via `bitvec::load_be`, then left-shifted to produce the canonical left-aligned layout. One call — no loop, no allocation.
|
||||
|
||||
---
|
||||
|
||||
!!! abstract "Algorithm — Super-kmer reverse complement"
|
||||
```text
|
||||
procedure SuperKmerRevcomp(seq, SEQL):
|
||||
seql ← nucleotide length
|
||||
n ← ⌈seql / 4⌉ -- number of bytes
|
||||
shift ← n × 8 − seql × 2 -- padding bits to flush
|
||||
|
||||
-- step 1: swap bytes outside-in, applying REVCOMP4 to each (256-byte L1 table)
|
||||
lo ← 0 ; hi ← n − 1
|
||||
while lo < hi:
|
||||
seq[lo], seq[hi] ← REVCOMP4[seq[hi]], REVCOMP4[seq[lo]]
|
||||
lo ← lo + 1 ; hi ← hi − 1
|
||||
if lo == hi: seq[lo] ← REVCOMP4[seq[lo]]
|
||||
|
||||
-- step 2: left-rotate entire bit array by shift, zero trailing bits (SIMD via bitvec)
|
||||
if shift > 0:
|
||||
bits.rotate_left(shift)
|
||||
bits[n×8 − shift .. n×8].fill(0)
|
||||
```
|
||||
@@ -0,0 +1,13 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/superkmer.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obikseq/src/superkmer.rs` — layout mémoire SuperKmer (header 32 bits + séquence byte-alignée), encodage ASCII, revcomp, deque minimiseur
|
||||
- `obiskbuilder/src/lib.rs` — fenêtre glissante monotone pour le maintien du minimiseur
|
||||
|
||||
## Notes
|
||||
|
||||
Document d'implémentation détaillé. Vérifier que le layout header (longueur, orientation,
|
||||
position minimiseur) n'a pas changé. La doc mentionne un revcomp SIMD — vérifier si c'est
|
||||
toujours le cas ou si l'implémentation est scalaire.
|
||||
@@ -0,0 +1,170 @@
|
||||
# Unitig-based MPHF evidence encoding
|
||||
|
||||
## Role of unitigs in the index
|
||||
|
||||
The MPHF maps each canonical kmer to an integer slot but provides no inverse: a slot index alone cannot reconstruct the kmer. The **evidence file** supplies this inverse: for each MPHF slot it stores a pointer into the unitig sequence file, from which k nucleotides can be extracted.
|
||||
|
||||
Unitigs are the natural compact representation: a run of L nucleotides encodes L − k + 1 consecutive canonical kmers. The entire kmer set of a partition is reconstructible from its unitig binary file.
|
||||
|
||||
---
|
||||
|
||||
## Binary file formats
|
||||
|
||||
### `unitigs.bin` — sequence chunks
|
||||
|
||||
A sequence of binary records. Each record:
|
||||
|
||||
```
|
||||
[u8: seql − k] [ceil(seql / 4) bytes: 2-bit packed nucleotides]
|
||||
```
|
||||
|
||||
- `seql − k` (0–255): nucleotide length minus k, so `seql = byte[0] + k` and `n_kmers = byte[0] + 1`.
|
||||
- Packed nucleotides: A=00, C=01, G=10, T=11, MSB-first within each byte; last byte zero-padded.
|
||||
- Byte count for packed sequence: `ceil(seql / 4)`.
|
||||
|
||||
Unitigs with more than `MAX_KMERS_PER_CHUNK = 256` k-mers are transparently split into overlapping chunks. Each chunk has at most 256 k-mers (= `seql − k + 1 ≤ 256`); consecutive chunks overlap by k−1 nucleotides so no kmer is lost:
|
||||
|
||||
```
|
||||
chunk 1: nucleotides [0, MAX_KMERS_PER_CHUNK + k − 2] (256 kmers)
|
||||
chunk 2: nucleotides [256, end] (remaining kmers)
|
||||
overlap: k−1 nucleotides shared between the two chunks
|
||||
```
|
||||
|
||||
### `unitigs.bin.idx` — block-sampled offset index
|
||||
|
||||
```
|
||||
magic : 4 bytes = "UIX3"
|
||||
block_bits: u32 LE — granularity parameter (0–31)
|
||||
n_unitigs : u32 LE — total number of chunks in unitigs.bin
|
||||
n_kmers : u64 LE — total number of kmers across all chunks
|
||||
offsets : [u32 LE] — byte offsets into unitigs.bin, one per 2^block_bits chunks + sentinel
|
||||
```
|
||||
|
||||
One offset entry is stored every `2^block_bits` chunks; the array is sentinel-terminated (last entry = file size). `DEFAULT_BLOCK_BITS = 0` stores one offset per chunk (exact table, no scan).
|
||||
|
||||
### `evidence.bin` — per-slot MPHF evidence
|
||||
|
||||
A flat array of u32 values, one per MPHF slot, no header:
|
||||
|
||||
```
|
||||
bits [31:7] = chunk_id (25 bits)
|
||||
bits [6:0] = rank (7 bits, 0–127)
|
||||
```
|
||||
|
||||
File size = `n_slots × 4` bytes. `chunk_id` is the 0-based index of the record in `unitigs.bin`; `rank` is the position of the canonical kmer within that chunk (counting only canonical kmers). Encoding: `raw = (chunk_id << 7) | (rank & 0x7F)`. Decoding: `chunk_id = raw >> 7`, `rank = raw & 0x7F`.
|
||||
|
||||
---
|
||||
|
||||
## Building and reading the index
|
||||
|
||||
### `build_unitig_idx(path, block_bits)`
|
||||
|
||||
Scans `unitigs.bin` sequentially: for each chunk at byte offset `offset`, if `chunk_count & mask == 0` (where `mask = (1 << block_bits) − 1`), appends `offset as u32` to `block_offsets`. After the scan, appends a sentinel (= total file size), then writes the `.idx` file. Called after the unitig file is fully written and closed.
|
||||
|
||||
### `open()`, `open_sequential()`, `open_direct_access()`
|
||||
|
||||
`UnitigFileReader` has three constructors:
|
||||
|
||||
- `open(path)` — smart default: if `unitigs.bin.idx` exists, delegates to `open_direct_access`; otherwise delegates to `open_sequential`. Prefer this in call sites that don't require one specific mode.
|
||||
- `open_sequential(path)` — never reads `.idx`. Sequential iterators only; `chunk_start(i)` falls back to an O(i) mmap scan rather than panicking.
|
||||
- `open_direct_access(path)` — requires `.idx` to be present. Enables O(1) or O(2^block_bits) `chunk_start(i)`, used by `verify_canonical_kmer` at query time.
|
||||
|
||||
`CanonicalKmerIter` — a clonable sequential iterator returned by `UnitigFileReader::iter_canonical_kmers()`. It holds an `Arc<Mmap>` so cloning resets the cursor to the start without reopening the file. This makes it usable with `par_bridge()` for parallel MPHF construction without random access.
|
||||
|
||||
### `chunk_start(i)` — access modes
|
||||
|
||||
When `.idx` is loaded (`open_direct_access`):
|
||||
|
||||
- `block_bits = 0`: single array lookup, O(1).
|
||||
- `block_bits > 0`: lookup block, then scan ≤ 2^block_bits records, O(2^block_bits).
|
||||
|
||||
When `.idx` is absent (`open_sequential`): `chunk_start(i)` performs an O(i) sequential mmap scan from offset 0. No panic — the function degrades gracefully. This degraded path is used by `find_strict()` on Approx layers (sequential scan of all canonical kmers).
|
||||
|
||||
### Decoding a kmer from slot `s`
|
||||
|
||||
```rust
|
||||
let (chunk_id, rank) = evidence.decode(s); // u32 → (chunk_id: u32, rank: u8)
|
||||
let kmer = unitigs.raw_kmer(chunk_id, rank); // 2-bit packed slice → left-aligned u64
|
||||
```
|
||||
|
||||
Two memory accesses: one 4-byte read from `evidence.bin`, one packed-bit extraction from `unitigs.bin` via the mmap. The retrieved sequence is already canonical (only canonical kmers are inserted into the De Bruijn graph).
|
||||
|
||||
---
|
||||
|
||||
## Field widths and capacity
|
||||
|
||||
| field | bits | range | capacity check (*B. nana*, 256 partitions) |
|
||||
|------------|------|---------------|---------------------------------------------|
|
||||
| `seql − k` | 8 | 0–255 | max `n_kmers` per chunk = 256 = `MAX_KMERS_PER_CHUNK` |
|
||||
| `rank` | 7 | 0–127 | observed max ~46 kmers/chunk; structural max k−m+1 = 21 |
|
||||
| `chunk_id` | 25 | 0–33 554 431 | avg U ≈ 275 k chunks/partition |
|
||||
|
||||
The rank field is 7 bits (max 127) even though chunks can contain up to 256 k-mers, because rank counts only canonical kmers within the chunk, and the canonical kmer count is at most half the total.
|
||||
|
||||
---
|
||||
|
||||
## Evidence bit-cost
|
||||
|
||||
Strategy B (chunk_id + rank) is the implemented strategy. For *B. nana* (k=31, 256 partitions, P ≈ 10.4 M unique kmers/partition, U ≈ 275 k chunks/partition, m_u ≈ 37.9 kmers/chunk):
|
||||
|
||||
| field | theoretical cost | value |
|
||||
|------------|-------------------------|---------|
|
||||
| chunk_id | ⌈log₂ U⌉ | 19 bits |
|
||||
| rank | ⌈log₂ m_u⌉ (≈ fixed) | 6 bits |
|
||||
| **stored** | aligned u32 | **32 bits/slot** |
|
||||
|
||||
The u32 layout is chosen for alignment and simplicity; no bit-addressing arithmetic is needed.
|
||||
|
||||
Comparison with strategy A (global nucleotide offset): `⌈log₂(P · (1 + (k−1)/m_u))⌉ = 25 bits`. Strategy A is theoretically 2 bits cheaper; strategy B's advantage is **locality** (decoding touches one chunk's cache lines) and a bounded, constant-width rank field independent of partition size.
|
||||
|
||||
---
|
||||
|
||||
## Unitig decomposition non-determinism
|
||||
|
||||
The unitig extraction from `GraphDeBruijn` is **not deterministic**: two runs on identical input can produce different unitig counts and sequences while covering exactly the same canonical kmer set.
|
||||
|
||||
The hash map (`hashbrown::HashMap` with `Xxh3Builder`) has run-dependent iteration order. The `start_iter` first pass emits every node where `can_extend_left` is false — this includes true dead-ends and branch points (nodes with ≥2 left neighbours). When a branch point is encountered before its upstream neighbours, it claims the downstream chain and those upstream neighbours later produce length-k degenerate unitigs. When upstream neighbours appear first, they extend through the branch point.
|
||||
|
||||
**Example** — fork topology (k = 31):
|
||||
|
||||
```
|
||||
A → B ← C
|
||||
↓
|
||||
D
|
||||
```
|
||||
|
||||
B has two left neighbours, so `can_extend_left = false`. Two valid tilings:
|
||||
|
||||
| iteration order | unitigs | count |
|
||||
|---|---|---|
|
||||
| A first | ABD, C | 2 |
|
||||
| B first | BD, A, C | 3 |
|
||||
|
||||
Both cover the same 4 canonical kmers. Pure cycles are unaffected: all cycle nodes have both extensions present, so none are emitted in the first pass; each cycle produces exactly one unitig regardless of entry point (only the cut point varies).
|
||||
|
||||
This non-determinism is benign for MPHF construction: the MPHF is built from the kmer set, which is identical across tilings.
|
||||
|
||||
---
|
||||
|
||||
## Partition-size tradeoff
|
||||
|
||||
Measured on *B. nana* (k=31, m=11), summing across all partitions:
|
||||
|
||||
| N partitions | m_u |
|
||||
|---|---|
|
||||
| 1 | 41.89 |
|
||||
| 16 | 38.19 |
|
||||
| 256 | 37.90 |
|
||||
| 1 024 | 37.89 |
|
||||
|
||||
`m_u` is set by De Bruijn graph topology (heterozygosity, repeats, sequencing errors), not partition count. The variation from 1 to 1024 partitions is under 10%; within 16–1024 it is under 1%. Unitigs provide ~3.1× nucleotide compaction over super-kmers at 256 partitions.
|
||||
|
||||
Evidence cost decreases by 1 bit/kmer with each doubling of partition count (via `log₂ U = log₂(P/m_u)`). The sequence storage term `2 · (1 + (k−1)/m_u) ≈ 3.6 bits/kmer` is approximately constant.
|
||||
|
||||
---
|
||||
|
||||
## Alternative: fingerprint evidence
|
||||
|
||||
`evidence.bin` can be replaced by `fingerprint.bin` at index build time (`--approx`) or after the fact (`reindex --approx`). The fingerprint stores b bits per MPHF slot (the low b bits of `kmer.seq_hash()`); verification becomes a single bitfield comparison instead of a unitig dereference. False-positive rate per k-mer query: 1/2^b. With the Findere z parameter, z consecutive k-mers must all match, reducing the effective window FP rate to 1/2^(b·z) while skipping z−1 of every z k-mers. No `.idx` file is written or read in approx mode.
|
||||
|
||||
See [Approximate evidence (Findere fingerprint)](evidence_elimination.md) for the full design and CLI parameters.
|
||||
@@ -0,0 +1,18 @@
|
||||
<!-- coverage sidecar — ne pas ajouter au nav mkdocs -->
|
||||
# Coverage: implementation/unitig_evidence.md
|
||||
|
||||
## Code couvert
|
||||
|
||||
- `obiskio/src/unitig_index.rs` — format unitigs.bin + unitigs.bin.idx, UnitigFileWriter, UnitigFileReader, build_unitig_idx(), DEFAULT_BLOCK_BITS=0, chemin chaud block_bits=0 dans chunk_start()
|
||||
- `obilayeredmap/src/evidence.rs` — encodage Evidence (chunk_id 25 bits | rank 7 bits), EvidenceWriter
|
||||
- `obidebruinj/src/debruijn.rs` — extraction unitigs, chunking à MAX_KMERS_PER_CHUNK
|
||||
|
||||
## Notes
|
||||
|
||||
FORT RISQUE DE DÉRIVE. Changements récents :
|
||||
- `DEFAULT_BLOCK_BITS` est passé de 6 à 0 (accès O(1) par défaut)
|
||||
- `block_bits` est maintenant un paramètre runtime de `build_unitig_idx()` et `UnitigFileWriter`
|
||||
- `chunk_start()` a un chemin chaud explicite pour block_bits=0 (accès tableau direct, 0 scan)
|
||||
- `open()` vs `open_sequential()` : distinction nouvelle, importante pour la compréhension du coût
|
||||
- `iter_unitigs()` ajouté comme alias public de `iter_chunks_sequential()`
|
||||
Mettre à jour la description du format .idx et le modèle de coût d'accès aléatoire.
|
||||
Reference in New Issue
Block a user