Files
obikmer/docmd/architecture/index_architecture.md
T

314 lines
12 KiB
Markdown
Raw Normal View History

# Kmer index architecture
## Fundamental invariant
A given canonical kmer belongs to **exactly one partition** and **exactly one layer** within that partition. This property makes all aggregation operations decomposable and parallelisable without coordination.
---
## Three-level hierarchy
```
KmerIndex (index.meta + KmerPartition)
├── partition_0/index/ one directory per minimiser bucket
│ ├── meta.json PartitionMeta { n_layers }
│ ├── layer_0/
│ │ ├── layer_meta.json LayerMeta { evidence: EvidenceKind }
│ │ ├── mphf.bin PtrHash MPHF
│ │ ├── unitigs.bin unitig spine (never overwritten)
│ │ ├── evidence.bin exact evidence (Exact only)
│ │ ├── unitigs.bin.idx block index (Exact only)
│ │ ├── fingerprint.bin fingerprints (Approx only)
│ │ ├── counts/ PersistentCompactIntMatrix (with_counts = true)
│ │ └── presence/ PersistentBitMatrix
│ └── layer_1/
│ └── ...
└── partition_1/index/
└── ...
```
**KmerIndex**: root entry point. Owns `IndexMeta` (written to `index.meta`) and a `KmerPartition` that routes canonical kmers to partition directories. All partition-level operations are dispatched in parallel via rayon.
**Partition directory**: one directory per minimiser bucket. `PartitionMeta` (stored as `meta.json`) records `n_layers`. Layers within a partition cover disjoint kmer sets.
**Layer directory**: one `MphfLayer` plus optional data stores. `LayerMeta` (stored as `layer_meta.json`) records which `EvidenceKind` was used. The MPHF and `unitigs.bin` are immutable once built; evidence files are the only part replaced by `reindex`.
---
## IndexConfig and IndexMeta
```rust
pub struct IndexConfig {
pub kmer_size: usize,
pub minimizer_size: usize,
pub n_bits: usize, // log2(n_partitions)
pub with_counts: bool,
pub evidence: EvidenceKind,
pub block_bits: u8, // .idx granularity: 2^block_bits unitigs/block; 0 = one entry per unitig
}
pub struct IndexMeta {
pub version: u32,
pub config: IndexConfig,
pub genomes: Vec<GenomeInfo>, // ordered; index = genome column number
}
pub struct GenomeInfo {
pub label: String,
pub meta: HashMap<String, String>, // arbitrary categorical metadata
}
```
`IndexMeta` is serialised as `index.meta` (JSON). It is the authority for the ordered list of genomes and for the parameters that govern all subsequent operations on the index.
---
## EvidenceKind
```rust
pub enum EvidenceKind {
Exact,
Approx { b: u8, z: u8 },
}
```
Controls which files are written per layer and which query path is taken:
| Variant | Files written | False-positive rate |
|---|---|---|
| `Exact` | `evidence.bin`, `unitigs.bin.idx` | 0 |
| `Approx { b, z }` | `fingerprint.bin` | ≈ W / 2^(b·z) per read (Findere) |
`EvidenceKind` is stored both in `IndexConfig` (index-wide default, updated by `reindex`) and in each `LayerMeta` (per-layer record of what was actually built).
---
## MphfLayer — autonomous kmer → slot mapping
```rust
pub struct MphfLayer {
mphf: PtrHash<>,
ev: LayerEvidence, // Exact { evidence, unitigs } | Approx { fingerprint }
n: usize,
}
```
`MphfLayer::find(kmer)` dispatches transparently to `find_exact` or `find_approx` based on the evidence loaded at `open` time (read from `layer_meta.json`). Returns `Some(slot)` only if the kmer is confirmed present; `None` for absent or out-of-range.
```
find_exact: slot = mphf(kmer); decode evidence → (chunk_id, rank); verify kmer in unitigs
find_approx: slot = mphf(kmer); check fingerprint[slot] == seq_hash(kmer)
```
`block_bits` controls the `.idx` file written alongside `evidence.bin`. At `block_bits = 0`, every unitig chunk has an index entry, giving O(1) random access; larger values trade access time for a smaller `.idx`.
The MPHF and `unitigs.bin` are never rebuilt by any post-build operation.
---
## Layer\<D\> — MPHF + data payload
```rust
pub struct Layer<D: LayerData = ()> {
mphf: MphfLayer,
data: D,
}
```
`D` selects the attached data payload:
| `D` | Data directory | `Item` returned by `query` |
|---|---|---|
| `()` | — | `()` (set membership only) |
| `PersistentCompactIntMatrix` | `counts/` | `Box<[u32]>` (counts per genome) |
| `PersistentBitMatrix` | `presence/` | `Box<[bool]>` (presence per genome) |
`Layer::query(kmer)` delegates to `MphfLayer::find`, then calls `data.read(slot)` if a slot is returned. Both exact and approximate evidence are handled transparently; the caller sees only `Option<Hit<D::Item>>`.
Build-time entry points:
```rust
Layer<()>::build(out_dir, block_bits) // set membership
Layer<PersistentCompactIntMatrix>::build(out_dir, block_bits, count_of)
Layer<PersistentBitMatrix>::build_presence(out_dir, block_bits, n_genomes, present_in)
Layer::<()>::build_evidence(layer_dir, kind, block_bits) // evidence only (reindex path)
```
---
## DataStore — slot-indexed data
`PersistentCompactIntMatrix` and `PersistentBitMatrix` are slot-indexed stores. They know nothing about kmers or MPHFs.
| Type | `Item` | Aggregation method | Use |
|---|---|---|---|
| `PersistentCompactIntMatrix` | `Box<[u32]>` | `sum() → Array1<u64>` | counts per genome per slot |
| `PersistentBitMatrix` | `Box<[bool]>` | `count_ones() → Array1<u64>` | presence per genome per slot |
---
## Aggregation traits — `obicompactvec::traits`
Three traits unify the aggregation API across all hierarchy levels.
```rust
trait ColumnWeights: Send + Sync {
fn col_weights(&self) -> Array1<u64>;
}
trait CountPartials: ColumnWeights {
fn partial_bray(&self) -> Array2<u64>;
fn partial_euclidean(&self) -> Array2<f64>;
fn partial_threshold_jaccard(&self, threshold: u32) -> (Array2<u64>, Array2<u64>);
fn partial_relfreq_bray(&self, global: &Array1<u64>) -> Array2<f64>;
fn partial_relfreq_euclidean(&self, global: &Array1<u64>) -> Array2<f64>;
fn partial_hellinger(&self, global: &Array1<u64>) -> Array2<f64>;
// provided finalisation methods with default impls
fn bray_dist_matrix(&self) -> Array2<f64> { }
fn relfreq_bray_dist_matrix(&self) -> Array2<f64> { }
// …
}
trait BitPartials: ColumnWeights {
fn partial_jaccard(&self) -> (Array2<u64>, Array2<u64>);
fn partial_hamming(&self) -> Array2<u64>;
// provided
fn jaccard_dist_matrix(&self) -> Array2<f64> { }
fn hamming_dist_matrix(&self) -> Array2<u64> { }
}
```
Leaf implementors:
| Type | Traits |
|---|---|
| `PersistentCompactIntMatrix` | `ColumnWeights`, `CountPartials` |
| `PersistentBitMatrix` | `ColumnWeights`, `BitPartials` |
---
## LayeredStore\<S\> — recursive aggregation wrapper
```rust
pub struct LayeredStore<S>(Vec<S>);
```
Three blanket impls propagate all traits up the hierarchy:
```rust
impl<S: ColumnWeights> ColumnWeights for LayeredStore<S> { }
impl<S: CountPartials> CountPartials for LayeredStore<S> { }
impl<S: BitPartials> BitPartials for LayeredStore<S> { }
```
This makes `LayeredStore<LayeredStore<PersistentCompactIntMatrix>>` automatically implement `CountPartials` — no separate `PartitionedStore` type is needed:
```
PersistentCompactIntMatrix leaf (one layer)
LayeredStore<PersistentCompactIntMatrix> one partition (layers are disjoint)
LayeredStore<LayeredStore<…>> whole index (partitions are independent)
```
Normalised metrics require global column sums — computed in a two-pass cascade:
```rust
// on LayeredStore<LayeredStore<PersistentCompactIntMatrix>>
fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
let global = self.col_weights(); // pass 1 — sums up hierarchy
let p = self.partial_relfreq_bray(&global); // pass 2 — global broadcast read-only
p.mapv(|v| 1.0 - v)
}
```
Because each kmer belongs to exactly one `(partition, layer)` pair, `col_weights()` has no double-counting across the hierarchy.
---
## Progressive aggregation principle
No level reaches two levels down. Each level sums contributions from the level immediately below:
```
PersistentCompactIntMatrix::col_weights() — one (partition, layer)
↓ Σ across layers
LayeredStore<PersistentCompactIntMatrix>::col_weights() — one partition
↓ Σ across partitions
LayeredStore<LayeredStore<…>>::col_weights() — global
```
The same cascade applies to every partial method.
---
## Multi-genome column invariant
After any merge, every layer in every partition has exactly `n_genomes` columns, where `n_genomes` is the current total in `index.meta`. This holds for both `PersistentCompactIntMatrix` and `PersistentBitMatrix`.
Maintained by three coordinated operations:
**Existing layers — column append.** `Layer::append_genome_column` appends one column to each existing layer. Slots matching the incoming genome receive its count or `true`; all other slots receive 0 or `false`.
**New layers — absent columns prepended.** When a new layer is created for kmers unique to the incoming genome, `n_existing_genomes` absent columns are prepended before the incoming genome's column, so the new layer immediately has the same column count as all other layers.
**First merge, Presence mode — `init_presence_matrix`.** The initial single-genome index has no `presence/` directory (presence is implicit). On the first merge, `Layer<()>::init_presence_matrix` materialises genome 0's presence column (all `true`) retroactively, raising the column count from 0 to 1 before appending column 1.
This invariant is the precondition for correct progressive aggregation: every level can blindly sum matrices from below because all matrices have the same shape.
---
## Query model
### Point query
```
minimiser(kmer) → partition p
for each layer l in p:
if let Some(slot) = MphfLayer_l.find(kmer):
return data_l.read(slot)
return None
```
O(n_layers) MPHF probes worst case; O(1) expected. The result comes from exactly one `(partition, layer)`.
### Aggregation
```
result = reduce(
for p in partitions: // parallel
for l in layers(p): // parallel
partial(data_p_l)
)
```
For normalised metrics, replace with the two-pass cascade.
---
## Parallelism model
| Level | Unit | Coordination |
|---|---|---|
| Across partitions | inner stores of `LayeredStore<LayeredStore<S>>` | none |
| Across layers within a partition | inner stores of `LayeredStore<S>` | none — disjoint kmer sets |
| Normalised pass 1 (`col_weights`) | per inner store | none — additive |
| Normalised pass 2 (partial) | per inner store | `global` broadcast read-only |
| Within a matrix (distance) | upper-triangle pair `(i,j)` | none — rayon `par_iter` |
---
## reindex — evidence conversion in place
`KmerIndex::reindex(target, block_bits)` converts every layer's evidence bundle to `target` without touching the MPHF or `unitigs.bin`:
- `→ Exact`: builds `evidence.bin` + `unitigs.bin.idx`; removes `fingerprint.bin`
- `→ Approx { b, z }`: builds `fingerprint.bin`; removes `evidence.bin` + `unitigs.bin.idx`
On success, `IndexConfig::evidence` and `IndexConfig::block_bits` are updated in `index.meta`. Each layer's `layer_meta.json` is also rewritten with the new `EvidenceKind`.
---
## estimate — parameter dry-run
`estimate` resolves approximate-evidence parameters (`z`, `b`, target FP rate) and prints the resulting effective kmer size and per-kmer / per-z-window false-positive rates without touching any index. Used to calibrate `Approx { b, z }` before building or reindexing.