Skip to content

# Kmer index architecture

## Fundamental invariant

A given canonical kmer belongs to exactly one partition and exactly one layer within that partition. This property makes all aggregation operations decomposable and parallelisable without coordination.

## Three-level hierarchy

    KmerIndex  (index.meta + KmerPartition)
    ├── partition_0/index/          one directory per minimiser bucket
    │   ├── meta.json               PartitionMeta { n_layers }
    │   ├── layer_0/
    │   │   ├── layer_meta.json     LayerMeta { evidence: EvidenceKind }
    │   │   ├── mphf.bin            PtrHash MPHF
    │   │   ├── unitigs.bin         unitig spine (never overwritten)
    │   │   ├── evidence.bin        exact evidence  (Exact only)
    │   │   ├── unitigs.bin.idx     block index     (Exact only)
    │   │   ├── fingerprint.bin     fingerprints    (Approx only)
    │   │   ├── counts/             PersistentCompactIntMatrix  (with_counts = true)
    │   │   └── presence/           PersistentBitMatrix
    │   └── layer_1/
    │       └── ...
    └── partition_1/index/
        └── ...

KmerIndex: root entry point. Owns `IndexMeta` (written to `index.meta`) and a `KmerPartition` that routes canonical kmers to partition directories. All partition-level operations are dispatched in parallel via rayon.

Partition directory: one directory per minimiser bucket. `PartitionMeta` (stored as `meta.json`) records `n_layers`. Layers within a partition cover disjoint kmer sets.

Layer directory: one `MphfLayer` plus optional data stores. `LayerMeta` (stored as `layer_meta.json`) records which `EvidenceKind` was used. The MPHF and `unitigs.bin` are immutable once built; evidence files are the only part replaced by `reindex`.

## IndexConfig and IndexMeta

    pub struct IndexConfig {
        pub kmer_size:      usize,
        pub minimizer_size: usize,
        pub n_bits:         usize,       // log2(n_partitions)
        pub with_counts:    bool,
        pub evidence:       EvidenceKind,
        pub block_bits:     u8,          // .idx granularity: 2^block_bits unitigs/block; 0 = one entry per unitig
    }

    pub struct IndexMeta {
        pub version: u32,
        pub config:  IndexConfig,
        pub genomes: Vec<GenomeInfo>,    // ordered; index = genome column number
    }

    pub struct GenomeInfo {
        pub label: String,
        pub meta:  HashMap<String, String>,  // arbitrary categorical metadata
    }

`IndexMeta` is serialised as `index.meta` (JSON). It is the authority for the ordered list of genomes and for the parameters that govern all subsequent operations on the index.

## EvidenceKind

    pub enum EvidenceKind {
        Exact,
        Approx { b: u8, z: u8 },
    }

Controls which files are written per layer and which query path is taken:

Variant | Files written | False-positive rate
--- | --- | ---
`Exact` | `evidence.bin`, `unitigs.bin.idx` | 0
`Approx { b, z }` | `fingerprint.bin` | ≈ W / 2^(b·z) per read (Findere)

`EvidenceKind` is stored both in `IndexConfig` (index-wide default, updated by `reindex`) and in each `LayerMeta` (per-layer record of what was actually built).

## MphfLayer — autonomous kmer → slot mapping

    pub struct MphfLayer {
        mphf: PtrHash<…>,
        ev:   LayerEvidence,   // Exact { evidence, unitigs } | Approx { fingerprint }
        n:    usize,
    }

`MphfLayer::find(kmer)` dispatches transparently to `find_exact` or `find_approx` based on the evidence loaded at `open` time (read from `layer_meta.json`). Returns `Some(slot)` only if the kmer is confirmed present; `None` for absent or out-of-range.

    find_exact:  slot = mphf(kmer); decode evidence → (chunk_id, rank); verify kmer in unitigs
    find_approx: slot = mphf(kmer); check fingerprint[slot] == seq_hash(kmer)

`block_bits` controls the `.idx` file written alongside `evidence.bin`. At `block_bits = 0`, every unitig chunk has an index entry, giving O(1) random access; larger values trade access time for a smaller `.idx`.

The MPHF and `unitigs.bin` are never rebuilt by any post-build operation.

## Layer\<D> — MPHF + data payload

    pub struct Layer<D: LayerData = ()> {
        mphf: MphfLayer,
        data: D,
    }

`D` selects the attached data payload:

`D` | Data directory | `Item` returned by `query`
--- | --- | ---
`()` | — | `()` (set membership only)
`PersistentCompactIntMatrix` | `counts/` | `Box<[u32]>` (counts per genome)
`PersistentBitMatrix` | `presence/` | `Box<[bool]>` (presence per genome)

`Layer::query(kmer)` delegates to `MphfLayer::find`, then calls `data.read(slot)` if a slot is returned. Both exact and approximate evidence are handled transparently; the caller sees only `Option<Hit<D::Item>>`.

Build-time entry points:

    Layer<()>::build(out_dir, block_bits)                         // set membership
    Layer<PersistentCompactIntMatrix>::build(out_dir, block_bits, count_of)
    Layer<PersistentBitMatrix>::build_presence(out_dir, block_bits, n_genomes, present_in)

    Layer::<()>::build_evidence(layer_dir, kind, block_bits)      // evidence only (reindex path)

## DataStore — slot-indexed data

`PersistentCompactIntMatrix` and `PersistentBitMatrix` are slot-indexed stores. They know nothing about kmers or MPHFs.

Type | `Item` | Aggregation method | Use
--- | --- | --- | ---
`PersistentCompactIntMatrix` | `Box<[u32]>` | `sum() → Array1<u64>` | counts per genome per slot
`PersistentBitMatrix` | `Box<[bool]>` | `count_ones() → Array1<u64>` | presence per genome per slot

## Aggregation traits — `obicompactvec::traits`

Three traits unify the aggregation API across all hierarchy levels.

    trait ColumnWeights: Send + Sync {
        fn col_weights(&self) -> Array1<u64>;
    }

    trait CountPartials: ColumnWeights {
        fn partial_bray(&self)                                       -> Array2<u64>;
        fn partial_euclidean(&self)                                  -> Array2<f64>;
        fn partial_threshold_jaccard(&self, threshold: u32)          -> (Array2<u64>, Array2<u64>);
        fn partial_relfreq_bray(&self, global: &Array1<u64>)         -> Array2<f64>;
        fn partial_relfreq_euclidean(&self, global: &Array1<u64>)    -> Array2<f64>;
        fn partial_hellinger(&self, global: &Array1<u64>)            -> Array2<f64>;
        // provided finalisation methods with default impls
        fn bray_dist_matrix(&self)                                   -> Array2<f64> { … }
        fn relfreq_bray_dist_matrix(&self)                           -> Array2<f64> { … }
        // …
    }

    trait BitPartials: ColumnWeights {
        fn partial_jaccard(&self)  -> (Array2<u64>, Array2<u64>);
        fn partial_hamming(&self)  -> Array2<u64>;
        // provided
        fn jaccard_dist_matrix(&self) -> Array2<f64> { … }
        fn hamming_dist_matrix(&self) -> Array2<u64> { … }
    }

Leaf implementors:

Type | Traits
--- | ---
`PersistentCompactIntMatrix` | `ColumnWeights`, `CountPartials`
`PersistentBitMatrix` | `ColumnWeights`, `BitPartials`

## LayeredStore\<S> — recursive aggregation wrapper

    pub struct LayeredStore<S>(Vec<S>);

Three blanket impls propagate all traits up the hierarchy:

    impl<S: ColumnWeights> ColumnWeights for LayeredStore<S> { … }
    impl<S: CountPartials> CountPartials  for LayeredStore<S> { … }
    impl<S: BitPartials>   BitPartials    for LayeredStore<S> { … }

This makes `LayeredStore<LayeredStore<PersistentCompactIntMatrix>>` automatically implement `CountPartials` — no separate `PartitionedStore` type is needed:

    PersistentCompactIntMatrix                         leaf (one layer)
    LayeredStore<PersistentCompactIntMatrix>           one partition (layers are disjoint)
    LayeredStore<LayeredStore<…>>                      whole index   (partitions are independent)

Normalised metrics require global column sums — computed in a two-pass cascade:

    // on LayeredStore<LayeredStore<PersistentCompactIntMatrix>>
    fn relfreq_bray_dist_matrix(&self) -> Array2<f64> {
        let global = self.col_weights();             // pass 1 — sums up hierarchy
        let p      = self.partial_relfreq_bray(&global); // pass 2 — global broadcast read-only
        p.mapv(|v| 1.0 - v)
    }

Because each kmer belongs to exactly one `(partition, layer)` pair, `col_weights()` has no double-counting across the hierarchy.

## Progressive aggregation principle

No level reaches two levels down. Each level sums contributions from the level immediately below:

    PersistentCompactIntMatrix::col_weights()              — one (partition, layer)
            ↓ Σ across layers
    LayeredStore<PersistentCompactIntMatrix>::col_weights() — one partition
            ↓ Σ across partitions
    LayeredStore<LayeredStore<…>>::col_weights()            — global

The same cascade applies to every partial method.

## Multi-genome column invariant

After any merge, every layer in every partition has exactly `n_genomes` columns, where `n_genomes` is the current total in `index.meta`. This holds for both `PersistentCompactIntMatrix` and `PersistentBitMatrix`.

Maintained by three coordinated operations:

Existing layers — column append. `Layer::append_genome_column` appends one column to each existing layer. Slots matching the incoming genome receive its count or `true`; all other slots receive 0 or `false`.

New layers — absent columns prepended. When a new layer is created for kmers unique to the incoming genome, `n_existing_genomes` absent columns are prepended before the incoming genome's column, so the new layer immediately has the same column count as all other layers.

First merge, Presence mode — `init_presence_matrix`. The initial single-genome index has no `presence/` directory (presence is implicit). On the first merge, `Layer<()>::init_presence_matrix` materialises genome 0's presence column (all `true`) retroactively, raising the column count from 0 to 1 before appending column 1.

This invariant is the precondition for correct progressive aggregation: every level can blindly sum matrices from below because all matrices have the same shape.

## Query model

### Point query

    minimiser(kmer) → partition p
    for each layer l in p:
        if let Some(slot) = MphfLayer_l.find(kmer):
            return data_l.read(slot)
    return None

O(n_layers) MPHF probes worst case; O(1) expected. The result comes from exactly one `(partition, layer)`.

### Aggregation

    result = reduce(
        for p in partitions:       // parallel
            for l in layers(p):    // parallel
                partial(data_p_l)
    )

For normalised metrics, replace with the two-pass cascade.

## Parallelism model

Level | Unit | Coordination
--- | --- | ---
Across partitions | inner stores of `LayeredStore<LayeredStore<S>>` | none
Across layers within a partition | inner stores of `LayeredStore<S>` | none — disjoint kmer sets
Normalised pass 1 (`col_weights`) | per inner store | none — additive
Normalised pass 2 (partial) | per inner store | `global` broadcast read-only
Within a matrix (distance) | upper-triangle pair `(i,j)` | none — rayon `par_iter`

## reindex — evidence conversion in place

`KmerIndex::reindex(target, block_bits)` converts every layer's evidence bundle to `target` without touching the MPHF or `unitigs.bin`:

- `→ Exact`: builds `evidence.bin` + `unitigs.bin.idx`; removes `fingerprint.bin`
- `→ Approx { b, z }`: builds `fingerprint.bin`; removes `evidence.bin` + `unitigs.bin.idx`

On success, `IndexConfig::evidence` and `IndexConfig::block_bits` are updated in `index.meta`. Each layer's `layer_meta.json` is also rewritten with the new `EvidenceKind`.

## estimate — parameter dry-run

`estimate` resolves approximate-evidence parameters (`z`, `b`, target FP rate) and prints the resulting effective kmer size and per-kmer / per-z-window false-positive rates without touching any index. Used to calibrate `Approx { b, z }` before building or reindexing.