2026-06-17 13:14:11 +02:00
# obicompactvec — Complete Reference
## Module structure
```
src/obicompactvec/src/
lib.rs public re-exports
2026-06-18 07:10:08 +02:00
views.rs BitSliceView<'a>, IntSliceView<'a> — zero-copy read views
traits.rs ColumnWeights, CountPartials, BitPartials (matrix aggregation)
2026-06-17 13:14:11 +02:00
bitvec.rs PersistentBitVec, PersistentBitVecBuilder, BitIter
reader.rs PersistentCompactIntVec (read-only)
builder.rs PersistentCompactIntVecBuilder (read-write)
2026-06-17 15:13:22 +02:00
tempintvec.rs TempCompactIntVec, TempCompactIntVecBuilder (temp-file-backed)
tempbitvec.rs TempBitVec, TempBitVecBuilder (temp-file-backed)
2026-06-17 13:14:11 +02:00
bitmatrix.rs PersistentBitMatrix, PersistentBitMatrixBuilder
intmatrix.rs PersistentCompactIntMatrix, PersistentCompactIntMatrixBuilder
2026-06-17 15:13:22 +02:00
colgroup.rs ColGroup, MatrixGroupOps trait
2026-06-17 13:14:11 +02:00
format.rs file format constants, encode/decode helpers
layer_meta.rs LayerMeta (column metadata)
meta.rs matrix metadata
```
2026-06-17 14:24:57 +02:00
```mermaid
graph TD
2026-06-18 07:10:08 +02:00
views --> bitvec
views --> builder
views --> tempbitvec
views --> tempintvec
views --> bitmatrix
views --> intmatrix
2026-06-17 14:24:57 +02:00
format --> reader
format --> builder
reader --> intmatrix
2026-06-17 15:13:22 +02:00
reader --> tempintvec
2026-06-17 14:24:57 +02:00
builder --> intmatrix
2026-06-17 15:13:22 +02:00
builder --> tempintvec
2026-06-18 07:10:08 +02:00
bitvec --> tempbitvec
bitvec --> bitmatrix
2026-06-17 15:13:22 +02:00
tempintvec --> intmatrix
tempintvec --> bitmatrix
tempbitvec --> intmatrix
tempbitvec --> bitmatrix
colgroup --> intmatrix
colgroup --> bitmatrix
2026-06-17 14:24:57 +02:00
layer_meta --> bitmatrix
layer_meta --> intmatrix
meta --> bitmatrix
meta --> intmatrix
```
2026-06-17 13:14:11 +02:00
---
## Compact int encoding
All integer vectors use the same two-tier encoding regardless of storage backend.
**Primary array** — one `u8` per slot:
- Values **0–254** are stored directly. No overhead.
- Value **255 is a sentinel** : the slot's actual value is ≥ 255 and lives in the overflow store.
**Overflow store** — maps slot index to a `u32` value ≥ 255:
2026-06-18 07:10:08 +02:00
- In `PersistentCompactIntVecBuilder` : a `HashMap<usize, u32>` in RAM.
2026-06-17 13:14:11 +02:00
- In `PersistentCompactIntVec` (reader): a sorted `[(slot: u64, value: u32)]` array in the mmap, with a sparse L1-resident index for binary search.
2026-06-17 14:24:57 +02:00
```mermaid
flowchart LR
slot --> P["primary[slot]: u8"]
P -->|"< 255"| V["value = byte (0–254)"]
P -->|"= 255 sentinel"| OV["overflow store"]
2026-06-18 07:10:08 +02:00
OV -->|"Builder"| HM["HashMap<usize, u32>\nin RAM"]
2026-06-17 14:24:57 +02:00
OV -->|"PersistentCompactIntVec"| SA["sorted [(slot,value)] in mmap\n+ sparse L1 index"]
```
2026-06-17 13:14:11 +02:00
**Key property — sentinel 255 = +∞ on `u8`:**
- `min(a, 255) = a` for all `a ≤ 254` → correct when only one side is overflow
- `max(a, 255) = 255` → correct sentinel when either side is overflow
- Only the **both-overflow** case requires reading actual values from the overflow store.
In practice, k (overflow count) ≪ n (total slots). Observed genomic data: ~0.07% of kmer slots are in overflow.
---
2026-06-18 07:10:08 +02:00
## View types
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
The previous trait hierarchy (`BitSlice` , `BitSliceMut` , `IntSlice` , `IntSliceMut` ) has been replaced by two concrete zero-copy view structs with inherent methods. Views are ** `Copy` ** — passing them is free. All read operations live on these two types.
### `BitSliceView<'a>`
```rust
#[derive(Clone, Copy)]
pub struct BitSliceView < 'a > { pub ( crate ) words : & 'a [ u64 ], pub ( crate ) n : usize }
2026-06-17 14:24:57 +02:00
```
2026-06-18 07:10:08 +02:00
Bit `i` is at `words[i >> 6]` bit `i & 63` (LSB-first). Padding bits in the last word are zero.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
| Method | Cost |
|---|---|
| `len()` , `is_empty()` | O(1) |
| `get(slot)` | O(1) |
| `count_ones()` | POPCNT per word, O(n/64) |
| `count_zeros()` | `n − count_ones()` , O(n/64) |
| `iter() -> BitSliceIter<'a>` | O(1) setup, O(n) iteration |
| `partial_jaccard_dist(other: BitSliceView)` | `(a&b).popcount` , `(a\|b).popcount` per word, O(n/64) |
| `jaccard_dist(other: BitSliceView)` | from partial, O(n/64) |
| `hamming_dist(other: BitSliceView)` | `(a^b).popcount` per word, O(n/64) |
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
`BitSliceIter<'a>` : word-level scan; one word per 64 iterations.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
### `IntSliceView<'a>`
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
```rust
#[derive(Clone, Copy)]
pub struct IntSliceView < 'a > {
pub ( crate ) primary : & 'a [ u8 ],
pub ( crate ) overflow_raw : & 'a [ u8 ], // sorted [(slot:u64, value:u32)] entries
pub ( crate ) n_overflow : usize ,
pub ( crate ) n : usize ,
}
2026-06-17 13:14:11 +02:00
```
2026-06-18 07:10:08 +02:00
`overflow_raw` contains `n_overflow` entries of `OVERFLOW_ENTRY_SIZE` bytes each, sorted by slot. The sort invariant is established at `close()` /`freeze()` time.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
| Method | Cost |
|---|---|
| `len()` , `is_empty()` | O(1) |
| `primary_bytes()` | O(1) |
| `overflow_entries() -> impl Iterator<(usize,u32)>` | O(n_overflow) iteration |
| `get(slot)` | O(1) primary; binary search O(log k) for overflow slots |
| `iter() -> IntSliceViewIter<'a>` | merge scan, O(n + k) |
| `sum()` | byte scan + overflow, O(n + k) |
| `count_nonzero()` | byte scan, O(n) |
| Distance methods (`bray_dist` , `euclidean_dist` , `jaccard_dist` , …) | O(n + k) |
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
`IntSliceViewIter<'a>` : merge scan using `overflow_pos` index. Requires sorted overflow — guaranteed by the construction lifecycle.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
**Builder `view()` vs reader `view()`:** `PersistentCompactIntVecBuilder` stores overflow as an unsorted `HashMap` , not raw bytes. Its `view()` returns an `IntSliceView` with `overflow_raw = &[]` and `n_overflow = 0` . This is intentional — the view is primarily useful after `freeze()` . During building, callers that need overflow use `overflow_entries()` directly.
2026-06-17 13:14:11 +02:00
---
## Concrete types
2026-06-17 14:24:57 +02:00
```mermaid
classDiagram
2026-06-18 07:10:08 +02:00
class BitSliceView {
+words: &[u64]
+n: usize
+get(slot) bool
+count_ones() u64
+iter() BitSliceIter
+jaccard_dist/hamming_dist(other: BitSliceView)
2026-06-17 14:24:57 +02:00
}
2026-06-18 07:10:08 +02:00
class IntSliceView {
+primary: &[u8]
+overflow_raw: &[u8]
+n_overflow: usize
+n: usize
+get(slot) u32
+iter() IntSliceViewIter
+overflow_entries() Iterator
+bray_dist/euclidean_dist/…(other: IntSliceView)
2026-06-17 14:24:57 +02:00
}
class PersistentBitVec {
-mmap: Mmap
-n: usize
2026-06-18 07:10:08 +02:00
+view() BitSliceView
+get(slot) bool
+count_ones/zeros() u64
2026-06-17 14:24:57 +02:00
+iter() BitIter
2026-06-18 07:10:08 +02:00
+partial_jaccard_dist(&Self) (u64,u64)
+jaccard_dist/hamming_dist(&Self) …
2026-06-17 14:24:57 +02:00
}
class PersistentBitVecBuilder {
-mmap: MmapMut
-n: usize
2026-06-18 07:10:08 +02:00
+view() BitSliceView
+set(slot, bool)
+or/and/xor/not(BitSliceView)
+copy_from(BitSliceView)
+close() / finish() → PersistentBitVec
2026-06-17 14:24:57 +02:00
}
class PersistentCompactIntVec {
-mmap: Mmap
2026-06-18 07:10:08 +02:00
-n: usize
-n_overflow: usize
-step: usize
2026-06-17 14:24:57 +02:00
-index: Vec~(usize,usize)~
2026-06-18 07:10:08 +02:00
+view() IntSliceView
2026-06-17 14:24:57 +02:00
+get(slot) u32
2026-06-18 07:10:08 +02:00
+iter() Iter
+sum/count_nonzero() u64
+bray_dist/euclidean_dist/… (&Self)
2026-06-17 14:24:57 +02:00
}
class PersistentCompactIntVecBuilder {
-mmap: MmapMut
-n: usize
-overflow: HashMap~usize,u32~
2026-06-18 07:10:08 +02:00
+view() IntSliceView
+set(slot, u32) / get(slot) u32
+inc / inc_present / inc_present_fast
+inc_predicate / inc_predicate_fast
+add/min/max/diff/mask_with(…View)
+primary_bytes/primary_bytes_mut()
+close() / finish() → PersistentCompactIntVec
2026-06-17 14:24:57 +02:00
}
2026-06-18 07:10:08 +02:00
PersistentBitVec --> BitSliceView : view()
PersistentBitVecBuilder --> BitSliceView : view()
PersistentCompactIntVec --> IntSliceView : view()
PersistentCompactIntVecBuilder --> IntSliceView : view() (primary only)
2026-06-17 14:24:57 +02:00
PersistentBitVecBuilder --> PersistentBitVec : close() then open()
PersistentCompactIntVecBuilder --> PersistentCompactIntVec : close() then open()
```
2026-06-18 07:10:08 +02:00
### `PersistentBitVec` / `PersistentBitVecBuilder`
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
`PersistentBitVec` is the read-only type. `view()` returns a `BitSliceView<'_>` over the mmap word array. Direct inherent methods delegate to the view: `count_ones()` , `count_zeros()` , `partial_jaccard_dist(&Self)` , `jaccard_dist(&Self)` , `hamming_dist(&Self)` .
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
`BitIter<'a>` — exported iterator for `PersistentBitVec::iter()` :
2026-06-17 13:14:11 +02:00
```rust
pub struct BitIter < 'a > { pub ( crate ) words : & 'a [ u64 ], pub ( crate ) slot : usize , pub ( crate ) n : usize }
```
2026-06-18 07:10:08 +02:00
`PersistentBitVecBuilder` is the read-write type. Mutation operations accept `BitSliceView<'_>` :
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
| Method | Cost |
|---|---|
| `set(slot, bool)` | O(1) |
| `view() -> BitSliceView<'_>` | O(1) |
| `or/and/xor(BitSliceView)` | word-level, O(n/64), SIMD-friendly |
| `not()` | `w ^= u64::MAX` per word, re-masks last word | O(n/64) |
| `copy_from(BitSliceView)` | `copy_from_slice` | O(n/64) |
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
### `PersistentCompactIntVec` / `PersistentCompactIntVecBuilder`
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
`PersistentCompactIntVec` is the read-only type. `view()` returns an `IntSliceView<'_>` over the mmap primary and overflow arrays. Inherent `iter()` is a merge scan (`Iter` struct). Inherent `sum()` and `count_nonzero()` use fast byte-scan helpers.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
`PersistentCompactIntVecBuilder` is the read-write type. Mutation methods on the builder fall into two categories:
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
**Point mutations:**
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
| Method | Note |
|---|---|
| `set(slot, u32)` | writes primary[slot] or 255+overflow |
| `get(slot) -> u32` | reads primary byte or HashMap |
| `inc(slot)` | `get` + `set` , O(1) |
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
**Bulk computation methods** — accept view arguments:
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
| Method | Semantics | Overflow |
|---|---|---|
| `inc_present(BitSliceView)` | `+= 1` at each 1-bit | via `inc` , safe for any group size |
| `inc_present_fast(BitSliceView)` | same, raw u8 `+= 1` | `debug_assert` no 255 reached |
| `inc_predicate(IntSliceView, pred)` | `+= 1` where `pred(col[s])` | two-pass, safe |
| `inc_predicate_fast(IntSliceView, pred)` | same, raw u8 | `debug_assert` no 255 reached |
| `add(IntSliceView)` | `self[s] += other[s]` | primary fast path + overflow fallback |
| `min(IntSliceView)` | byte min + both-overflow fixup | see algorithm below |
| `max(IntSliceView)` | pre-pass + byte max | see algorithm below |
| `diff(IntSliceView)` | saturating sub | self<255 hot path |
| `mask_with(BitSliceView)` | zeros slots where mask bit = 0 | O(n_zeros) |
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
** `inc_present_fast` / `inc_predicate_fast` invariant:** caller guarantees no counter reaches 255 during the operation (group size < 255 for `inc_present_fast` , or chunk size < 255 for `inc_predicate_fast` ). Violation is caught by `debug_assert` in dev builds.
** `min` algorithm:**
Exploits 255 = +∞: byte-level min is correct unless both sides are overflow.
```
snapshot self_ov: Vec<(slot,val)>
snapshot other_ov: HashMap<slot,val>
clear_overflow()
Pass 1 — byte min, SIMD-vectorizable, O(n)
Pass 2 — both-overflow fixup, O(k_self):
for (slot, self_val) in self_ov:
if slot ∈ other_ov: set(slot, min(self_val, other_ov[slot]))
```
** `max` algorithm:**
Cannot do byte max first — `max(255, b<255)=255` overwrites self's original overflow value. Pre-pass reads self's value at other's overflow slots before the byte pass.
```
Pre-pass O(k_other): for (slot, other_val) in other.overflow_entries():
set(slot, max(self.get(slot), other_val))
Pass 1 — byte max, SIMD-vectorizable, O(n)
```
2026-06-17 13:14:11 +02:00
---
## Matrix types
2026-08-26 14:00:04 +02:00
Both matrix types are enums behind a transparent API — the caller never matches on the variant. `PersistentCompactIntMatrix` has three variants (`Columnar` , `Packed` , `Sparse` ). `PersistentBitMatrix` has four:
2026-06-17 13:14:11 +02:00
2026-08-19 19:27:50 +02:00
| Variant | Storage | When |
2026-06-17 13:14:11 +02:00
|---|---|---|
2026-08-19 19:27:50 +02:00
| `Columnar` | one `.pbiv` /`.pciv` file per column + `meta.json` | build-time default (`*Builder::new` ) |
2026-08-26 14:00:04 +02:00
| `Packed` | single `matrix.pbmx` /`matrix.pcmx` mmap file | query-optimised, produced by `pack_bit_matrix` /`pack_compact_int_matrix` |
2026-09-18 19:17:59 +02:00
| `Sparse` | bit: `sparse_meta.json` + PFIV/Elias-Fano component files, row-major. Int: same support files (built on `PersistentSparseBitMatrix` internally) plus `singleton_values.pciv` /`multi_values.pciv` /`multi_offsets` for the per-row, non-deduplicated values | `pack --sparse` ; see "Sparse matrix formats" below for the sparse-vs-dense access-pattern trade-off |
2026-08-19 19:27:50 +02:00
| `Implicit` (bit only) | no file at all | mono-genome presence layers — `n_cols` is always reported as `1` , every value is `true` |
2026-06-17 13:14:11 +02:00
2026-08-26 14:00:04 +02:00
`PersistentBitMatrix::open(layer_dir)` auto-detects the variant, in order: `matrix.pbmx` → Packed, `presence/meta.json` → Columnar, `presence/sparse_meta.json` → Sparse, `layer_meta.json` (no presence dir at all) → Implicit. `PersistentCompactIntMatrix::open(layer_dir)` mirrors the same priority order minus `Implicit` (there's no implicit count matrix — counts always have at least one on-disk column): `matrix.pcmx` → Packed, `counts/meta.json` → Columnar, `counts/singleton_values.pciv` → Sparse. `col_view` /`col` /`sub_matrix` panic on `Sparse` /`Implicit` where the operation has no direct-slice equivalent (Sparse is k-mer-major, not column-major; Implicit has no backing storage) — callers needing per-column data on those variants go through `row` /`fill_row` .
2026-08-19 19:27:50 +02:00
`col_view(c)` returns the appropriate view directly:
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
```rust
// PersistentBitMatrix
pub fn col_view ( & self , c : usize ) -> BitSliceView < '_ >
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
// PersistentCompactIntMatrix
pub fn col_view ( & self , c : usize ) -> IntSliceView < '_ >
```
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
No wrapper enums (`BitColView` , `IntColView` ): the caller receives a `Copy` view struct immediately usable with any view method or bulk builder method.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
`pack_compact_int_matrix` and `pack_bit_matrix` convert columnar → packed format.
2026-06-17 13:14:11 +02:00
---
2026-09-18 19:17:59 +02:00
## Sparse matrix formats
Motivated by real k-mer-family data being extremely row-sparse (most rows —
k-mers — present in only a handful of genome columns out of dozens to
thousands): a row-major, deduplicated alternative to the dense column-major
formats above, built once per layer like the rest of the pipeline (never the
whole multi-billion-row index at once). Implements the exact-duplicate
special case of Alanko, Bille, Gørtz, Navarro, Puglisi, "Compact Data
Structures for Collections of Sets" (2025,
`biblio/Alanko et al. - Compact Data Structures for Collections of
Sets.pdf` ) — a plain dedup dictionary, not their full subset-containment
hierarchy.
### `PersistentSparseBitMatrix`
Four mmap-backed on-disk components: an `is_multi` rank-capable flag per row
(singleton vs. multi-genome), a fixed-bit-width array for singleton rows
(genome index directly, `ceil(log2(n_cols))` bits), a separate fixed-width
array for multi-genome rows (`dict_id` , `ceil(log2(n_distinct_multi_sets))`
bits — kept apart from the singleton array because `n_distinct_multi_sets`
can be large in absolute terms even when multi-genome rows are a small
*fraction* of all rows, and a shared array would force every row, singletons
included, to pay the wider width), and a deduplicated dictionary of distinct
multi-genome sets (Elias-Fano-encoded byte offsets + a varint-encoded values
blob). Supporting primitives: `PersistentFixedIntVec` (arbitrary,
runtime-parameterised bit width, width `0` included — needed once an
`EliasFano` reopen bug surfaced, see below), `PersistentRankSelectBitVec`
(rank1/rank0/select1 on top of the crate's existing `count_ones` , via
`common_traits::SelectInWord` ), `EliasFano` (composes the two). `BinaryMatrix`
(`n` , `n_cols` , `row` /`fill_row` , `fill_sub_matrix` , `count_ones` ) is the
trait unifying dense and sparse at the call sites that need both
interchangeably; column-oriented methods (`col` , `col_view` , the
`partial_*_dist_matrix` family) stay dense-only — `Sparse` /`Implicit` panic
there (see the `PersistentBitMatrix::open` note above).
Measured on real data (one layer, 30.2M rows, 91 genomes):
| | dense | sparse | ratio |
|---|---|---|---|
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
| build time / peak RSS | — | 4.26s / 628MB | per-layer, in-memory construction |
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse faster (smaller structure, better cache fit) |
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse ** ~4.8x faster** — the `--shannon` /family-lookup access shape |
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** — no native column method, every read decodes a full row to keep one bit |
The row-access wins (compactness was the stated design goal, not speed) come
from dense's genome-major layout scattering a single row read across a much
bigger file — costlier than sparse's rank/select/varint decode once the file
is this much smaller. Column access is the flip side of the same layout
choice; `nonzero_iter` (below) is what removes it from the hot path that
actually matters (`query` ).
Two reopen-correctness bugs, now fixed at the source: (1) `EliasFano::open`
re-derived its low-bits width from the persisted low-vector's own width
byte; the width-`0` case was built with a dummy 1-bit placeholder (the
builder used to reject true width `0` ), silently doubling every reopened
value. Fixed by making `PersistentFixedIntVec` genuinely support width `0`
(no storage, `get` always `0` ). (2) An empty row (cardinality 0) used to be
recorded as a singleton at genome `0` , indistinguishable on read-back from a
*real* singleton at genome `0` . Fixed by routing cardinality-0 rows through
the dictionary path (a genuine empty entry).
Still deferred: full Alanko et al. subset-hierarchy compression (only the
exact-duplicate case is built), a sparse `PersistentCompactIntMatrix`
variant beyond what's described below, BRWT-style column-correlation
exploitation, and a row-major rewrite of
`partial_jaccard_dist_matrix` /`partial_hamming_dist_matrix` /etc. as a
per-row co-occurrence accumulation (`O(Σ_rows k̄²)` ) so `obikindex` 's
`--metric` distance-matrix path stops paying the column-access penalty on
`Sparse` — needs its own design pass (how it plugs into
`BitPartials` /`ColumnWeights` so both matrix types keep serving `--metric` ),
not just "port the loop".
### `nonzero_iter` — the one traversal per format
`fn nonzero_iter<'a>(&'a self, slots: &'a [usize]) -> Box<dyn Iterator<Item = (usize, usize, u32)> + 'a>`
(item: index into `slots` , column, value) is the single foundational
per-format traversal primitive every consumer builds on — `fill_sub_matrix`
and a `for_each` -style callback are both trivial, free consumers of it
(`.for_each(f)` is `Iterator::for_each` ; `fill_sub_matrix` drains the
iterator and scatters into `out[][]` ), instead of two independently
hand-maintained traversals that can silently diverge. `Box<dyn Iterator>` ,
not a bare `impl Iterator` : `Columnar` /`Packed` /`Sparse` /`Implicit` are
genuinely different concrete types and the method isn't on a
dyn-compatible trait (`BinaryMatrix` is used as `dyn BinaryMatrix`
elsewhere, and RPITIT methods aren't dyn-compatible) — one `Box` per
`nonzero_iter` call, not per cell.
Per-format implementation: `PersistentSparseBitMatrix` decodes natively,
buffered one row at a time (`std::iter::from_fn` over the existing
`for_each_genome_in_row` , no `n_cols` -wide allocation ever). `Columnar` /
`Packed` loop columns, collecting each column's hits via
`BitSliceView::nonzero_among_sorted` /`IntSliceView::nonzero_among_sorted`
(a `filter` /`filter_map` over the vector-level `get()` , no new state
machine — `std` 's own `Filter` iterator already is one) into one `Vec` via
`.extend()` (not `flat_map` : a `flat_map` closure can't lazily return
something borrowing its own captured sort permutation across separate calls
without either boxing per-column or fighting the borrow checker — eager
collection sidesteps it at zero extra cost, since `fill_sub_matrix` already
fully materialises anyway). `Implicit` is trivial (`slots.iter().map(|&i| (i, 0, 1))` ).
This closed a real dispatch bug by construction, not by patching it:
`PersistentBitMatrix::fill_sub_matrix` /`sub_matrix` used to reimplement a
naive per-`(column, slot)` loop instead of delegating to
`PersistentSparseBitMatrix` 's own efficient native method for `Self::Sparse`
— two hand-written traversals for the same format, free to drift apart, and
they had (`obikphylo` 's own cache layer had independently built a parallel
enum wrapper that *did* dispatch correctly, evidence the bug was worked
around rather than fixed at the source). With `fill_sub_matrix` /`sub_matrix`
rewritten to drain `nonzero_iter` , there is only one traversal per format
left to get right. `KmerPartition::query_partition_with`
(`obikindex/src/partition/query_layer.rs` ) — previously column-major
(`for g { for slot { col_value } }` , which for `Sparse` rebuilt a full row
per single cell touched, O(hits × n_cols) instead of O(hits) — the original
motivation for this primitive) — now issues one `layer.nonzero_iter(&slot_list)`
call per layer, format-agnostic. Dense/sparse query performance parity
confirmed on real data (previously sparse 30-50% slower than dense,
reproducibly; now within ~1-3% either way — noise level).
### `PersistentSparseCompactIntMatrix`
The count-matrix analogue, built on `PersistentSparseBitMatrix` as its
"which columns are non-zero" support, values **not** deduplicated (unlike
the bit side, two rows can share the same non-zero column set — same
`dict_id` — while carrying different counts, so the dict-multiplicity
shortcut `BitPartials for PersistentSparseBitMatrix` uses doesn't carry
over). `CountPartials` here: a single row-major pass (`row_major_pairwise`
in `sparse_intmatrix.rs` , decodes each row once via
`for_each_cell_in_row` , nests over that row's own co-present columns) —
`O(Σ k̄²)` over populated rows, avoiding the naive `O(n_cols² × n)`
column-pair rescan. `min(a,b)` -based kernels (`bray` , `relfreq_bray` ) need
no correction — they vanish when either side is absent. `a·b` and `√(a·b)`
kernels (`euclidean` /`relfreq_euclidean` /`hellinger` ) do need one, since
`(a-0)² = a² ≠ 0` unlike the `min` -based formulas — reconstructed from
per-column marginals via `Σ(a-b)² = Σa²+Σb²-2Σab` .
`threshold_jaccard(1)` shortcuts to `BitPartials::partial_jaccard`
(threshold 1 is exactly presence); `threshold_jaccard(0)` is closed-form
(every `u32` is `≥ 0` ).
Wiring the `threshold==1` shortcut surfaced two pre-existing formula bugs in
`BitPartials for PersistentSparseBitMatrix` (present since that type's own
introduction, never caught because no test had compared `Sparse` 's raw
`partial_*` output against dense on real data — only the diagonal-blind
`jaccard_dist_matrix` /`hamming_dist_matrix` finalisations were tested),
fixed at the source:
1. `partial_jaccard` 's diagonal used to be `(0, 2×col_weights[i])` instead
of a genuine self-comparison `(col_weights[i], col_weights[i])` —
`col_weights_and_pair_counts` 's `inter` never pairs a column with itself
by construction, so the diagonal needs its own case.
2. `partial_hamming` 's off-diagonal formula itself was wrong: it computed
`total - union` (rows where *neither* column is present) instead of the
actual Hamming distance `col_weights[i] + col_weights[j] - 2×inter[i,j]`
(symmetric-difference size) — the two only coincide when
`col_weights[i] + col_weights[j] == total` , which small/synthetic test
data can easily satisfy by accident.
2026-06-17 13:14:11 +02:00
## Aggregation traits (matrix level)
### ColumnWeights
```rust
trait ColumnWeights : Send + Sync {
fn col_weights ( & self ) -> Array1 < u64 > ; // sum per column
fn partial_kmer_counts ( & self ) -> Array1 < u64 > ; // default = col_weights()
}
```
`partial_kmer_counts` is overridden for count matrices to return `count_nonzero` per column (distinct kmers) rather than total count.
### CountPartials
Abstract required methods: `partial_bray` , `partial_euclidean` , `partial_threshold_jaccard` , `partial_relfreq_bray` , `partial_relfreq_euclidean` , `partial_hellinger` .
2026-06-18 07:10:08 +02:00
**Additivity rule:** self-contained partials (`partial_bray` , `partial_euclidean` , `partial_threshold_jaccard` ) can be element-wise summed across all `(partition, layer)` pairs. Normalised partials (`partial_relfreq_*` , `partial_hellinger` ) require the **global** `col_weights` (accumulated across all layers and all partitions) as parameter.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
** `partial_threshold_jaccard` returns `(inter, union)` ** because `union[i,j]` depends on both columns simultaneously.
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
Provided finalisations:
2026-06-17 13:14:11 +02:00
| Finalisation | Formula |
|---|---|
| `bray_dist_matrix()` | `1 − 2·partial_bray[i,j] / (w[i] + w[j])` |
| `euclidean_dist_matrix()` | `√partial_euclidean[i,j]` |
| `threshold_jaccard_dist_matrix(t)` | `1 − inter[i,j] / union[i,j]` |
2026-06-18 07:10:08 +02:00
| `relfreq_bray_dist_matrix()` | `1 − partial_relfreq_bray[i,j]` |
2026-06-17 13:14:11 +02:00
| `relfreq_euclidean_dist_matrix()` | `√partial_relfreq_euclidean[i,j]` |
| `hellinger_dist_matrix()` | `√partial_hellinger[i,j] / √2` |
| `hellinger_euclidean_dist_matrix()` | `√partial_hellinger[i,j]` |
2026-07-09 11:37:52 +02:00
| `threshold_mash_dist_matrix(k, t)` | Mash distance, derived from `threshold_jaccard_dist_matrix(t)` — no separate partial |
2026-06-17 13:14:11 +02:00
### BitPartials
2026-06-18 07:10:08 +02:00
Required: `partial_jaccard() -> (Array2<u64>, Array2<u64>)` , `partial_hamming() -> Array2<u64>` . Both additive across layers and partitions.
2026-06-17 13:14:11 +02:00
2026-07-09 11:37:52 +02:00
Provided finalisations also include `jaccard_dist_matrix()` , `hamming_dist_matrix()` , and `mash_dist_matrix(k)` .
### Mash distance
`mash_dist_matrix` /`threshold_mash_dist_matrix` add no new additive primitive: both are a pointwise transform of the existing Jaccard distance matrix, per the Mash mutation-rate estimator [@Mash -distances-doc; @Fan2015 -mash-formula]:
```
D = -1/k · ln(2J / (1+J)), J = 1 - d_jaccard
```
`J ≤ 0` (i.e. `d_jaccard ≥ 1` , no shared k-mers) maps to `D = 1` (maximal distance) rather than the `ln` singularity at `J = 0` .
2026-06-17 13:14:11 +02:00
---
2026-06-17 15:13:22 +02:00
## Temp-file-backed types
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
**All inter-function results use temp-file-backed types** so the OS can page them out under memory pressure. This matters in practice: processing dozens of layers × hundreds of partitions in parallel would otherwise accumulate gigabytes of live anonymous memory.
2026-06-17 14:24:57 +02:00
2026-06-17 15:13:22 +02:00
### Lifecycle
2026-06-17 14:24:57 +02:00
```
2026-06-17 15:13:22 +02:00
TempCompactIntVecBuilder::new(n) → writable mmap in TempDir
2026-06-18 07:10:08 +02:00
↓ (inc_present_fast / inc_predicate_fast / add / mask_with / …)
2026-06-17 15:13:22 +02:00
.freeze() → TempCompactIntVec (read-only mmap + TempDir)
↓ (optional)
.make_persistent(path) → PersistentCompactIntVec (permanent file)
```
Same pattern for `TempBitVecBuilder` → `TempBitVec` → `PersistentBitVec` .
2026-06-18 07:10:08 +02:00
**Drop order** : `TempCompactIntVec { vec: PersistentCompactIntVec, _temp: TempDir }` — Rust drops fields in declaration order. `vec` (mmap) released before `_temp` (directory deleted). No explicit `drop()` needed.
2026-06-17 15:13:22 +02:00
### TempCompactIntVec / TempCompactIntVecBuilder
```rust
pub struct TempCompactIntVec {
vec : PersistentCompactIntVec ,
_temp : TempDir , // dropped after vec
}
pub ( crate ) struct TempCompactIntVecBuilder {
builder : PersistentCompactIntVecBuilder ,
temp : TempDir ,
}
```
2026-06-18 07:10:08 +02:00
`TempCompactIntVec` : read access via `get(slot)` , `sum()` , `iter()` , `view() -> IntSliceView<'_>` .
`TempCompactIntVecBuilder` : full delegation to inner `PersistentCompactIntVecBuilder` — all bulk computation methods (`inc_present_fast` , `inc_predicate_fast` , `add` , `min` , `max` , `diff` , `mask_with` ) are exposed as `pub(crate)` .
2026-06-17 15:13:22 +02:00
### TempBitVec / TempBitVecBuilder
```rust
pub struct TempBitVec {
vec : PersistentBitVec ,
_temp : TempDir ,
}
pub ( crate ) struct TempBitVecBuilder {
builder : PersistentBitVecBuilder ,
temp : TempDir ,
}
```
2026-06-18 07:10:08 +02:00
`TempBitVec` : read access via `get(slot)` , `count_ones()` , `view() -> BitSliceView<'_>` , `iter()` .
`TempBitVecBuilder` : exposes `set(slot, bool)` , `or(BitSliceView)` , and:
```rust
pub ( crate ) fn or_where ( & mut self , col : IntSliceView < '_ > , pred : impl Fn ( u32 ) -> bool )
```
`or_where` — two passes, no intermediate allocation:
```
Pass 1 — primary bytes, O(n):
for slot in 0..n:
b = col.primary_bytes()[slot]
if b < 255 AND pred(b as u32): self.set(slot, true)
Pass 2 — overflow, O(k):
for (slot, val) in col.overflow_entries():
if pred(val): self.set(slot, true)
```
2026-06-17 15:13:22 +02:00
---
## Filter / Select API
2026-06-17 14:24:57 +02:00
2026-06-17 13:14:11 +02:00
### ColGroup
```rust
2026-06-17 15:13:22 +02:00
pub struct ColGroup { pub name : String , pub indices : Vec < usize > }
2026-06-17 13:14:11 +02:00
```
2026-06-18 07:10:08 +02:00
Defined **once at the index level** from column metadata. Valid in all matrices of all layers and partitions — column structure is identical across the entire hierarchy; only rows (kmer slots) are partitioned.
2026-06-17 13:14:11 +02:00
### Composition axis
2026-06-18 07:10:08 +02:00
- **Across partitions**: kmer space is partitioned → partial results **concatenated** (disjoint kmer ranges).
- **Across layers**: same kmer space, different counts → partial results **aggregated** (add, OR, etc.).
2026-06-17 13:14:11 +02:00
2026-06-17 15:13:22 +02:00
### MatrixGroupOps
2026-06-17 14:24:57 +02:00
2026-06-18 07:34:29 +02:00
Five required primitives + two default methods derived from them. All return temp-file-backed types.
2026-06-17 13:14:11 +02:00
```rust
2026-06-17 15:13:22 +02:00
pub trait MatrixGroupOps {
2026-06-18 07:34:29 +02:00
// required
2026-06-17 15:13:22 +02:00
fn partial_group_presence_count ( & self , g : & ColGroup , threshold : u32 )
-> io ::Result < TempCompactIntVec > ;
fn partial_group_sum ( & self , g : & ColGroup )
-> io ::Result < TempCompactIntVec > ;
fn partial_group_any ( & self , g : & ColGroup , threshold : u32 )
-> io ::Result < TempBitVec > ;
2026-06-18 07:34:29 +02:00
fn partial_group_min ( & self , g : & ColGroup )
-> io ::Result < TempCompactIntVec > ;
fn partial_group_max ( & self , g : & ColGroup )
-> io ::Result < TempCompactIntVec > ;
// defaults derived from partial_group_presence_count
fn partial_group_all ( & self , g : & ColGroup , threshold : u32 )
-> io ::Result < TempBitVec > ; // slot=1 iff count == g.indices.len()
fn partial_group_none ( & self , g : & ColGroup , threshold : u32 )
-> io ::Result < TempBitVec > ; // slot=1 iff count == 0
2026-06-17 13:14:11 +02:00
}
```
2026-06-18 07:34:29 +02:00
Implemented for both `PersistentCompactIntMatrix` and `PersistentBitMatrix` .
For **bit matrices** : values are 0/1, so `partial_group_sum` = `partial_group_presence_count(g, 1)` ; `partial_group_min` is AND (set first column then mask-with remaining); `partial_group_max` is OR via `partial_group_any` + `inc_present` .
2026-06-17 15:13:22 +02:00
** `partial_group_presence_count` — chunking for large groups:**
2026-06-18 07:34:29 +02:00
When `g.indices.len() < 255` : per-slot counts stay within `u8` range. Use `inc_present_fast` (bit) or `inc_predicate_fast(col_view(c), |v| v >= threshold)` (int) — raw u8 increment, no overflow entry written.
2026-06-17 15:13:22 +02:00
2026-06-18 07:34:29 +02:00
When `g.indices.len() ≥ 255` : process in chunks of 254 columns, accumulate via `.add(chunk_frozen.view())` .
2026-06-17 15:13:22 +02:00
2026-06-18 07:34:29 +02:00
** `partial_group_min` (int matrix)**: copy first column via `.add(col_view(first))` (start from 0 ⇒ copy), then `.min(col_view(c))` for remaining.
2026-06-17 15:13:22 +02:00
2026-06-18 07:34:29 +02:00
** `partial_group_max` (int matrix)**: `.max(col_view(c))` for all columns (start from 0 ⇒ first column acts as copy).
2026-06-17 15:13:22 +02:00
2026-06-18 07:34:29 +02:00
** `partial_group_any` ** uses `or_where` on `TempBitVecBuilder` (two-pass: primary bytes then overflow entries).
2026-06-17 13:14:11 +02:00
2026-06-18 07:34:29 +02:00
** `partial_group_all` / `partial_group_none` ** (default): call `partial_group_presence_count` , then iterate slots to produce the bit result. O(n) extra pass, not chunked.
2026-06-18 07:10:08 +02:00
2026-06-18 07:34:29 +02:00
### add_col_from — matrix builder integration
Both matrix builders accept temp-file results directly:
2026-06-18 07:10:08 +02:00
```rust
2026-06-18 07:34:29 +02:00
// PersistentBitMatrixBuilder
fn add_col_from ( & mut self , src : & TempBitVec ) -> io ::Result < () >
fn add_col_from_int ( & mut self , src : & TempCompactIntVec ) -> io ::Result < () > // nonzero → 1
2026-06-17 13:14:11 +02:00
2026-06-18 07:34:29 +02:00
// PersistentCompactIntMatrixBuilder
fn add_col_from ( & mut self , src : & TempCompactIntVec ) -> io ::Result < () >
fn add_col_from_bit ( & mut self , src : & TempBitVec ) -> io ::Result < () > // bit → 0/1 u32
2026-06-17 13:14:11 +02:00
```
2026-06-18 07:34:29 +02:00
`add_col_from` copies the temp file to the matrix directory and increments `n_cols` ; `close()` writes `meta.json` with the final column count. No separate `write_meta` step needed.
2026-06-18 07:10:08 +02:00
### mask_with
2026-06-17 13:14:11 +02:00
2026-06-18 07:10:08 +02:00
Direct method on `PersistentCompactIntVecBuilder` (and delegation via `TempCompactIntVecBuilder` ). Zeros every slot where the corresponding mask bit is 0. Iterates only zero bits — O(n_zeros), O(1) when mask is all-ones.
2026-06-17 13:14:11 +02:00
```
for (w_idx, word) in mask.words():
2026-06-17 15:13:22 +02:00
if word == u64::MAX: continue // skip all-ones words
2026-06-17 13:14:11 +02:00
zeros = !word
while zeros != 0:
bit = trailing_zeros(zeros)
s = w_idx * 64 + bit
2026-06-18 07:10:08 +02:00
if primary[s] != 0: set(s, 0) // clears overflow entry too
2026-06-17 13:14:11 +02:00
zeros &= zeros − 1
```
2026-06-17 15:13:22 +02:00
Terminal operation for Filter (retain only selected kmer slots in a count vector) and Select (positional selection without MPHF).