Skip to content

# PersistentCompactIntVec and PersistentCompactIntMatrix

## Purpose

`PersistentCompactIntVec` stores a dense array of non-negative integers indexed by MPHF slot where the vast majority of values are small (0–254) and large values are rare. It is designed for mmap-compatible random and sequential access with minimal memory footprint and optimal cache behaviour.

Motivation from observed count distributions in genomics data: 99.9% of k-mer counts fit in a u8; overflow (count ≥ 255) affects ~0.07% of distinct k-mers but can reach values above 10⁶ (chloroplast, ribosomal repeats).

`PersistentCompactIntMatrix` wraps multiple `PersistentCompactIntVec` columns in a directory, exposing a column-major matrix with row-access API. A vector is a matrix with 1 column.

## PersistentCompactIntVec — single-column file

### Design

Two-tier structure:

1. Primary array — `[u8; n]`, stored at offset 40 in the PCIV file and mmap'd. Values 0–254 are stored directly. Value 255 is a sentinel meaning "look in overflow".
2. Overflow section — sorted list of `(slot: u64, value: u32)` pairs for all slots where the true value ≥ 255, with a sparse L1-fitting index for fast lookup.

    primary[slot] < 255  →  return primary[slot]
    primary[slot] == 255 →  binary search in overflow

### File format

Single `.pciv` file. Write order: header placeholder → primary → overflow + index → header overwrite at offset 0.

    offset 0:
      magic:      [u8; 4]   = b"PCIV"
      _pad:       [u8; 4]   = 0
      n:          u64        number of slots
      n_overflow: u64        number of overflow entries
      n_index:    u64        number of sparse index entries
      step:       u64        sparse index step (0 = no index)

    offset 40:
      primary:    [u8; n]    one byte per slot, 255 = overflow sentinel

    offset 40 + n:
      data:       [(slot: u64, value: u32); n_overflow]   12 bytes each, sorted by slot

    offset 40 + n + n_overflow × 12:
      index:      [(slot: u64, pos: u64); n_index]         16 bytes each, sparse index

The index entries point into `data`: `index[i] = (slot of data[i×step], i×step)`.

All integer fields are little-endian. Slot indices are stored as `u64` in the file; they are `usize` in Rust code.

### Lifecycle

#### Builder (`PersistentCompactIntVecBuilder`)

Used during construction. The primary section is mmap'd immediately at construction time (both for `new` and `build_from`), so the file exists and is addressable from the start. The overflow is held in a `HashMap<usize, u32>` in RAM.

    struct PersistentCompactIntVecBuilder {
        path:     PathBuf,
        mmap:     MmapMut,            // primary section live in the file from the start
        n:        usize,
        overflow: HashMap<usize, u32>, // values ≥ 255
    }

`new(n: usize, path: &Path) -> io::Result<Self>`

Creates the file, pre-allocates `HEADER_SIZE + n` zero bytes, mmaps it. The primary is zero-initialised (all slots = 0). Returns immediately ready for `set` / `get`.

`build_from(source: &PersistentCompactIntVec, path: &Path) -> io::Result<Self>`

Copies the source PCIV file to `path` (OS-level copy — no per-slot iteration), mmaps the copy, then loads the overflow section into a `HashMap`. Initialisation cost: O(file copy) + O(n_overflow), not O(n).

At `close()`, the primary section is not rewritten: it is already in the file via mmap. Only the overflow data, the sparse index, and the header are updated.

`set(slot: usize, value: u32)` / `get(slot: usize) -> u32`

Direct mmap byte access for the primary; HashMap for the overflow. Both O(1). Mutations can move a slot between tiers freely (downward mutation removes the HashMap entry; upward mutation adds it).

Element-wise operations — `min`, `max`, `add`, `diff`

Each takes a `&PersistentCompactIntVec` of equal length and updates `self` in place via `set`:

    builder.min(&other);   // self[i] = min(self[i], other[i])
    builder.max(&other);   // self[i] = max(self[i], other[i])
    builder.add(&other);   // self[i] = self[i].checked_add(other[i])  (panics on u32 overflow)
    builder.diff(&other);  // self[i] = self[i].saturating_sub(other[i])

All iterate `other` with `other.iter()` (merge-scan, O(n_other)).

`close(self) -> io::Result<()>`

1. Flush and drop the mmap (primary changes are now on disk).
2. Sort the overflow HashMap into `Vec<(usize, u32)>`.
3. Truncate the file to `HEADER_SIZE + n` (removes old data+index if `build_from` was used).
4. Append sorted overflow data, then sparse index.
5. Seek to offset 0, overwrite the header with final values.

#### Reader (`PersistentCompactIntVec`)

Used at query time. The whole file is mmap'd; only the sparse index is copied into a `Vec` at open time (≤ 32 KB, L1-resident).

    struct PersistentCompactIntVec {
        mmap:           Mmap,
        n:              usize,
        n_overflow:     usize,
        step:           usize,
        index:          Vec<(usize, usize)>,  // (slot, pos) — L1-resident
        primary_offset: usize,               // = 40 (HEADER_SIZE)
        data_offset:    usize,               // = 40 + n
        path:           PathBuf,
    }

`open(path: &Path) -> io::Result<Self>`

Mmaps the file, parses the 40-byte header, copies the sparse index entries into a `Vec`. The primary and data sections stay mmap'd.

`get(slot: usize) -> u32` — random access

    primary[slot] < 255  →  return it directly

    step == 0:
        binary_search(data[0..n_overflow], slot)

    step > 0:
        i = upper_bound(index[..].slot, slot) − 1     // in L1-resident Vec
        binary_search(data[index[i].pos .. index[i+1].pos], slot)

`iter() -> Iter<'_>` — sequential scan, O(n)

Merge-scan: reads primary bytes in order; on sentinel 255, advances a sequential pointer into the sorted data section rather than doing a binary search. This gives O(n + n_overflow) with no random access into the data section.

`Iter` implements `ExactSizeIterator`. `&PersistentCompactIntVec` implements `IntoIterator`.

Aggregate

    fn sum(&self) -> u64   // Σ self[i] as u64, via iter()

Distance methods

All take `&other` of equal length, iterate both with `zip(self.iter(), other.iter())`, and return `f64`.

Method | Formula
--- | ---
`bray_dist` | `1 − 2·Σmin(aᵢ,bᵢ) / (Σaᵢ + Σbᵢ)`
`relfreq_bray_dist` | Bray-Curtis on relative frequencies: `1 − Σmin(pᵢ,qᵢ)` where `pᵢ = aᵢ/Σa`
`euclidean_dist` | `√Σ(aᵢ − bᵢ)²`
`relfreq_euclidean_dist` | Euclidean on relative frequencies
`hellinger_euclidean_dist` | `√Σ(√pᵢ − √qᵢ)²` — Euclidean on sqrt(relfreq)
`hellinger_dist` | `hellinger_euclidean_dist / √2` — standard Hellinger distance ∈ [0, 1]
`threshold_jaccard_dist(&other, threshold: u32)` | `1 − \|A∩B\| / \|A∪B\|` where presence iff count ≥ threshold
`jaccard_dist` | `threshold_jaccard_dist(&other, 1)`

Edge cases (both vectors all-zero, or union empty for Jaccard): distance = 0.0.

### Step computation

Chosen at `close()` once `n_overflow` is known:

    L1_INDEX_ENTRIES = 2048

    step = 0                                if n_overflow ≤ 2048
    step = ⌈n_overflow / 2048⌉             otherwise

### Complexity

Operation | Time | Notes
--- | --- | ---
`set` / `get` (builder) | O(1) | mmap byte + HashMap
`get` (reader, no overflow) | O(1) | single mmap byte
`get` (reader, with index) | O(log step) | ≤ 2 memory regions
`get` (reader, no index) | O(log n_overflow) | data fits in a few cache lines
`iter()` full scan | O(n + n_overflow) | merge-scan, no binary search
`sum`, distances | O(n) | via `iter()` / `zip(iter(), iter())`
`min` / `max` / `add` / `diff` | O(n) | via `other.iter()` + builder `set`
`close` | O(n_overflow log n_overflow) | sort + sequential write
`open` | O(n_index) | index copy into Vec
`build_from` | O(file_size) + O(n_overflow) | OS copy + HashMap load

## PersistentCompactIntMatrix — column-major directory

### Design

A directory containing `meta.json` and N column files `col_000000.pciv`, `col_000001.pciv`, …, each a `PersistentCompactIntVec`. This is the type used by `LayerData` — a single-column matrix is functionally equivalent to a vector but shares the same interface as multi-column matrices.

    counts/
      meta.json          {"n": <n_slots>, "n_cols": <N>}
      col_000000.pciv
      col_000001.pciv
      ...

### Builder (`PersistentCompactIntMatrixBuilder`)

    struct PersistentCompactIntMatrixBuilder {
        dir:    PathBuf,
        n:      usize,
        n_cols: usize,
    }

`new(n: usize, dir: &Path) -> io::Result<Self>`

Creates the directory (including parents). Does not write `meta.json` yet.

`add_col(&mut self) -> io::Result<PersistentCompactIntVecBuilder>`

Creates `col_NNNNNN.pciv` for the next column and returns its builder. The caller fills the column and calls `builder.close()` before calling `add_col` again.

`close(self) -> io::Result<()>`

Writes `meta.json` with the final `n` and `n_cols`. Must be called after all column builders are closed.

### Reader (`PersistentCompactIntMatrix`)

    struct PersistentCompactIntMatrix {
        cols: Vec<PersistentCompactIntVec>,
        n:    usize,
    }

`open(dir: &Path) -> io::Result<Self>`

Reads `meta.json`, opens all `col_NNNNNN.pciv` files.

`row(slot: usize) -> Box<[u32]>`

Returns the full row: `[col_0[slot], col_1[slot], …, col_{N-1}[slot]]`. One mmap access per column. O(N).

`col(c: usize) -> &PersistentCompactIntVec`

Direct access to a single column for column-oriented operations (distance computations, iteration).

### LayerData implementation

    impl LayerData for PersistentCompactIntMatrix {
        type Item = Box<[u32]>;
        fn open(layer_dir: &Path) -> OLMResult<Self> { /* opens layer_dir/counts/ */ }
        fn read(&self, slot: usize) -> Box<[u32]>    { self.row(slot) }
    }

## Aggregation traits — `obicompactvec::traits`

`PersistentCompactIntMatrix` implements two aggregation traits used by `LayeredStore<S>` for cross-layer and cross-partition distance computations.

### ColumnWeights

    impl ColumnWeights for PersistentCompactIntMatrix {
        fn col_weights(&self) -> Array1<u64>   // = self.sum()
    }

`col_weights()[c]` = sum of all values in column `c` across all slots.

### CountPartials

    impl CountPartials for PersistentCompactIntMatrix {
        // Self-contained partials (additive across layers, no external parameter)
        fn partial_bray(&self)                                      -> Array2<u64>
        fn partial_euclidean(&self)                                 -> Array2<f64>
        fn partial_threshold_jaccard(&self, threshold: u32)         -> (Array2<u64>, Array2<u64>)

        // Normalised partials (require global col_weights across all layers/partitions)
        fn partial_relfreq_bray(&self, global: &Array1<u64>)        -> Array2<f64>
        fn partial_relfreq_euclidean(&self, global: &Array1<u64>)   -> Array2<f64>
        fn partial_hellinger(&self, global: &Array1<u64>)           -> Array2<f64>

        // Provided finalisations (default implementations on the trait)
        fn bray_dist_matrix(&self)                                  -> Array2<f64>
        fn euclidean_dist_matrix(&self)                             -> Array2<f64>
        fn threshold_jaccard_dist_matrix(&self, threshold: u32)     -> Array2<f64>
        fn relfreq_bray_dist_matrix(&self)                          -> Array2<f64>
        fn relfreq_euclidean_dist_matrix(&self)                     -> Array2<f64>
        fn hellinger_dist_matrix(&self)                             -> Array2<f64>
    }

Self-contained partials are additively decomposable: summing `partial_bray()` across all `(partition, layer)` pairs and finalising gives the same result as computing on the combined data.

Normalised partials require the global column weights (sum across all layers and all partitions). The `global` parameter must reflect the complete index, not a per-layer sum. The provided `relfreq_bray_dist_matrix()` etc. call `col_weights()` first (pass 1) then the normalised partial (pass 2); when called on a `LayeredStore<LayeredStore<…>>` these two-pass calls cascade automatically through the blanket impls.

`partial_bray` returns `Array2<u64>` (sum_min only, not a tuple). The denominator is always reconstructible as `col_weights()[i] + col_weights()[j]`.

`partial_threshold_jaccard` returns `(inter, union)` as a pair because `union[i,j]` is not reconstructible from per-column statistics — it depends on both columns simultaneously.