obikmer/DevDocMD/implementation/partition_layer_cache.md

# Partition and layer caching (discussion)

Status: problem confirmed, design direction agreed (2026-08-20). (1)/(2)
themselves not started; ownership split (below) still open. Preparatory
encapsulation work (partition/layer path and metadata accessors on
`KmerPartition`) landed the same day — see "Preparatory work done" below.

## Type-to-concept mapping: Index / Partition / Layer

The conceptual nesting `Index { Partition { Layer { MPHF, Evidence, Matrix
} } } }` does **not** have one Rust type per level — worth stating
explicitly, since two of the names below are misleading.

- **Index** = `obikindex::KmerIndex` — `{ root_path, meta: IndexMeta,
  partition: KmerPartition }`. The field is named `partition` (singular),
  but it holds the *whole* multi-partition structure below — the name
  suggests "one partition", the value is all of them.
- **Partition, the collection (not one partition)** =
  `obikpartition::KmerPartition`. Despite the singular name, this owns
  *every* partition of the index: `root_path`, `n_partitions`, and
  per-partition accessors that all take an explicit index `i`
  (`partition_dir(i)`, `index_dir(i)`, `layer_dir(i, l)`,
  `partition_meta(i)`, `n_layers(i)`, `index_mode(i)` — see the
  `part_dir`/`layer_dir` and `PartitionMeta`-encapsulation work above).
  It never holds one partition's layers open in memory — no `Vec<Layer<D>>`
  here, only path arithmetic and metadata reads. A more honest name would
  be `KmerPartitionSet` or `KmerPartitioner`; renaming is out of scope for
  now, just worth knowing it's not "a partition."
- **Partition, one of them** = **no dedicated type exists today**. The
  structural equivalent of "one partition, its layers held open" is
  `obilayeredmap::LayeredMap<D>` — `{ root, meta: PartitionMeta, layers:
  Vec<Layer<D>> }` — but `LayeredMap` itself doesn't know it's playing that
  role: it operates purely on whatever `root` path it's given and has no
  notion of `KmerPartition`, `n_partitions`, or a partition index `i` (see
  "Layering: who owns what" below — established independently while
  fixing `open_data`/`layer_dir`). Every caller that wants "one partition,
  opened" builds the path itself
  (`kmer_partition.index_dir(i)`/`.layer_dir(i, l)`) and either passes it
  to `LayeredMap::open` or — as `obikphylo::siblings::cache` does —
  bypasses `LayeredMap` entirely and opens each `Layer<D>` directly.
- **Layer** = `obilayeredmap::Layer<D>` — `{ mphf: MphfLayer, data: D }`.
  - **MPHF** = `MphfLayer.mphf: MemCase<MphfEps>` — kmer → slot.
  - **Evidence** = `MphfLayer.ev: LayerEvidence` (`Exact`/`Approx`/
    `Hybrid` — `evidence.bin`/`fingerprint.bin`; see `EvidenceKind`).
  - **Matrix** = `Layer<D>.data: D` — `PersistentBitMatrix` /
    `PersistentCompactIntMatrix` / `PersistentSparseBitMatrix` / `()`.

Today's actual nesting, in Rust terms:

```
KmerIndex
 └─ partition: KmerPartition                 (all N partitions — paths + metadata only)
     └─ (opened on demand, per i)  LayeredMap<D>   ← "one partition", unnamed as such
         └─ layers: Vec<Layer<D>>
             └─ Layer<D> { mphf: MphfLayer, data: D }
                 ├─ mphf.mphf   → MPHF
                 ├─ mphf.ev     → Evidence
                 └─ data        → Matrix
```

`obikphylo::siblings::cache::PartitionCache` — the thing (2) is meant to
generalise — skips the middle of this nesting entirely: it doesn't build a
`Vec<LayeredMap<D>>`, it builds `mats: Vec<Vec<Mat>>` directly — outer
index = partition `i` (via `KmerPartition`), inner index = layer `l` (via
`Layer<D>`/`Mat`) — reconstructing "one partition, its layers open" as a
bare nested `Vec` because no owning type for that concept exists to reuse.
That gap is exactly what (2) needs to fill, and (1) is exactly what would
let a per-partition type hold layers of mixed `D` (today `LayeredMap<D>`
can't, being monomorphic — see "The gap in `obilayeredmap`'s existing
cache" below).

## Major restructuring (2026-08-20): `KmerPartitions` merged into `KmerIndex`

Prompted by a direct question: why keep `KmerIndex`/`KmerPartitions` split
when, one level down, `KmerPartitions` is going to directly hold
`Vec<KmerPartition>` rather than being split again into
"collection-holder" + "collection"? Investigating the actual justification
("`KmerPartitions` has an independent lifecycle, used before an index
exists") turned out to be **false** — `KmerPartitions::create` was called
in exactly one place, inside `KmerIndex::create`, and every
`open_with_config` reopen outside `KmerIndex`'s own constructors was a
redundant re-derivation of a `KmerPartitions` already reachable via
`index.partition()` (the exact kind of duplication this whole doc has been
tracking). Once that was gone, so was the reason to keep them separate.

Second correction, from the same conversation: `obikpartitionner` had
accumulated query/merge/select/rebuild/dump/distance logic that has
nothing to do with partitioning super-kmers — it operates on *layers*,
which don't exist yet at the phase `obikpartitionner` is actually
responsible for (scatter → dereplicate → count, all pre-layer). That
logic moved to `obikindex`, which already depends on `obilayeredmap` and
never needed `obikpartitionner` for it. No crate-dependency inversion was
needed — `obikindex → obikpartitionner` stays the same direction as before.

**Result:**
- `obikpartitionner` (renamed back from `obikpartition`) now contains only
  `PartitionRouter` (superkmer routing: `write`/`write_batch`/`flush`/
  `close`, `dereplicate`, `count_kmer`, `KmerSpectrum`) and the
  `partition_dir(root, i)` naming primitive both `PartitionRouter` and
  `KmerIndex` build on. `KmerPartitions` no longer exists as a type.
- `KmerIndex` (`obikindex`) absorbed `KmerPartitions`'s read-side entirely:
  `partition_dir`/`index_dir`/`layer_dir`/`partition_meta`/`n_layers`/
  `partition_mode`/`n_partitions` (the last now derived from
  `2^config.n_bits`, no longer a stored, independently-set duplicate field
  — `kmer_size`/`minimizer_size` used to be double-stored, in both
  `KmerPartitions` and `IndexMeta.config`, a latent-drift risk flagged
  earlier in this doc; now single-sourced from `IndexMeta.config`). Seven
  whole files moved from `obikpartitionner` into `obikindex` verbatim as
  `impl KmerIndex` blocks, kept as separate files (not merged into
  existing same-topic files): `index_layer.rs`, `query_layer.rs`,
  `merge_layer/`, `select_layer.rs`, `rebuild_layer.rs`, `dump_layer.rs`,
  plus `distance.rs`'s `count_store`/`presence_store` (renamed
  `matrix_store.rs` to avoid colliding with `obikindex`'s own pre-existing
  `distance.rs`), and their shared support (`common.rs`'s `load_meta`/
  `olm_to_sk`, `filter.rs`, `graph_pipeline.rs`).
- `obikphylo::siblings::cache::PartitionCache::build` now takes `&KmerIndex`
  directly instead of a separately-opened `&KmerPartitions` — this deleted
  the redundant-reopen pattern at all 8 call sites
  (`alignment`/`build`/`cardinality`/`distance`/`entropy`×2/
  `sankoff_bundle`/`stats`), the same bug flagged earlier in this
  conversation as a side effect of investigating the false "independent
  lifecycle" claim.
- `KmerIndex::partition()`/`partition_mut()` are gone; `scatter()`
  (`obikmer`) and any write-side code get a transient `PartitionRouter` via
  `KmerIndex::partition_router()`.
- A real bug caught by the test suite during this move:
  `PartitionRouter::open` initially defaulted to `closed: true` (inherited
  from `KmerPartitions::open_with_config`'s old read-only-reopen
  semantics), which broke every write through a router obtained via
  `partition_router()`. Fixed — `PartitionRouter` is exclusively a
  write/processing tool now, so `open` always starts open.

Full workspace test suite green (0 failed) after, including all 27
`obikphylo::siblings` tests.

## (1) done (2026-08-20): `Layer` is now the heterogeneous handle, `Mat` is gone

Resolved the naming question left open above. `Layer<D>` (the old
generic/monomorphic type) renamed to `TypedLayer<D>` throughout
(`obilayeredmap`, `obikindex`, `obikphylo` — 12 files, mechanical) to free
`Layer` for the type that's actually meant to be everyone's default
handle. `obilayeredmap::content_layer::Layer` (re-exported at the crate
root) is that type — `Count(TypedLayer<PersistentCompactIntMatrix>)`/
`Presence(TypedLayer<PersistentBitMatrix>)`, `Layer::open` doing the same
disk probe `Mat::open` used to, `find_slot`/`index_batch`/`n_cols`/
`fill_sub_matrix_carries` dispatching per variant exactly as `Mat` did.

`obikphylo::siblings::cache::Mat` deleted outright — `PartitionCache` now
holds `Vec<Vec<obilayeredmap::Layer>>` directly. The one sibling-specific
method `Mat` carried (`iter_minorants_batch`) is not on `obilayeredmap::
Layer` (phylo concepts don't belong in `obilayeredmap`) — it's an
`impl SiblingLayerExt for obilayeredmap::Layer` in `iter.rs`, dispatching
to each variant's existing `impl<D: LayerData> SiblingLayerExt for
TypedLayer<D>`.

Full workspace suite green (0 failed) after, including all 27
`obikphylo::siblings` tests.

Still not built: (2) — `KmerPartition` (singular, one partition's open
`Vec<Layer>`) and a multi-partition cache in `obikpartitionner` to replace
`obikphylo::siblings::cache::PartitionCache` and `obikindex::query_layer`'s
still-separate `QueryLayer` (which still independently bundles MPHF+matrix,
2-way not using `Layer` at all). Both remaining consumers now sit one
`Layer::open` call away from unifying onto (2) once it exists.

## The problem

Reading a layer's data (MPHF + matrix) is not free: `MphfLayer::open` mmaps
`mphf.bin` plus (`evidence.bin`/`fingerprint.bin` + `unitigs.bin`), and the
matrix side mmaps `matrix.pbmx`/`matrix.pcmx` (or one file per genome column
if not yet packed). Any code path that reopens a layer per lookup instead of
once per run pays this cost repeatedly.

`obikphylo::siblings::cache::PartitionCache` was built to avoid exactly this
for `build_sibling_annex`/`sibling_annex_stats`: those commands probe many
partitions, once per source layer, over the whole run. Profiling a real run
showed wall-clock time dominated by repeated `open()`/mmap syscalls, not
computation — parallelising the naive per-lookup opens spread the cost
across cores without reducing it. `PartitionCache::build` opens every
partition's every layer once, up front, in parallel, and keeps the handles
alive for the run.

## Three independent implementations of the same bundle

Searching the codebase for "who bundles MPHF + matrix, with per-layer format
auto-detection" turns up three unrelated implementations:

| | lives in | scope | cached? |
|---|---|---|---|
| `Layer<D>` | `obilayeredmap` | one layer, `D` fixed at compile time | held alive by whoever owns the `Layer`, no policy of its own |
| `Mat` | `obikphylo::siblings::cache` | one layer, format resolved per instance from an enum of 3 `Layer<D>` variants | yes, via `PartitionCache` |
| `QueryLayer` | `obikpartition::query_layer` | one layer, `(MphfLayer, PersistentBitMatrix\|PersistentCompactIntMatrix)` pair, bypasses `Layer<D>` entirely | **no** — opened fresh inside `query_partition_with` on every call |

`query_partition_with` is `obikmer query`'s normal query path — the one
most exposed to repeated cross-partition lookups — and it is the one with
no cache at all. `obikphylo` built a cache first only because sibling-annex
construction hits the cost hardest, not because the need is sibling-specific.

## The gap in `obilayeredmap`'s existing cache

`obilayeredmap::LayeredMap<D>` already caches correctly at the granularity
of one partition: `open(root)` opens every layer once, keeps `Vec<Layer<D>>`
alive for the `LayeredMap`'s lifetime. But it is monomorphic — every layer
in the `Vec` must share the same concrete `D`. In practice this is false:
layers in the same partition are packed independently over time (`pack
--sparse` converts one layer's presence matrix at a time), so a partition
can genuinely mix `PersistentBitMatrix`-backed and
`PersistentSparseBitMatrix`-backed presence layers. `LayeredMap<D>` cannot
represent that mix — `Mat`'s 3-way enum exists specifically to work around
this gap, one crate away from where the gap actually is.

## Resource cost: mmap does not hold a file descriptor

Before deciding how many layers/partitions a cache may hold open
simultaneously, the binding constraint needs to be identified correctly.

Confirmed against upstream documentation, not inferred from behaviour:

> "After the mmap() call has returned, the file descriptor, fd, can be
> closed immediately without invalidating the mapping."
> — [mmap(2), man7.org](https://man7.org/linux/man-pages/man2/mmap.2.html)

> "The close(2) function does not unmap pages"
> — [mmap(2), Apple Developer](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mmap.2.html)

> "A file backed Mmap ... will remain valid even after the File is dropped.
> ... the Mmap handle is completely independent of the File used to create
> it."
> — [memmap2::Mmap, docs.rs](https://docs.rs/memmap2/latest/memmap2/struct.Mmap.html)

Every read-only mmap in this codebase already follows this: `Mmap::map(&File::open(path)?)?`
— the `File` is a temporary, dropped (fd closed) immediately after the
mapping is established; every persistent struct (`PersistentBitVec`,
`PersistentCompactIntVec`, `PackedBitMatrix`, `Evidence`, `FingerprintVec`,
...) stores only the `Mmap`, never the `File`. So a cache built on these
types does **not** consume the process's open-file-descriptor budget
(`ulimit -n`, notoriously low by default on macOS) proportionally to how
many mmapped files it holds.

It does consume a different resource — the process's virtual-memory mapping
table (one entry per active `mmap()` region). Linux exposes this as
`vm.max_map_count` (default 65530). No documented macOS equivalent (fixed
numeric ceiling) was found; the constraint there appears to be virtual
address space rather than an explicit mapping counter, but this is not
sourced and should not be assumed. This is the resource actually worth
measuring before deciding on cache size, not fd count — and it is why
*packing* (`matrix.pbmx`/`matrix.pcmx`, one mmap for all columns) matters
independently of any caching decision: an unpacked `Columnar` matrix opens
one mmap **per genome column**, multiplying the mapping count a cache would
have to hold by `n_genomes`.

## Layering: who owns what

Established in discussion, not yet coded:

- `obilayeredmap` operates within a single partition's index root; it has
  no notion of "partition" (confirmed independently while fixing
  `open_data`/`layer_dir` — see git history around 2026-08-20). One
  `LayeredMap<D>` (or its heterogeneous-`D` successor) = one partition.
- Nobody currently owns "the collection of partitions" as reusable state.
  `obikpartition::KmerPartition` is the closest candidate — it already
  owns `n_partitions()`/`part_dir(i)` — but today it is a pure
  config/path resolver, not a state holder: `dereplicate`, `count_kmer`,
  `obikindex`'s `distance.rs`/`stats.rs`, and
  `obikphylo::PartitionCache::build` each independently write their own
  `(0..n_partitions).into_par_iter().map(...)` loop and open what they need
  from scratch.

## Direction agreed, not yet implemented

1. A heterogeneous-format layer type in `obilayeredmap` (name TBD),
   generalising `Mat`'s 3-way enum + auto-detection — `obilayeredmap`
   already implements `LayerData` for all three concrete matrix types
   involved, so it already has the knowledge `Mat` needed and had to
   duplicate.
2. A cache spanning multiple partitions, keeping (2)'s layer type open per
   partition per layer, for the whole lifetime of a long-running command —
   generalising `PartitionCache` minus its sibling-specific parts
   (`fast_mode`, `find_presence_batch`) — living in `obikpartition`
   (owner of the partition dimension) and built on top of (1).
3. `obikpartition::query_partition_with` — the normal query path,
   currently uncached — becomes a consumer of (2), not just
   `obikphylo::PartitionCache`.

Open before implementing: exact API shape of (1) and (2), whether `Mat` is
deleted outright or kept as a thin sibling-specific wrapper, and whether (2)
needs an eviction policy or can simply hold every partition open for the
process lifetime (revisit once the VM-mapping-count question above has a
real number behind it for this codebase's scale).

## Preparatory work done (2026-08-20)

Groundwork for (1)/(2), landed ahead of the design itself:

- `KmerPartition` (`obikpartition`) gained `partition_dir`/`index_dir`/
  `layer_dir` as the single source of truth for a partition's on-disk
  layout, replacing per-module duplicated `const INDEX_SUBDIR: &str =
  "index"` (7 copies) and ad hoc path joins — including one found
  duplicated *inside `KmerPartition` itself* (`ensure_writer` rebuilt
  `part_dir`'s own logic by hand).
- `KmerPartition` gained `partition_meta`/`n_layers`/`index_mode`, wrapping
  `obilayeredmap::meta::PartitionMeta::load` (via the existing
  `common::load_meta`, which also recovers indexes built before
  `meta.json` existed). Before this, `obikphylo` and `obikindex` imported
  `obilayeredmap::meta::PartitionMeta` directly and called `::load()`
  themselves at 21 call sites, each redoing its own error-mapping —
  every one of those crates knew the on-disk metadata format instead of
  going through an interface. Fixed everywhere except one remaining spot
  (below). Caught as a side effect: `dump_layer.rs`/`query_layer.rs` had
  been calling `PartitionMeta::load` directly, bypassing `load_meta`
  entirely — they never got the missing-`meta.json` recovery the other
  callers did.
- Layer introspection API discussed but **not yet implemented** — three
  axes, deliberately kept separate after an initial draft conflated them:
  - `LayerContent { Count, Presence }` — what the layer stores; a `const`
    on `LayerData` (compile-time, zero-cost), not a runtime field.
  - `StorageKind { Implicit, Columnar, Packed, Sparse }` — how it's
    stored; only meaningful for `D` that actually carry data (`Layer<()>`
    has neither this nor `LayerContent` — it's a write-time-only state,
    never a queryable content: once a layer is closed, "no matrix file"
    reads back as `Presence`/`Implicit` via `PersistentBitMatrix::open`'s
    own fallback, not as some third "empty" content).
  - `EvidenceKind { Exact, Approx, Hybrid }` — from `MphfLayer`'s own
    already-in-memory `LayerEvidence` discriminant.
  - Not all `(LayerContent, StorageKind)` pairs are legal: `Count` never
    has `Implicit` or `Sparse`.

**Implemented (2026-08-20).** `LayerContent`/`StorageKind`/`EvidenceKind`
now exist, each with two forms:
- A runtime accessor on an already-open value (`Layer<D>::content()`/
  `storage_kind()`/`evidence_kind()`, `PersistentBitMatrix::storage_kind()`,
  `PersistentCompactIntMatrix::storage_kind()`, `MphfLayer::evidence_kind()`)
  — reads a discriminant already in memory, zero disk access.
- A lightweight `detect()`/`detect_storage()` disk probe that mirrors the
  corresponding `open()`'s own priority order by hand (file-existence
  checks only, no mmap) — usable *before* committing to a `D`, unlike the
  runtime accessors. Exposed per-layer on `LayeredMap<D>` as
  `detect_layer_content`/`detect_layer_storage`/`detect_layer_evidence`
  (work regardless of `D`, since they only use `self.root` + the layer
  index).

`StorageKind` lives in `obicompactvec` (owner of `PersistentBitMatrix`/
`PersistentCompactIntMatrix`); `LayerContent`/`EvidenceKind` live in
`obilayeredmap`. `HasLayerContent`/`HasStorageKind` gate `Layer<()>` out of
`content()`/`storage_kind()` (no matrix, nothing to report), matching the
"empty is transitional" conclusion above. 42 new tests across
`obilayeredmap`'s `tests/layer.rs` and `tests/map.rs`; full workspace
suite green (0 failed) after.

Not done: these `detect()` probes don't yet replace `Mat::open`'s or
`QueryLayer::open`'s own hand-rolled equivalents (still duplicated content/
storage decisions, now a *third* copy of the same logic to keep in sync)
— that consolidation is (1)/(2)'s job, not this prep step's.

## One bug found while reading around this (signalled, not fixed); one earlier claim retracted

- `obicompactvec::bitmatrix::sparse.rs`'s module doc says
  "Not used by any production code path yet" — false since `obikmer pack
  --sparse` (`cmd/pack/mod.rs`) is wired to `pack_sparse_bit_matrix` and
  `Mat::open` already reads the result back in the sibling-annex path.
  Stale comment, not corrected.
- **Retracted (2026-08-20)**: an earlier pass through this doc claimed
  `obikpartition::query_layer::QueryLayer::open` had no sparse-format
  detection and would silently corrupt reads on a `pack --sparse`d layer.
  False — `PersistentBitMatrix` (`obicompactvec::bitmatrix::persistent`)
  is a 4-way enum (`Columnar`/`Packed`/`Sparse`/`Implicit`), not 3-way as
  first read; its `open()` already detects `Sparse` via
  `presence/sparse_meta.json`, and every method on the type (`row`,
  `fill_row`, `nonzero_iter`, …) already dispatches all 4 arms.
  `QueryLayer::open`'s `PersistentBitMatrix::open(layer_dir)` call was
  never the bug. Root cause of the false claim: a `grep -n
  "Implicit\|Columnar\|Packed"` used to read the enum definition silently
  skipped the `Sparse(...)` line because it matched none of those three
  words — a self-inflicted blind spot from a filtered read, not a fact
  about the code. Lesson: for a `pub enum` whose variant list matters,
  read the definition unfiltered, don't grep for the variant names you
  expect to find.
- One real consequence of that same correction, **fixed (2026-08-20)**:
  `obikphylo::siblings::cache::Mat::SparsePresence(Layer<
  PersistentSparseBitMatrix>)` was redundant — `Mat::Presence(Layer<
  PersistentBitMatrix>)` alone already handles sparse layers
  transparently, since `PersistentBitMatrix` absorbs `Sparse` internally.
  Removed the variant, the `presence/is_multi.prsb` probe in `Mat::open`
  (now just opens `Layer::<PersistentBitMatrix>` unconditionally for the
  non-count case — sparse-vs-dense is `PersistentBitMatrix::open`'s own
  concern), and every now-single-armed match in `find_slot`/`index_batch`/
  `iter_minorants_batch`/`n_cols`/`fill_sub_matrix_carries`. Full
  workspace test suite green after, including the 27 `obikphylo::siblings`
  tests that exercise `pack_matrices(true)`/sparse through `Mat`.

## Remaining instance of the PartitionMeta-encapsulation problem

`obikphylo::siblings::family_scan::scan_layer_families` still re-derives
`index_dir` from `layer_dir.parent()` and calls `PartitionMeta::load`
itself, purely to get `.mode` for `Mat::open`. Fixing it the way the 21
other call sites were fixed needs more than a 1:1 swap: `scan_layer_families`
only receives a bare `layer_dir: &Path`, not a `(partition, part, layer)`
triple, and its single upstream source of layer paths,
`sibling_layer_dirs`, returns a flat `Vec<PathBuf>` with the partition/layer
indices already discarded. Fixing it properly means either having
`sibling_layer_dirs` return `(PathBuf, IndexMode)` (or `(part, layer)`)
pairs, or threading `&KmerPartition` + indices through instead of paths —
and touching every one of `scan_layer_families`'s 8 callers (`distance.rs`,
`alignment.rs`, `cardinality.rs`, `entropy.rs` ×2, `sankoff_bundle.rs` ×2,
`stats.rs`). Left alone this round; worth doing as part of the same pass
that builds (1)/(2), since those callers are exactly the sibling-annex
consumers (2) is meant to serve.