obikmer/DevDocMD/implementation/partition_layer_cache.md

# Partition and layer caching (discussion)

Status: problem confirmed, design direction agreed (2026-08-20). No
implementation started. Ownership split (below) still open.

## The problem

Reading a layer's data (MPHF + matrix) is not free: `MphfLayer::open` mmaps
`mphf.bin` plus (`evidence.bin`/`fingerprint.bin` + `unitigs.bin`), and the
matrix side mmaps `matrix.pbmx`/`matrix.pcmx` (or one file per genome column
if not yet packed). Any code path that reopens a layer per lookup instead of
once per run pays this cost repeatedly.

`obikphylo::siblings::cache::PartitionCache` was built to avoid exactly this
for `build_sibling_annex`/`sibling_annex_stats`: those commands probe many
partitions, once per source layer, over the whole run. Profiling a real run
showed wall-clock time dominated by repeated `open()`/mmap syscalls, not
computation — parallelising the naive per-lookup opens spread the cost
across cores without reducing it. `PartitionCache::build` opens every
partition's every layer once, up front, in parallel, and keeps the handles
alive for the run.

## Three independent implementations of the same bundle

Searching the codebase for "who bundles MPHF + matrix, with per-layer format
auto-detection" turns up three unrelated implementations:

| | lives in | scope | cached? |
|---|---|---|---|
| `Layer<D>` | `obilayeredmap` | one layer, `D` fixed at compile time | held alive by whoever owns the `Layer`, no policy of its own |
| `Mat` | `obikphylo::siblings::cache` | one layer, format resolved per instance from an enum of 3 `Layer<D>` variants | yes, via `PartitionCache` |
| `QueryLayer` | `obikpartitionner::query_layer` | one layer, `(MphfLayer, PersistentBitMatrix\|PersistentCompactIntMatrix)` pair, bypasses `Layer<D>` entirely | **no** — opened fresh inside `query_partition_with` on every call |

`query_partition_with` is `obikmer query`'s normal query path — the one
most exposed to repeated cross-partition lookups — and it is the one with
no cache at all. `obikphylo` built a cache first only because sibling-annex
construction hits the cost hardest, not because the need is sibling-specific.

## The gap in `obilayeredmap`'s existing cache

`obilayeredmap::LayeredMap<D>` already caches correctly at the granularity
of one partition: `open(root)` opens every layer once, keeps `Vec<Layer<D>>`
alive for the `LayeredMap`'s lifetime. But it is monomorphic — every layer
in the `Vec` must share the same concrete `D`. In practice this is false:
layers in the same partition are packed independently over time (`pack
--sparse` converts one layer's presence matrix at a time), so a partition
can genuinely mix `PersistentBitMatrix`-backed and
`PersistentSparseBitMatrix`-backed presence layers. `LayeredMap<D>` cannot
represent that mix — `Mat`'s 3-way enum exists specifically to work around
this gap, one crate away from where the gap actually is.

## Resource cost: mmap does not hold a file descriptor

Before deciding how many layers/partitions a cache may hold open
simultaneously, the binding constraint needs to be identified correctly.

Confirmed against upstream documentation, not inferred from behaviour:

> "After the mmap() call has returned, the file descriptor, fd, can be
> closed immediately without invalidating the mapping."
> — [mmap(2), man7.org](https://man7.org/linux/man-pages/man2/mmap.2.html)

> "The close(2) function does not unmap pages"
> — [mmap(2), Apple Developer](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mmap.2.html)

> "A file backed Mmap ... will remain valid even after the File is dropped.
> ... the Mmap handle is completely independent of the File used to create
> it."
> — [memmap2::Mmap, docs.rs](https://docs.rs/memmap2/latest/memmap2/struct.Mmap.html)

Every read-only mmap in this codebase already follows this: `Mmap::map(&File::open(path)?)?`
— the `File` is a temporary, dropped (fd closed) immediately after the
mapping is established; every persistent struct (`PersistentBitVec`,
`PersistentCompactIntVec`, `PackedBitMatrix`, `Evidence`, `FingerprintVec`,
...) stores only the `Mmap`, never the `File`. So a cache built on these
types does **not** consume the process's open-file-descriptor budget
(`ulimit -n`, notoriously low by default on macOS) proportionally to how
many mmapped files it holds.

It does consume a different resource — the process's virtual-memory mapping
table (one entry per active `mmap()` region). Linux exposes this as
`vm.max_map_count` (default 65530). No documented macOS equivalent (fixed
numeric ceiling) was found; the constraint there appears to be virtual
address space rather than an explicit mapping counter, but this is not
sourced and should not be assumed. This is the resource actually worth
measuring before deciding on cache size, not fd count — and it is why
*packing* (`matrix.pbmx`/`matrix.pcmx`, one mmap for all columns) matters
independently of any caching decision: an unpacked `Columnar` matrix opens
one mmap **per genome column**, multiplying the mapping count a cache would
have to hold by `n_genomes`.

## Layering: who owns what

Established in discussion, not yet coded:

- `obilayeredmap` operates within a single partition's index root; it has
  no notion of "partition" (confirmed independently while fixing
  `open_data`/`layer_dir` — see git history around 2026-08-20). One
  `LayeredMap<D>` (or its heterogeneous-`D` successor) = one partition.
- Nobody currently owns "the collection of partitions" as reusable state.
  `obikpartitionner::KmerPartition` is the closest candidate — it already
  owns `n_partitions()`/`part_dir(i)` — but today it is a pure
  config/path resolver, not a state holder: `dereplicate`, `count_kmer`,
  `obikindex`'s `distance.rs`/`stats.rs`, and
  `obikphylo::PartitionCache::build` each independently write their own
  `(0..n_partitions).into_par_iter().map(...)` loop and open what they need
  from scratch.

## Direction agreed, not yet implemented

1. A heterogeneous-format layer type in `obilayeredmap` (name TBD),
   generalising `Mat`'s 3-way enum + auto-detection — `obilayeredmap`
   already implements `LayerData` for all three concrete matrix types
   involved, so it already has the knowledge `Mat` needed and had to
   duplicate.
2. A cache spanning multiple partitions, keeping (2)'s layer type open per
   partition per layer, for the whole lifetime of a long-running command —
   generalising `PartitionCache` minus its sibling-specific parts
   (`fast_mode`, `find_presence_batch`) — living in `obikpartitionner`
   (owner of the partition dimension) and built on top of (1).
3. `obikpartitionner::query_partition_with` — the normal query path,
   currently uncached — becomes a consumer of (2), not just
   `obikphylo::PartitionCache`.

Open before implementing: exact API shape of (1) and (2), whether `Mat` is
deleted outright or kept as a thin sibling-specific wrapper, and whether (2)
needs an eviction policy or can simply hold every partition open for the
process lifetime (revisit once the VM-mapping-count question above has a
real number behind it for this codebase's scale).