|
|
|
@@ -1,74 +1,120 @@
|
|
|
|
|
# Partition and layer caching (discussion)
|
|
|
|
|
|
|
|
|
|
Status: problem confirmed, design direction agreed (2026-08-20). (1)/(2)
|
|
|
|
|
themselves not started; ownership split (below) still open. Preparatory
|
|
|
|
|
encapsulation work (partition/layer path and metadata accessors on
|
|
|
|
|
`KmerPartition`) landed the same day — see "Preparatory work done" below.
|
|
|
|
|
Status (2026-08-20, latest pass): (1) done — `obilayeredmap::Layer`
|
|
|
|
|
exists, `Mat` is gone. (1b) done — `Layer::Empty`, the first non-ready
|
|
|
|
|
state, added (panics on every read method). (2) — `KmerPartition` + a
|
|
|
|
|
multi-partition cache — **not started**, precisely specified below after
|
|
|
|
|
a real mix-up: an earlier
|
|
|
|
|
version of this doc used the name `KmerPartition` (singular) for what was
|
|
|
|
|
actually the *collection* type (later renamed `KmerPartitions`, later
|
|
|
|
|
merged into `KmerIndex` — see "Major restructuring" below), and never
|
|
|
|
|
retracted that usage before this section was rewritten. An agent working
|
|
|
|
|
from that stale wording built the wrong thing. **If you are about to
|
|
|
|
|
implement (2), read "Definitions: `obikpartition` and `KmerPartition`"
|
|
|
|
|
below — it is the current, authoritative naming — before touching any
|
|
|
|
|
other section of this file, some of which still describe superseded
|
|
|
|
|
states of the code and are kept only as dated history.**
|
|
|
|
|
|
|
|
|
|
## Definitions: `obikpartition` and `KmerPartition` (not yet created)
|
|
|
|
|
|
|
|
|
|
**`obikpartition`** — a new workspace crate, not created yet. Holds the
|
|
|
|
|
**Partition** tier of the `Index { Partition { Layer } }` model, the same
|
|
|
|
|
way `obilayeredmap` already holds the **Layer** tier as its own crate
|
|
|
|
|
rather than living inside `obikindex`. Depends only on `obilayeredmap`
|
|
|
|
|
(for `Layer`) and lower (`obikseq`, `obiskio`). Does **not** depend on
|
|
|
|
|
`obikindex`, `obikpartitionner`, or `obikphylo`. Dependency direction:
|
|
|
|
|
`obikindex → obikpartition → obilayeredmap`; `obikphylo → obikindex`
|
|
|
|
|
(and/or `obikpartition` directly if it ends up needing it without going
|
|
|
|
|
through `KmerIndex`).
|
|
|
|
|
|
|
|
|
|
**`KmerPartition`** (singular) — the one type this crate exists for.
|
|
|
|
|
Represents **one partition's already-open layers** — a read cache, built
|
|
|
|
|
once per partition and held for the run, not rebuilt per lookup. Shape:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
pub struct KmerPartition {
|
|
|
|
|
layers: Vec<obilayeredmap::Layer>,
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Nothing else. In particular:
|
|
|
|
|
- **No path computation.** `KmerPartition::open` takes an already-resolved
|
|
|
|
|
`index_dir: &Path` (plus `mode: &IndexMode`, `n_layers: usize`,
|
|
|
|
|
`with_counts: bool` — whatever it needs, as plain arguments), the same
|
|
|
|
|
discipline `obikpartitionner::PartitionRouter::open` already follows.
|
|
|
|
|
Computing `index_dir`/`layer_dir` from a partition number is
|
|
|
|
|
`KmerIndex`'s job (`obikindex`, which owns that already — see "Major
|
|
|
|
|
restructuring" below); `KmerPartition` never reaches back into
|
|
|
|
|
`KmerIndex` to get it (would require `obikpartition → obikindex`, the
|
|
|
|
|
wrong direction).
|
|
|
|
|
- **No routing/write state.** Writing raw superkmers, `dereplicate`,
|
|
|
|
|
`count_kmer` stay in `obikpartitionner::PartitionRouter` — a completely
|
|
|
|
|
different crate, a completely different phase (pre-layer, whereas
|
|
|
|
|
`KmerPartition` only makes sense once layers exist).
|
|
|
|
|
- **No multi-partition collection baked in.** `KmerPartition` is *one*
|
|
|
|
|
partition. Whatever ends up caching several of them (replacing
|
|
|
|
|
`obikphylo::siblings::cache::PartitionCache`'s `Vec<Vec<Layer>>` and
|
|
|
|
|
`obikindex::query_layer`'s per-call reopen) holds `Vec<KmerPartition>` —
|
|
|
|
|
that collection can live in `obikpartition` too, or in `obikindex`
|
|
|
|
|
alongside `KmerIndex`; not yet decided, secondary to getting
|
|
|
|
|
`KmerPartition` itself right first.
|
|
|
|
|
|
|
|
|
|
**Do not confuse with `KmerPartitions`** (plural — note the `s`): that
|
|
|
|
|
type is **gone**. It used to be `obikpartitionner`'s (nee `obikpartition`,
|
|
|
|
|
briefly — see the crate-rename history below, itself a separate rename
|
|
|
|
|
from this one) do-everything struct — routing, dereplication, *and* path
|
|
|
|
|
lookups all in one. It was deleted on 2026-08-20; its read-side (paths,
|
|
|
|
|
`n_layers`, `partition_meta`) was absorbed into `KmerIndex`, its
|
|
|
|
|
write-side became `PartitionRouter`. `KmerPartition` (this section,
|
|
|
|
|
singular, no final `s`) is a brand-new type with a different job, in a
|
|
|
|
|
crate that doesn't exist yet — not a revival, not a renaming, of
|
|
|
|
|
`KmerPartitions`.
|
|
|
|
|
|
|
|
|
|
## Type-to-concept mapping: Index / Partition / Layer
|
|
|
|
|
|
|
|
|
|
The conceptual nesting `Index { Partition { Layer { MPHF, Evidence, Matrix
|
|
|
|
|
} } } }` does **not** have one Rust type per level — worth stating
|
|
|
|
|
explicitly, since two of the names below are misleading.
|
|
|
|
|
} } } }`, current state:
|
|
|
|
|
|
|
|
|
|
- **Index** = `obikindex::KmerIndex` — `{ root_path, meta: IndexMeta,
|
|
|
|
|
partition: KmerPartition }`. The field is named `partition` (singular),
|
|
|
|
|
but it holds the *whole* multi-partition structure below — the name
|
|
|
|
|
suggests "one partition", the value is all of them.
|
|
|
|
|
- **Partition, the collection (not one partition)** =
|
|
|
|
|
`obikpartition::KmerPartition`. Despite the singular name, this owns
|
|
|
|
|
*every* partition of the index: `root_path`, `n_partitions`, and
|
|
|
|
|
per-partition accessors that all take an explicit index `i`
|
|
|
|
|
- **Index** = `obikindex::KmerIndex` — `{ root_path, meta: IndexMeta }`.
|
|
|
|
|
Also directly exposes the partition-path/metadata accessors
|
|
|
|
|
(`partition_dir(i)`, `index_dir(i)`, `layer_dir(i, l)`,
|
|
|
|
|
`partition_meta(i)`, `n_layers(i)`, `index_mode(i)` — see the
|
|
|
|
|
`part_dir`/`layer_dir` and `PartitionMeta`-encapsulation work above).
|
|
|
|
|
It never holds one partition's layers open in memory — no `Vec<Layer<D>>`
|
|
|
|
|
here, only path arithmetic and metadata reads. A more honest name would
|
|
|
|
|
be `KmerPartitionSet` or `KmerPartitioner`; renaming is out of scope for
|
|
|
|
|
now, just worth knowing it's not "a partition."
|
|
|
|
|
- **Partition, one of them** = **no dedicated type exists today**. The
|
|
|
|
|
structural equivalent of "one partition, its layers held open" is
|
|
|
|
|
`obilayeredmap::LayeredMap<D>` — `{ root, meta: PartitionMeta, layers:
|
|
|
|
|
Vec<Layer<D>> }` — but `LayeredMap` itself doesn't know it's playing that
|
|
|
|
|
role: it operates purely on whatever `root` path it's given and has no
|
|
|
|
|
notion of `KmerPartition`, `n_partitions`, or a partition index `i` (see
|
|
|
|
|
"Layering: who owns what" below — established independently while
|
|
|
|
|
fixing `open_data`/`layer_dir`). Every caller that wants "one partition,
|
|
|
|
|
opened" builds the path itself
|
|
|
|
|
(`kmer_partition.index_dir(i)`/`.layer_dir(i, l)`) and either passes it
|
|
|
|
|
to `LayeredMap::open` or — as `obikphylo::siblings::cache` does —
|
|
|
|
|
bypasses `LayeredMap` entirely and opens each `Layer<D>` directly.
|
|
|
|
|
- **Layer** = `obilayeredmap::Layer<D>` — `{ mphf: MphfLayer, data: D }`.
|
|
|
|
|
`partition_meta(i)`, `n_layers(i)`, `partition_mode(i)`,
|
|
|
|
|
`n_partitions()`) since `KmerPartitions` merged into it (see "Major
|
|
|
|
|
restructuring" below) — `KmerIndex` today *is* "index + collection of
|
|
|
|
|
partitions' paths & metadata," just without a `Vec` of open layers.
|
|
|
|
|
- **Partition, the collection** = no dedicated type today; the closest
|
|
|
|
|
thing is `KmerIndex` itself (previous bullet). Once `KmerPartition`
|
|
|
|
|
(singular, see Definitions above) exists, a `Vec<KmerPartition>`
|
|
|
|
|
somewhere would be this — still open, see "Direction agreed" below.
|
|
|
|
|
- **Partition, one of them** = `obikpartition::KmerPartition` — **to be
|
|
|
|
|
built**, see Definitions above. Nothing plays this role today;
|
|
|
|
|
`obikphylo::siblings::cache::PartitionCache` and
|
|
|
|
|
`obikindex::query_layer::QueryLayer` each independently reinvent a
|
|
|
|
|
fragment of it.
|
|
|
|
|
- **Layer** = `obilayeredmap::Layer` (format-erased: `Count`/`Presence`,
|
|
|
|
|
each wrapping a `TypedLayer<D>`) — see "(1) done" below for how this
|
|
|
|
|
came to be; `TypedLayer<D>` (`{ mphf: MphfLayer, data: D }`, monomorphic)
|
|
|
|
|
is the lower-level, `D`-fixed building block `Layer` is built on, not
|
|
|
|
|
what other crates should reach for directly.
|
|
|
|
|
- **MPHF** = `MphfLayer.mphf: MemCase<MphfEps>` — kmer → slot.
|
|
|
|
|
- **Evidence** = `MphfLayer.ev: LayerEvidence` (`Exact`/`Approx`/
|
|
|
|
|
`Hybrid` — `evidence.bin`/`fingerprint.bin`; see `EvidenceKind`).
|
|
|
|
|
- **Matrix** = `Layer<D>.data: D` — `PersistentBitMatrix` /
|
|
|
|
|
`PersistentCompactIntMatrix` / `PersistentSparseBitMatrix` / `()`.
|
|
|
|
|
- **Matrix** = `TypedLayer<D>.data: D` — `PersistentBitMatrix` /
|
|
|
|
|
`PersistentCompactIntMatrix`.
|
|
|
|
|
|
|
|
|
|
Today's actual nesting, in Rust terms:
|
|
|
|
|
Target nesting once `KmerPartition` exists:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
KmerIndex
|
|
|
|
|
└─ partition: KmerPartition (all N partitions — paths + metadata only)
|
|
|
|
|
└─ (opened on demand, per i) LayeredMap<D> ← "one partition", unnamed as such
|
|
|
|
|
└─ layers: Vec<Layer<D>>
|
|
|
|
|
└─ Layer<D> { mphf: MphfLayer, data: D }
|
|
|
|
|
KmerIndex (obikindex)
|
|
|
|
|
└─ (opened on demand, per i) KmerPartition (obikpartition — not yet built)
|
|
|
|
|
└─ layers: Vec<Layer> (obilayeredmap)
|
|
|
|
|
└─ Layer::Count/Presence(TypedLayer<D>)
|
|
|
|
|
└─ TypedLayer<D> { mphf: MphfLayer, data: D }
|
|
|
|
|
├─ mphf.mphf → MPHF
|
|
|
|
|
├─ mphf.ev → Evidence
|
|
|
|
|
└─ data → Matrix
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`obikphylo::siblings::cache::PartitionCache` — the thing (2) is meant to
|
|
|
|
|
generalise — skips the middle of this nesting entirely: it doesn't build a
|
|
|
|
|
`Vec<LayeredMap<D>>`, it builds `mats: Vec<Vec<Mat>>` directly — outer
|
|
|
|
|
index = partition `i` (via `KmerPartition`), inner index = layer `l` (via
|
|
|
|
|
`Layer<D>`/`Mat`) — reconstructing "one partition, its layers open" as a
|
|
|
|
|
bare nested `Vec` because no owning type for that concept exists to reuse.
|
|
|
|
|
That gap is exactly what (2) needs to fill, and (1) is exactly what would
|
|
|
|
|
let a per-partition type hold layers of mixed `D` (today `LayeredMap<D>`
|
|
|
|
|
can't, being monomorphic — see "The gap in `obilayeredmap`'s existing
|
|
|
|
|
cache" below).
|
|
|
|
|
|
|
|
|
|
## Major restructuring (2026-08-20): `KmerPartitions` merged into `KmerIndex`
|
|
|
|
|
|
|
|
|
|
Prompted by a direct question: why keep `KmerIndex`/`KmerPartitions` split
|
|
|
|
@@ -163,6 +209,51 @@ still-separate `QueryLayer` (which still independently bundles MPHF+matrix,
|
|
|
|
|
2-way not using `Layer` at all). Both remaining consumers now sit one
|
|
|
|
|
`Layer::open` call away from unifying onto (2) once it exists.
|
|
|
|
|
|
|
|
|
|
## (1b) done (2026-08-20): `Layer::Empty` — the first non-ready-to-read state
|
|
|
|
|
|
|
|
|
|
First step toward `Layer` representing a layer's whole life, not just the
|
|
|
|
|
open-for-reading end of it (see "Definitions" above: `KmerPartition` will
|
|
|
|
|
hold `Vec<Layer>` regardless of each layer's state, states in between
|
|
|
|
|
included). Added one variant:
|
|
|
|
|
|
|
|
|
|
```rust
|
|
|
|
|
pub enum Layer {
|
|
|
|
|
Empty { dir: PathBuf },
|
|
|
|
|
Count(TypedLayer<PersistentCompactIntMatrix>),
|
|
|
|
|
Presence(TypedLayer<PersistentBitMatrix>),
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`Layer::create(dir)` makes the directory and returns `Empty { dir }` —
|
|
|
|
|
nothing else; no MPHF/unitigs/evidence construction yet (that's the
|
|
|
|
|
deferred next step: `build_mphf()`/`build_unitigs()`/`build_evidence()`
|
|
|
|
|
methods to progress `Empty` → eventually `Count`/`Presence`). `Empty`
|
|
|
|
|
carries path accessors so builder code has one place to get
|
|
|
|
|
`mphf_path()`/`unitigs_path()`/`evidence_path()`/`fingerprint_path()`/
|
|
|
|
|
`counts_dir()`/`presence_dir()` from, instead of redeclaring the
|
|
|
|
|
`mphf.bin`/`unitigs.bin`/… filenames at each write site — reusing the
|
|
|
|
|
constants `layer.rs`/`mphf_layer.rs` already own (`COUNTS_DIR`/
|
|
|
|
|
`PRESENCE_DIR` widened from private to `pub(crate)`, file-name constants
|
|
|
|
|
already were).
|
|
|
|
|
|
|
|
|
|
Every read method (`content`/`evidence_kind`/`n`/`find_slot`/
|
|
|
|
|
`index_batch`/`n_cols`/`fill_sub_matrix_carries`) panics on `Empty` with a
|
|
|
|
|
one-line message naming the method — confirmed as the right behaviour:
|
|
|
|
|
calling any of them on an `Empty` layer means the caller assumed a layer
|
|
|
|
|
was ready when it wasn't, an implementation error to surface loudly, not
|
|
|
|
|
a case to design around (`Option`/`Result` would let it silently
|
|
|
|
|
propagate instead of failing at the actual mistake). Same panic added to
|
|
|
|
|
`obikphylo::siblings::iter.rs`'s `impl SiblingLayerExt for
|
|
|
|
|
obilayeredmap::Layer` (4 methods), the one other place that exhaustively
|
|
|
|
|
matched `Layer`'s variants.
|
|
|
|
|
|
|
|
|
|
Full workspace suite green (`cargo check --workspace --all-targets` then
|
|
|
|
|
`cargo test --workspace`, exit code 0) after.
|
|
|
|
|
|
|
|
|
|
Still deferred, per explicit instruction: `build_mphf()`/
|
|
|
|
|
`build_unitigs()`/`build_evidence()` to progress `Empty` further, and (2)
|
|
|
|
|
— `KmerPartition` itself — unchanged from above.
|
|
|
|
|
|
|
|
|
|
## The problem
|
|
|
|
|
|
|
|
|
|
Reading a layer's data (MPHF + matrix) is not free: `MphfLayer::open` mmaps
|
|
|
|
@@ -180,34 +271,47 @@ across cores without reducing it. `PartitionCache::build` opens every
|
|
|
|
|
partition's every layer once, up front, in parallel, and keeps the handles
|
|
|
|
|
alive for the run.
|
|
|
|
|
|
|
|
|
|
## Three independent implementations of the same bundle
|
|
|
|
|
## Three independent implementations of the same bundle (historical — (1) fixed this)
|
|
|
|
|
|
|
|
|
|
**As of 2026-08-20 this table describes the pre-(1) state.** `Mat` no
|
|
|
|
|
longer exists (deleted when `obilayeredmap::Layer` replaced it — see "(1)
|
|
|
|
|
done" above); `Layer<D>` in the table below is what's now called
|
|
|
|
|
`TypedLayer<D>`. `QueryLayer` is unaffected and still stands as described —
|
|
|
|
|
still uncached, still not using `Layer` at all — which is exactly what (2)
|
|
|
|
|
needs to fix. Kept for the original motivation, not as current fact:
|
|
|
|
|
|
|
|
|
|
Searching the codebase for "who bundles MPHF + matrix, with per-layer format
|
|
|
|
|
auto-detection" turns up three unrelated implementations:
|
|
|
|
|
auto-detection" turned up three unrelated implementations:
|
|
|
|
|
|
|
|
|
|
| | lives in | scope | cached? |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| `Layer<D>` | `obilayeredmap` | one layer, `D` fixed at compile time | held alive by whoever owns the `Layer`, no policy of its own |
|
|
|
|
|
| `Mat` | `obikphylo::siblings::cache` | one layer, format resolved per instance from an enum of 3 `Layer<D>` variants | yes, via `PartitionCache` |
|
|
|
|
|
| `QueryLayer` | `obikpartition::query_layer` | one layer, `(MphfLayer, PersistentBitMatrix\|PersistentCompactIntMatrix)` pair, bypasses `Layer<D>` entirely | **no** — opened fresh inside `query_partition_with` on every call |
|
|
|
|
|
| `Layer<D>` (now `TypedLayer<D>`) | `obilayeredmap` | one layer, `D` fixed at compile time | held alive by whoever owns the `Layer`, no policy of its own |
|
|
|
|
|
| `Mat` (now deleted; superseded by `obilayeredmap::Layer`) | `obikphylo::siblings::cache` | one layer, format resolved per instance from an enum of 3 `Layer<D>` variants | yes, via `PartitionCache` |
|
|
|
|
|
| `QueryLayer` (unchanged, still current) | `obikindex::query_layer` (moved crates since this was written — see "Major restructuring") | one layer, `(MphfLayer, PersistentBitMatrix\|PersistentCompactIntMatrix)` pair, bypasses `TypedLayer<D>`/`Layer` entirely | **no** — opened fresh inside `query_partition_with` on every call |
|
|
|
|
|
|
|
|
|
|
`query_partition_with` is `obikmer query`'s normal query path — the one
|
|
|
|
|
most exposed to repeated cross-partition lookups — and it is the one with
|
|
|
|
|
no cache at all. `obikphylo` built a cache first only because sibling-annex
|
|
|
|
|
construction hits the cost hardest, not because the need is sibling-specific.
|
|
|
|
|
most exposed to repeated cross-partition lookups — and it is still the one
|
|
|
|
|
with no cache at all. `obikphylo` built a cache first only because
|
|
|
|
|
sibling-annex construction hits the cost hardest, not because the need is
|
|
|
|
|
sibling-specific.
|
|
|
|
|
|
|
|
|
|
## The gap in `obilayeredmap`'s existing cache
|
|
|
|
|
## The gap in `obilayeredmap`'s existing cache (historical — (1) fixed this)
|
|
|
|
|
|
|
|
|
|
`obilayeredmap::LayeredMap<D>` already caches correctly at the granularity
|
|
|
|
|
of one partition: `open(root)` opens every layer once, keeps `Vec<Layer<D>>`
|
|
|
|
|
alive for the `LayeredMap`'s lifetime. But it is monomorphic — every layer
|
|
|
|
|
in the `Vec` must share the same concrete `D`. In practice this is false:
|
|
|
|
|
layers in the same partition are packed independently over time (`pack
|
|
|
|
|
--sparse` converts one layer's presence matrix at a time), so a partition
|
|
|
|
|
can genuinely mix `PersistentBitMatrix`-backed and
|
|
|
|
|
`PersistentSparseBitMatrix`-backed presence layers. `LayeredMap<D>` cannot
|
|
|
|
|
represent that mix — `Mat`'s 3-way enum exists specifically to work around
|
|
|
|
|
this gap, one crate away from where the gap actually is.
|
|
|
|
|
of one partition: `open(root)` opens every layer once, keeps
|
|
|
|
|
`Vec<TypedLayer<D>>` alive for the `LayeredMap`'s lifetime. But it is
|
|
|
|
|
monomorphic — every layer in the `Vec` must share the same concrete `D`.
|
|
|
|
|
In practice this was false: layers in the same partition are packed
|
|
|
|
|
independently over time (`pack --sparse` converts one layer's presence
|
|
|
|
|
matrix at a time). This motivated (1) — `obilayeredmap::Layer`, done — but
|
|
|
|
|
note the specific `PersistentSparseBitMatrix`-mixing scenario described
|
|
|
|
|
here turned out to be moot: `PersistentBitMatrix` itself absorbed sparse
|
|
|
|
|
storage as a 4th internal variant before (1) was built (see "One bug found
|
|
|
|
|
… one earlier claim retracted" below), so the only heterogeneity `Layer`
|
|
|
|
|
actually needs to represent is `Count` vs. `Presence`, not dense-vs-sparse
|
|
|
|
|
presence. `LayeredMap<D>` itself is unaffected by any of this — it's still
|
|
|
|
|
monomorphic, still not used by `Layer`/`KmerPartition` (which bypass it
|
|
|
|
|
entirely, opening each `TypedLayer<D>` directly, the same way `Mat` did).
|
|
|
|
|
|
|
|
|
|
## Resource cost: mmap does not hold a file descriptor
|
|
|
|
|
|
|
|
|
@@ -249,56 +353,65 @@ independently of any caching decision: an unpacked `Columnar` matrix opens
|
|
|
|
|
one mmap **per genome column**, multiplying the mapping count a cache would
|
|
|
|
|
have to hold by `n_genomes`.
|
|
|
|
|
|
|
|
|
|
## Layering: who owns what
|
|
|
|
|
## Layering: who owns what (superseded — see Definitions above)
|
|
|
|
|
|
|
|
|
|
Established in discussion, not yet coded:
|
|
|
|
|
|
|
|
|
|
- `obilayeredmap` operates within a single partition's index root; it has
|
|
|
|
|
no notion of "partition" (confirmed independently while fixing
|
|
|
|
|
`open_data`/`layer_dir` — see git history around 2026-08-20). One
|
|
|
|
|
`LayeredMap<D>` (or its heterogeneous-`D` successor) = one partition.
|
|
|
|
|
- Nobody currently owns "the collection of partitions" as reusable state.
|
|
|
|
|
`obikpartition::KmerPartition` is the closest candidate — it already
|
|
|
|
|
owns `n_partitions()`/`part_dir(i)` — but today it is a pure
|
|
|
|
|
config/path resolver, not a state holder: `dereplicate`, `count_kmer`,
|
|
|
|
|
`obikindex`'s `distance.rs`/`stats.rs`, and
|
|
|
|
|
`obikphylo::PartitionCache::build` each independently write their own
|
|
|
|
|
`(0..n_partitions).into_par_iter().map(...)` loop and open what they need
|
|
|
|
|
from scratch.
|
|
|
|
|
This section used to argue nobody owned "the collection of partitions."
|
|
|
|
|
That's resolved: `KmerIndex` (`obikindex`) owns it now, directly (see
|
|
|
|
|
"Major restructuring" below). What's still genuinely unowned is *one
|
|
|
|
|
partition's open layers* — `KmerPartition`, in the not-yet-created
|
|
|
|
|
`obikpartition` — see "Definitions" at the top of this file for the
|
|
|
|
|
current, authoritative answer. Left here only so old links/references to
|
|
|
|
|
this heading don't 404; don't read this section for current facts.
|
|
|
|
|
|
|
|
|
|
## Direction agreed, not yet implemented
|
|
|
|
|
|
|
|
|
|
1. A heterogeneous-format layer type in `obilayeredmap` (name TBD),
|
|
|
|
|
generalising `Mat`'s 3-way enum + auto-detection — `obilayeredmap`
|
|
|
|
|
already implements `LayerData` for all three concrete matrix types
|
|
|
|
|
involved, so it already has the knowledge `Mat` needed and had to
|
|
|
|
|
duplicate.
|
|
|
|
|
2. A cache spanning multiple partitions, keeping (2)'s layer type open per
|
|
|
|
|
partition per layer, for the whole lifetime of a long-running command —
|
|
|
|
|
generalising `PartitionCache` minus its sibling-specific parts
|
|
|
|
|
(`fast_mode`, `find_presence_batch`) — living in `obikpartition`
|
|
|
|
|
(owner of the partition dimension) and built on top of (1).
|
|
|
|
|
3. `obikpartition::query_partition_with` — the normal query path,
|
|
|
|
|
currently uncached — becomes a consumer of (2), not just
|
|
|
|
|
`obikphylo::PartitionCache`.
|
|
|
|
|
Only (2) remains — (1) shipped as `obilayeredmap::Layer` (see "(1) done"
|
|
|
|
|
above). Concretely, in order:
|
|
|
|
|
|
|
|
|
|
Open before implementing: exact API shape of (1) and (2), whether `Mat` is
|
|
|
|
|
deleted outright or kept as a thin sibling-specific wrapper, and whether (2)
|
|
|
|
|
needs an eviction policy or can simply hold every partition open for the
|
|
|
|
|
process lifetime (revisit once the VM-mapping-count question above has a
|
|
|
|
|
real number behind it for this codebase's scale).
|
|
|
|
|
1. Create the `obikpartition` crate (`obikindex → obikpartition →
|
|
|
|
|
obilayeredmap`, no other edges — see "Definitions" above for the exact
|
|
|
|
|
constraint and why).
|
|
|
|
|
2. `KmerPartition { layers: Vec<obilayeredmap::Layer> }` — `open`,
|
|
|
|
|
`n_layers`, `layer(i)`, `find`, plus whatever batch-lookup surface
|
|
|
|
|
`obikphylo::siblings::cache::PartitionCache` currently needs
|
|
|
|
|
(`find_presence_batch`/`find_presence_batch_fast`; `fast_mode` is
|
|
|
|
|
sibling-specific bookkeeping and should probably stay in `obikphylo`,
|
|
|
|
|
wrapping a `KmerPartition`/`Vec<KmerPartition>` rather than living
|
|
|
|
|
inside it — same "generic vs. domain-specific" split `iter_minorants_batch`
|
|
|
|
|
already went through for `Layer` in (1)).
|
|
|
|
|
3. Migrate `obikphylo::siblings::cache::PartitionCache` to hold
|
|
|
|
|
`Vec<KmerPartition>` instead of `Vec<Vec<Layer>>`.
|
|
|
|
|
4. Migrate `obikindex::query_layer::QueryLayer`/`query_partition_with` to
|
|
|
|
|
use `KmerPartition` too, closing the "no cache at all" gap on
|
|
|
|
|
`obikmer`'s normal query path (see "Three independent implementations,"
|
|
|
|
|
historical, above).
|
|
|
|
|
|
|
|
|
|
Open before implementing: exact API shape of `KmerPartition` (propose,
|
|
|
|
|
confirm before coding — non-trivial), and whether the multi-partition
|
|
|
|
|
`Vec<KmerPartition>` needs an eviction policy or can simply hold every
|
|
|
|
|
partition open for the process lifetime (revisit once the
|
|
|
|
|
VM-mapping-count question above has a real number behind it for this
|
|
|
|
|
codebase's scale — still not measured).
|
|
|
|
|
|
|
|
|
|
## Preparatory work done (2026-08-20)
|
|
|
|
|
|
|
|
|
|
Groundwork for (1)/(2), landed ahead of the design itself:
|
|
|
|
|
Groundwork for (1)/(2), landed ahead of the design itself. **Note**: at
|
|
|
|
|
the time this was written, the collection type these bullets describe was
|
|
|
|
|
named `KmerPartition` (singular) in this doc; it was renamed
|
|
|
|
|
`KmerPartitions` (plural) shortly after, then deleted entirely and merged
|
|
|
|
|
into `KmerIndex` (see "Major restructuring" above). The bullets below are
|
|
|
|
|
edited to say `KmerPartitions` throughout, to not collide with the
|
|
|
|
|
unrelated, brand-new singular `KmerPartition` defined at the top of this
|
|
|
|
|
file — the accessors described here live on `KmerIndex` today, not on
|
|
|
|
|
any type called `KmerPartition`.
|
|
|
|
|
|
|
|
|
|
- `KmerPartition` (`obikpartition`) gained `partition_dir`/`index_dir`/
|
|
|
|
|
`layer_dir` as the single source of truth for a partition's on-disk
|
|
|
|
|
layout, replacing per-module duplicated `const INDEX_SUBDIR: &str =
|
|
|
|
|
"index"` (7 copies) and ad hoc path joins — including one found
|
|
|
|
|
duplicated *inside `KmerPartition` itself* (`ensure_writer` rebuilt
|
|
|
|
|
`part_dir`'s own logic by hand).
|
|
|
|
|
- `KmerPartition` gained `partition_meta`/`n_layers`/`index_mode`, wrapping
|
|
|
|
|
- `KmerPartitions` (`obikpartitionner`, at the time) gained
|
|
|
|
|
`partition_dir`/`index_dir`/`layer_dir` as the single source of truth
|
|
|
|
|
for a partition's on-disk layout, replacing per-module duplicated
|
|
|
|
|
`const INDEX_SUBDIR: &str = "index"` (7 copies) and ad hoc path joins —
|
|
|
|
|
including one found duplicated *inside the struct itself*
|
|
|
|
|
(`ensure_writer` rebuilt `part_dir`'s own logic by hand).
|
|
|
|
|
- Same struct gained `partition_meta`/`n_layers`/`index_mode`, wrapping
|
|
|
|
|
`obilayeredmap::meta::PartitionMeta::load` (via the existing
|
|
|
|
|
`common::load_meta`, which also recovers indexes built before
|
|
|
|
|
`meta.json` existed). Before this, `obikphylo` and `obikindex` imported
|
|
|
|
@@ -392,16 +505,19 @@ storage decisions, now a *third* copy of the same logic to keep in sync)
|
|
|
|
|
|
|
|
|
|
`obikphylo::siblings::family_scan::scan_layer_families` still re-derives
|
|
|
|
|
`index_dir` from `layer_dir.parent()` and calls `PartitionMeta::load`
|
|
|
|
|
itself, purely to get `.mode` for `Mat::open`. Fixing it the way the 21
|
|
|
|
|
other call sites were fixed needs more than a 1:1 swap: `scan_layer_families`
|
|
|
|
|
only receives a bare `layer_dir: &Path`, not a `(partition, part, layer)`
|
|
|
|
|
triple, and its single upstream source of layer paths,
|
|
|
|
|
`sibling_layer_dirs`, returns a flat `Vec<PathBuf>` with the partition/layer
|
|
|
|
|
indices already discarded. Fixing it properly means either having
|
|
|
|
|
`sibling_layer_dirs` return `(PathBuf, IndexMode)` (or `(part, layer)`)
|
|
|
|
|
pairs, or threading `&KmerPartition` + indices through instead of paths —
|
|
|
|
|
and touching every one of `scan_layer_families`'s 8 callers (`distance.rs`,
|
|
|
|
|
`alignment.rs`, `cardinality.rs`, `entropy.rs` ×2, `sankoff_bundle.rs` ×2,
|
|
|
|
|
`stats.rs`). Left alone this round; worth doing as part of the same pass
|
|
|
|
|
that builds (1)/(2), since those callers are exactly the sibling-annex
|
|
|
|
|
consumers (2) is meant to serve.
|
|
|
|
|
itself, purely to get `.mode` for `obilayeredmap::Layer::open` (was
|
|
|
|
|
`Mat::open`, same gap, survived the `Mat` → `Layer` swap in (1) unchanged).
|
|
|
|
|
Fixing it the way the 21 other call sites were fixed needs more than a 1:1
|
|
|
|
|
swap: `scan_layer_families` only receives a bare `layer_dir: &Path`, not a
|
|
|
|
|
`(partition, part, layer)` triple, and its single upstream source of layer
|
|
|
|
|
paths, `sibling_layer_dirs`, returns a flat `Vec<PathBuf>` with the
|
|
|
|
|
partition/layer indices already discarded. Fixing it properly means either
|
|
|
|
|
having `sibling_layer_dirs` return `(PathBuf, IndexMode)` (or `(part,
|
|
|
|
|
layer)`) pairs, or threading `&KmerIndex` + indices through instead of
|
|
|
|
|
paths (not `&KmerPartition` — that type doesn't exist yet, and once it
|
|
|
|
|
does it still won't know `IndexMode`, which lives on `KmerIndex`/
|
|
|
|
|
`PartitionMeta`) — and touching every one of `scan_layer_families`'s 8
|
|
|
|
|
callers (`distance.rs`, `alignment.rs`, `cardinality.rs`, `entropy.rs` ×2,
|
|
|
|
|
`sankoff_bundle.rs` ×2, `stats.rs`). Left alone this round; worth doing as
|
|
|
|
|
part of the same pass that builds `KmerPartition`, since those callers are
|
|
|
|
|
exactly the sibling-annex consumers it's meant to serve.
|
|
|
|
|
Eric Coissac