obikmer/DevDocMD/implementation/partition_layer_cache.md

Partition and layer caching (discussion)

Status (2026-08-20, latest pass): (1) done — obilayeredmap::Layer exists, Mat is gone. (1b) done — Layer::Empty, the first non-ready state, added (panics on every read method). (2) — KmerPartition + a multi-partition cache — not started, precisely specified below after a real mix-up: an earlier version of this doc used the name KmerPartition (singular) for what was actually the collection type (later renamed KmerPartitions, later merged into KmerIndex — see "Major restructuring" below), and never retracted that usage before this section was rewritten. An agent working from that stale wording built the wrong thing. If you are about to implement (2), read "Definitions: obikpartition and KmerPartition" below — it is the current, authoritative naming — before touching any other section of this file, some of which still describe superseded states of the code and are kept only as dated history.

Definitions: obikpartition and KmerPartition (not yet created)

obikpartition — a new workspace crate, not created yet. Holds the Partition tier of the Index { Partition { Layer } } model, the same way obilayeredmap already holds the Layer tier as its own crate rather than living inside obikindex. Depends only on obilayeredmap (for Layer) and lower (obikseq, obiskio). Does not depend on obikindex, obikpartitionner, or obikphylo. Dependency direction: obikindex → obikpartition → obilayeredmap; obikphylo → obikindex (and/or obikpartition directly if it ends up needing it without going through KmerIndex).

KmerPartition (singular) — the one type this crate exists for. Represents one partition's already-open layers — a read cache, built once per partition and held for the run, not rebuilt per lookup. Shape:

pub struct KmerPartition {
    layers: Vec<obilayeredmap::Layer>,
}

Nothing else. In particular:

• No path computation. KmerPartition::open takes an already-resolved index_dir: &Path (plus mode: &IndexMode, n_layers: usize, with_counts: bool — whatever it needs, as plain arguments), the same discipline obikpartitionner::PartitionRouter::open already follows. Computing index_dir/layer_dir from a partition number is KmerIndex's job (obikindex, which owns that already — see "Major restructuring" below); KmerPartition never reaches back into KmerIndex to get it (would require obikpartition → obikindex, the wrong direction).
• No routing/write state. Writing raw superkmers, dereplicate, count_kmer stay in obikpartitionner::PartitionRouter — a completely different crate, a completely different phase (pre-layer, whereas KmerPartition only makes sense once layers exist).
• No multi-partition collection baked in. KmerPartition is one partition. Whatever ends up caching several of them (replacing obikphylo::siblings::cache::PartitionCache's Vec<Vec<Layer>> and obikindex::query_layer's per-call reopen) holds Vec<KmerPartition> — that collection can live in obikpartition too, or in obikindex alongside KmerIndex; not yet decided, secondary to getting KmerPartition itself right first.

Do not confuse with KmerPartitions (plural — note the s): that type is gone. It used to be obikpartitionner's (nee obikpartition, briefly — see the crate-rename history below, itself a separate rename from this one) do-everything struct — routing, dereplication, and path lookups all in one. It was deleted on 2026-08-20; its read-side (paths, n_layers, partition_meta) was absorbed into KmerIndex, its write-side became PartitionRouter. KmerPartition (this section, singular, no final s) is a brand-new type with a different job, in a crate that doesn't exist yet — not a revival, not a renaming, of KmerPartitions.

Type-to-concept mapping: Index / Partition / Layer

The conceptual nesting Index { Partition { Layer { MPHF, Evidence, Matrix } } } }, current state:

• Index = obikindex::KmerIndex — { root_path, meta: IndexMeta }. Also directly exposes the partition-path/metadata accessors (partition_dir(i), index_dir(i), layer_dir(i, l), partition_meta(i), n_layers(i), partition_mode(i), n_partitions()) since KmerPartitions merged into it (see "Major restructuring" below) — KmerIndex today is "index + collection of partitions' paths & metadata," just without a Vec of open layers.
• Partition, the collection = no dedicated type today; the closest thing is KmerIndex itself (previous bullet). Once KmerPartition (singular, see Definitions above) exists, a Vec<KmerPartition> somewhere would be this — still open, see "Direction agreed" below.
• Partition, one of them = obikpartition::KmerPartition — to be built, see Definitions above. Nothing plays this role today; obikphylo::siblings::cache::PartitionCache and obikindex::query_layer::QueryLayer each independently reinvent a fragment of it.
• Layer = obilayeredmap::Layer (format-erased: Count/Presence, each wrapping a TypedLayer<D>) — see "(1) done" below for how this came to be; TypedLayer<D> ({ mphf: MphfLayer, data: D }, monomorphic) is the lower-level, D-fixed building block Layer is built on, not what other crates should reach for directly.
  • MPHF = MphfLayer.mphf: MemCase<MphfEps> — kmer → slot.
  • Evidence = MphfLayer.ev: LayerEvidence (Exact/Approx/ Hybrid — evidence.bin/fingerprint.bin; see EvidenceKind).
  • Matrix = TypedLayer<D>.data: D — PersistentBitMatrix / PersistentCompactIntMatrix.

Target nesting once KmerPartition exists:

KmerIndex                                    (obikindex)
 └─ (opened on demand, per i)  KmerPartition  (obikpartition — not yet built)
     └─ layers: Vec<Layer>                    (obilayeredmap)
         └─ Layer::Count/Presence(TypedLayer<D>)
             └─ TypedLayer<D> { mphf: MphfLayer, data: D }
                 ├─ mphf.mphf   → MPHF
                 ├─ mphf.ev     → Evidence
                 └─ data        → Matrix

Major restructuring (2026-08-20): KmerPartitions merged into KmerIndex

Prompted by a direct question: why keep KmerIndex/KmerPartitions split when, one level down, KmerPartitions is going to directly hold Vec<KmerPartition> rather than being split again into "collection-holder" + "collection"? Investigating the actual justification ("KmerPartitions has an independent lifecycle, used before an index exists") turned out to be false — KmerPartitions::create was called in exactly one place, inside KmerIndex::create, and every open_with_config reopen outside KmerIndex's own constructors was a redundant re-derivation of a KmerPartitions already reachable via index.partition() (the exact kind of duplication this whole doc has been tracking). Once that was gone, so was the reason to keep them separate.

Second correction, from the same conversation: obikpartitionner had accumulated query/merge/select/rebuild/dump/distance logic that has nothing to do with partitioning super-kmers — it operates on layers, which don't exist yet at the phase obikpartitionner is actually responsible for (scatter → dereplicate → count, all pre-layer). That logic moved to obikindex, which already depends on obilayeredmap and never needed obikpartitionner for it. No crate-dependency inversion was needed — obikindex → obikpartitionner stays the same direction as before.

Result:

• obikpartitionner (renamed back from obikpartition) now contains only PartitionRouter (superkmer routing: write/write_batch/flush/ close, dereplicate, count_kmer, KmerSpectrum) and the partition_dir(root, i) naming primitive both PartitionRouter and KmerIndex build on. KmerPartitions no longer exists as a type.
• KmerIndex (obikindex) absorbed KmerPartitions's read-side entirely: partition_dir/index_dir/layer_dir/partition_meta/n_layers/ partition_mode/n_partitions (the last now derived from 2^config.n_bits, no longer a stored, independently-set duplicate field — kmer_size/minimizer_size used to be double-stored, in both KmerPartitions and IndexMeta.config, a latent-drift risk flagged earlier in this doc; now single-sourced from IndexMeta.config). Seven whole files moved from obikpartitionner into obikindex verbatim as impl KmerIndex blocks, kept as separate files (not merged into existing same-topic files): index_layer.rs, query_layer.rs, merge_layer/, select_layer.rs, rebuild_layer.rs, dump_layer.rs, plus distance.rs's count_store/presence_store (renamed matrix_store.rs to avoid colliding with obikindex's own pre-existing distance.rs), and their shared support (common.rs's load_meta/ olm_to_sk, filter.rs, graph_pipeline.rs).
• obikphylo::siblings::cache::PartitionCache::build now takes &KmerIndex directly instead of a separately-opened &KmerPartitions — this deleted the redundant-reopen pattern at all 8 call sites (alignment/build/cardinality/distance/entropy×2/ sankoff_bundle/stats), the same bug flagged earlier in this conversation as a side effect of investigating the false "independent lifecycle" claim.
• KmerIndex::partition()/partition_mut() are gone; scatter() (obikmer) and any write-side code get a transient PartitionRouter via KmerIndex::partition_router().
• A real bug caught by the test suite during this move: PartitionRouter::open initially defaulted to closed: true (inherited from KmerPartitions::open_with_config's old read-only-reopen semantics), which broke every write through a router obtained via partition_router(). Fixed — PartitionRouter is exclusively a write/processing tool now, so open always starts open.

Full workspace test suite green (0 failed) after, including all 27 obikphylo::siblings tests.

(1) done (2026-08-20): Layer is now the heterogeneous handle, Mat is gone

Resolved the naming question left open above. Layer<D> (the old generic/monomorphic type) renamed to TypedLayer<D> throughout (obilayeredmap, obikindex, obikphylo — 12 files, mechanical) to free Layer for the type that's actually meant to be everyone's default handle. obilayeredmap::content_layer::Layer (re-exported at the crate root) is that type — Count(TypedLayer<PersistentCompactIntMatrix>)/ Presence(TypedLayer<PersistentBitMatrix>), Layer::open doing the same disk probe Mat::open used to, find_slot/index_batch/n_cols/ fill_sub_matrix_carries dispatching per variant exactly as Mat did.

obikphylo::siblings::cache::Mat deleted outright — PartitionCache now holds Vec<Vec<obilayeredmap::Layer>> directly. The one sibling-specific method Mat carried (iter_minorants_batch) is not on obilayeredmap:: Layer (phylo concepts don't belong in obilayeredmap) — it's an impl SiblingLayerExt for obilayeredmap::Layer in iter.rs, dispatching to each variant's existing impl<D: LayerData> SiblingLayerExt for TypedLayer<D>.

Full workspace suite green (0 failed) after, including all 27 obikphylo::siblings tests.

Still not built: (2) — KmerPartition (singular, one partition's open Vec<Layer>) and a multi-partition cache in obikpartitionner to replace obikphylo::siblings::cache::PartitionCache and obikindex::query_layer's still-separate QueryLayer (which still independently bundles MPHF+matrix, 2-way not using Layer at all). Both remaining consumers now sit one Layer::open call away from unifying onto (2) once it exists.

(1b) done (2026-08-20): Layer::Empty — the first non-ready-to-read state

First step toward Layer representing a layer's whole life, not just the open-for-reading end of it (see "Definitions" above: KmerPartition will hold Vec<Layer> regardless of each layer's state, states in between included). Added one variant:

pub enum Layer {
    Empty { dir: PathBuf },
    Count(TypedLayer<PersistentCompactIntMatrix>),
    Presence(TypedLayer<PersistentBitMatrix>),
}

Layer::create(dir) makes the directory and returns Empty { dir } — nothing else; no MPHF/unitigs/evidence construction yet (that's the deferred next step: build_mphf()/build_unitigs()/build_evidence() methods to progress Empty → eventually Count/Presence). Empty carries path accessors so builder code has one place to get mphf_path()/unitigs_path()/evidence_path()/fingerprint_path()/ counts_dir()/presence_dir() from, instead of redeclaring the mphf.bin/unitigs.bin/… filenames at each write site — reusing the constants layer.rs/mphf_layer.rs already own (COUNTS_DIR/ PRESENCE_DIR widened from private to pub(crate), file-name constants already were).

Every read method (content/evidence_kind/n/find_slot/ index_batch/n_cols/fill_sub_matrix_carries) panics on Empty with a one-line message naming the method — confirmed as the right behaviour: calling any of them on an Empty layer means the caller assumed a layer was ready when it wasn't, an implementation error to surface loudly, not a case to design around (Option/Result would let it silently propagate instead of failing at the actual mistake). Same panic added to obikphylo::siblings::iter.rs's impl SiblingLayerExt for obilayeredmap::Layer (4 methods), the one other place that exhaustively matched Layer's variants.

Full workspace suite green (cargo check --workspace --all-targets then cargo test --workspace, exit code 0) after.

Still deferred, per explicit instruction: build_mphf()/ build_unitigs()/build_evidence() to progress Empty further, and (2) — KmerPartition itself — unchanged from above.

The problem

Reading a layer's data (MPHF + matrix) is not free: MphfLayer::open mmaps mphf.bin plus (evidence.bin/fingerprint.bin + unitigs.bin), and the matrix side mmaps matrix.pbmx/matrix.pcmx (or one file per genome column if not yet packed). Any code path that reopens a layer per lookup instead of once per run pays this cost repeatedly.

obikphylo::siblings::cache::PartitionCache was built to avoid exactly this for build_sibling_annex/sibling_annex_stats: those commands probe many partitions, once per source layer, over the whole run. Profiling a real run showed wall-clock time dominated by repeated open()/mmap syscalls, not computation — parallelising the naive per-lookup opens spread the cost across cores without reducing it. PartitionCache::build opens every partition's every layer once, up front, in parallel, and keeps the handles alive for the run.

Three independent implementations of the same bundle (historical — (1) fixed this)

As of 2026-08-20 this table describes the pre-(1) state. Mat no longer exists (deleted when obilayeredmap::Layer replaced it — see "(1) done" above); Layer<D> in the table below is what's now called TypedLayer<D>. QueryLayer is unaffected and still stands as described — still uncached, still not using Layer at all — which is exactly what (2) needs to fix. Kept for the original motivation, not as current fact:

Searching the codebase for "who bundles MPHF + matrix, with per-layer format auto-detection" turned up three unrelated implementations:

	lives in	scope	cached?
Layer<D> (now TypedLayer<D>)	obilayeredmap	one layer, D fixed at compile time	held alive by whoever owns the Layer, no policy of its own
Mat (now deleted; superseded by obilayeredmap::Layer)	obikphylo::siblings::cache	one layer, format resolved per instance from an enum of 3 Layer<D> variants	yes, via PartitionCache
QueryLayer (unchanged, still current)	obikindex::query_layer (moved crates since this was written — see "Major restructuring")	one layer, (MphfLayer, PersistentBitMatrix|PersistentCompactIntMatrix) pair, bypasses TypedLayer<D>/Layer entirely	no — opened fresh inside query_partition_with on every call

query_partition_with is obikmer query's normal query path — the one most exposed to repeated cross-partition lookups — and it is still the one with no cache at all. obikphylo built a cache first only because sibling-annex construction hits the cost hardest, not because the need is sibling-specific.

The gap in obilayeredmap's existing cache (historical — (1) fixed this)

obilayeredmap::LayeredMap<D> already caches correctly at the granularity of one partition: open(root) opens every layer once, keeps Vec<TypedLayer<D>> alive for the LayeredMap's lifetime. But it is monomorphic — every layer in the Vec must share the same concrete D. In practice this was false: layers in the same partition are packed independently over time (pack --sparse converts one layer's presence matrix at a time). This motivated (1) — obilayeredmap::Layer, done — but note the specific PersistentSparseBitMatrix-mixing scenario described here turned out to be moot: PersistentBitMatrix itself absorbed sparse storage as a 4th internal variant before (1) was built (see "One bug found … one earlier claim retracted" below), so the only heterogeneity Layer actually needs to represent is Count vs. Presence, not dense-vs-sparse presence. LayeredMap<D> itself is unaffected by any of this — it's still monomorphic, still not used by Layer/KmerPartition (which bypass it entirely, opening each TypedLayer<D> directly, the same way Mat did).

Resource cost: mmap does not hold a file descriptor

Before deciding how many layers/partitions a cache may hold open simultaneously, the binding constraint needs to be identified correctly.

Confirmed against upstream documentation, not inferred from behaviour:

"After the mmap() call has returned, the file descriptor, fd, can be closed immediately without invalidating the mapping." — mmap(2), man7.org

"The close(2) function does not unmap pages" — mmap(2), Apple Developer

"A file backed Mmap ... will remain valid even after the File is dropped. ... the Mmap handle is completely independent of the File used to create it." — memmap2::Mmap, docs.rs

Every read-only mmap in this codebase already follows this: Mmap::map(&File::open(path)?)? — the File is a temporary, dropped (fd closed) immediately after the mapping is established; every persistent struct (PersistentBitVec, PersistentCompactIntVec, PackedBitMatrix, Evidence, FingerprintVec, ...) stores only the Mmap, never the File. So a cache built on these types does not consume the process's open-file-descriptor budget (ulimit -n, notoriously low by default on macOS) proportionally to how many mmapped files it holds.

It does consume a different resource — the process's virtual-memory mapping table (one entry per active mmap() region). Linux exposes this as vm.max_map_count (default 65530). No documented macOS equivalent (fixed numeric ceiling) was found; the constraint there appears to be virtual address space rather than an explicit mapping counter, but this is not sourced and should not be assumed. This is the resource actually worth measuring before deciding on cache size, not fd count — and it is why packing (matrix.pbmx/matrix.pcmx, one mmap for all columns) matters independently of any caching decision: an unpacked Columnar matrix opens one mmap per genome column, multiplying the mapping count a cache would have to hold by n_genomes.

Layering: who owns what (superseded — see Definitions above)

This section used to argue nobody owned "the collection of partitions." That's resolved: KmerIndex (obikindex) owns it now, directly (see "Major restructuring" below). What's still genuinely unowned is one partition's open layers — KmerPartition, in the not-yet-created obikpartition — see "Definitions" at the top of this file for the current, authoritative answer. Left here only so old links/references to this heading don't 404; don't read this section for current facts.

Direction agreed, not yet implemented

Only (2) remains — (1) shipped as obilayeredmap::Layer (see "(1) done" above). Concretely, in order:

1. Create the obikpartition crate (obikindex → obikpartition → obilayeredmap, no other edges — see "Definitions" above for the exact constraint and why).
2. KmerPartition { layers: Vec<obilayeredmap::Layer> } — open, n_layers, layer(i), find, plus whatever batch-lookup surface obikphylo::siblings::cache::PartitionCache currently needs (find_presence_batch/find_presence_batch_fast; fast_mode is sibling-specific bookkeeping and should probably stay in obikphylo, wrapping a KmerPartition/Vec<KmerPartition> rather than living inside it — same "generic vs. domain-specific" split iter_minorants_batch already went through for Layer in (1)).
3. Migrate obikphylo::siblings::cache::PartitionCache to hold Vec<KmerPartition> instead of Vec<Vec<Layer>>.
4. Migrate obikindex::query_layer::QueryLayer/query_partition_with to use KmerPartition too, closing the "no cache at all" gap on obikmer's normal query path (see "Three independent implementations," historical, above).

Open before implementing: exact API shape of KmerPartition (propose, confirm before coding — non-trivial), and whether the multi-partition Vec<KmerPartition> needs an eviction policy or can simply hold every partition open for the process lifetime (revisit once the VM-mapping-count question above has a real number behind it for this codebase's scale — still not measured).

Preparatory work done (2026-08-20)

Groundwork for (1)/(2), landed ahead of the design itself. Note: at the time this was written, the collection type these bullets describe was named KmerPartition (singular) in this doc; it was renamed KmerPartitions (plural) shortly after, then deleted entirely and merged into KmerIndex (see "Major restructuring" above). The bullets below are edited to say KmerPartitions throughout, to not collide with the unrelated, brand-new singular KmerPartition defined at the top of this file — the accessors described here live on KmerIndex today, not on any type called KmerPartition.

• KmerPartitions (obikpartitionner, at the time) gained partition_dir/index_dir/layer_dir as the single source of truth for a partition's on-disk layout, replacing per-module duplicated const INDEX_SUBDIR: &str = "index" (7 copies) and ad hoc path joins — including one found duplicated inside the struct itself (ensure_writer rebuilt part_dir's own logic by hand).
• Same struct gained partition_meta/n_layers/index_mode, wrapping obilayeredmap::meta::PartitionMeta::load (via the existing common::load_meta, which also recovers indexes built before meta.json existed). Before this, obikphylo and obikindex imported obilayeredmap::meta::PartitionMeta directly and called ::load() themselves at 21 call sites, each redoing its own error-mapping — every one of those crates knew the on-disk metadata format instead of going through an interface. Fixed everywhere except one remaining spot (below). Caught as a side effect: dump_layer.rs/query_layer.rs had been calling PartitionMeta::load directly, bypassing load_meta entirely — they never got the missing-meta.json recovery the other callers did.
• Layer introspection API discussed but not yet implemented — three axes, deliberately kept separate after an initial draft conflated them:
  • LayerContent { Count, Presence } — what the layer stores; a const on LayerData (compile-time, zero-cost), not a runtime field.
  • StorageKind { Implicit, Columnar, Packed, Sparse } — how it's stored; only meaningful for D that actually carry data (Layer<()> has neither this nor LayerContent — it's a write-time-only state, never a queryable content: once a layer is closed, "no matrix file" reads back as Presence/Implicit via PersistentBitMatrix::open's own fallback, not as some third "empty" content).
  • EvidenceKind { Exact, Approx, Hybrid } — from MphfLayer's own already-in-memory LayerEvidence discriminant.
  • Not all (LayerContent, StorageKind) pairs are legal: Count never has Implicit or Sparse.

Implemented (2026-08-20). LayerContent/StorageKind/EvidenceKind now exist, each with two forms:

• A runtime accessor on an already-open value (Layer<D>::content()/ storage_kind()/evidence_kind(), PersistentBitMatrix::storage_kind(), PersistentCompactIntMatrix::storage_kind(), MphfLayer::evidence_kind()) — reads a discriminant already in memory, zero disk access.
• A lightweight detect()/detect_storage() disk probe that mirrors the corresponding open()'s own priority order by hand (file-existence checks only, no mmap) — usable before committing to a D, unlike the runtime accessors. Exposed per-layer on LayeredMap<D> as detect_layer_content/detect_layer_storage/detect_layer_evidence (work regardless of D, since they only use self.root + the layer index).

StorageKind lives in obicompactvec (owner of PersistentBitMatrix/ PersistentCompactIntMatrix); LayerContent/EvidenceKind live in obilayeredmap. HasLayerContent/HasStorageKind gate Layer<()> out of content()/storage_kind() (no matrix, nothing to report), matching the "empty is transitional" conclusion above. 42 new tests across obilayeredmap's tests/layer.rs and tests/map.rs; full workspace suite green (0 failed) after.

Not done: these detect() probes don't yet replace Mat::open's or QueryLayer::open's own hand-rolled equivalents (still duplicated content/ storage decisions, now a third copy of the same logic to keep in sync) — that consolidation is (1)/(2)'s job, not this prep step's.

One bug found while reading around this (signalled, not fixed); one earlier claim retracted

• obicompactvec::bitmatrix::sparse.rs's module doc says "Not used by any production code path yet" — false since obikmer pack --sparse (cmd/pack/mod.rs) is wired to pack_sparse_bit_matrix and Mat::open already reads the result back in the sibling-annex path. Stale comment, not corrected.
• Retracted (2026-08-20): an earlier pass through this doc claimed obikpartition::query_layer::QueryLayer::open had no sparse-format detection and would silently corrupt reads on a pack --sparsed layer. False — PersistentBitMatrix (obicompactvec::bitmatrix::persistent) is a 4-way enum (Columnar/Packed/Sparse/Implicit), not 3-way as first read; its open() already detects Sparse via presence/sparse_meta.json, and every method on the type (row, fill_row, nonzero_iter, …) already dispatches all 4 arms. QueryLayer::open's PersistentBitMatrix::open(layer_dir) call was never the bug. Root cause of the false claim: a grep -n "Implicit\|Columnar\|Packed" used to read the enum definition silently skipped the Sparse(...) line because it matched none of those three words — a self-inflicted blind spot from a filtered read, not a fact about the code. Lesson: for a pub enum whose variant list matters, read the definition unfiltered, don't grep for the variant names you expect to find.
• One real consequence of that same correction, fixed (2026-08-20): obikphylo::siblings::cache::Mat::SparsePresence(Layer< PersistentSparseBitMatrix>) was redundant — Mat::Presence(Layer< PersistentBitMatrix>) alone already handles sparse layers transparently, since PersistentBitMatrix absorbs Sparse internally. Removed the variant, the presence/is_multi.prsb probe in Mat::open (now just opens Layer::<PersistentBitMatrix> unconditionally for the non-count case — sparse-vs-dense is PersistentBitMatrix::open's own concern), and every now-single-armed match in find_slot/index_batch/ iter_minorants_batch/n_cols/fill_sub_matrix_carries. Full workspace test suite green after, including the 27 obikphylo::siblings tests that exercise pack_matrices(true)/sparse through Mat.

Remaining instance of the PartitionMeta-encapsulation problem

obikphylo::siblings::family_scan::scan_layer_families still re-derives index_dir from layer_dir.parent() and calls PartitionMeta::load itself, purely to get .mode for obilayeredmap::Layer::open (was Mat::open, same gap, survived the Mat → Layer swap in (1) unchanged). Fixing it the way the 21 other call sites were fixed needs more than a 1:1 swap: scan_layer_families only receives a bare layer_dir: &Path, not a (partition, part, layer) triple, and its single upstream source of layer paths, sibling_layer_dirs, returns a flat Vec<PathBuf> with the partition/layer indices already discarded. Fixing it properly means either having sibling_layer_dirs return (PathBuf, IndexMode) (or (part, layer)) pairs, or threading &KmerIndex + indices through instead of paths (not &KmerPartition — that type doesn't exist yet, and once it does it still won't know IndexMode, which lives on KmerIndex/ PartitionMeta) — and touching every one of scan_layer_families's 8 callers (distance.rs, alignment.rs, cardinality.rs, entropy.rs ×2, sankoff_bundle.rs ×2, stats.rs). Left alone this round; worth doing as part of the same pass that builds KmerPartition, since those callers are exactly the sibling-annex consumers it's meant to serve.