Push zunrplorkwkt #70
@@ -1,7 +1,9 @@
|
|||||||
# Partition and layer caching (discussion)
|
# Partition and layer caching (discussion)
|
||||||
|
|
||||||
Status: problem confirmed, design direction agreed (2026-08-20). No
|
Status: problem confirmed, design direction agreed (2026-08-20). (1)/(2)
|
||||||
implementation started. Ownership split (below) still open.
|
themselves not started; ownership split (below) still open. Preparatory
|
||||||
|
encapsulation work (partition/layer path and metadata accessors on
|
||||||
|
`KmerPartition`) landed the same day — see "Preparatory work done" below.
|
||||||
|
|
||||||
## The problem
|
## The problem
|
||||||
|
|
||||||
@@ -127,3 +129,116 @@ deleted outright or kept as a thin sibling-specific wrapper, and whether (2)
|
|||||||
needs an eviction policy or can simply hold every partition open for the
|
needs an eviction policy or can simply hold every partition open for the
|
||||||
process lifetime (revisit once the VM-mapping-count question above has a
|
process lifetime (revisit once the VM-mapping-count question above has a
|
||||||
real number behind it for this codebase's scale).
|
real number behind it for this codebase's scale).
|
||||||
|
|
||||||
|
## Preparatory work done (2026-08-20)
|
||||||
|
|
||||||
|
Groundwork for (1)/(2), landed ahead of the design itself:
|
||||||
|
|
||||||
|
- `KmerPartition` (`obikpartitionner`) gained `partition_dir`/`index_dir`/
|
||||||
|
`layer_dir` as the single source of truth for a partition's on-disk
|
||||||
|
layout, replacing per-module duplicated `const INDEX_SUBDIR: &str =
|
||||||
|
"index"` (7 copies) and ad hoc path joins — including one found
|
||||||
|
duplicated *inside `KmerPartition` itself* (`ensure_writer` rebuilt
|
||||||
|
`part_dir`'s own logic by hand).
|
||||||
|
- `KmerPartition` gained `partition_meta`/`n_layers`/`index_mode`, wrapping
|
||||||
|
`obilayeredmap::meta::PartitionMeta::load` (via the existing
|
||||||
|
`common::load_meta`, which also recovers indexes built before
|
||||||
|
`meta.json` existed). Before this, `obikphylo` and `obikindex` imported
|
||||||
|
`obilayeredmap::meta::PartitionMeta` directly and called `::load()`
|
||||||
|
themselves at 21 call sites, each redoing its own error-mapping —
|
||||||
|
every one of those crates knew the on-disk metadata format instead of
|
||||||
|
going through an interface. Fixed everywhere except one remaining spot
|
||||||
|
(below). Caught as a side effect: `dump_layer.rs`/`query_layer.rs` had
|
||||||
|
been calling `PartitionMeta::load` directly, bypassing `load_meta`
|
||||||
|
entirely — they never got the missing-`meta.json` recovery the other
|
||||||
|
callers did.
|
||||||
|
- Layer introspection API discussed but **not yet implemented** — three
|
||||||
|
axes, deliberately kept separate after an initial draft conflated them:
|
||||||
|
- `LayerContent { Count, Presence }` — what the layer stores; a `const`
|
||||||
|
on `LayerData` (compile-time, zero-cost), not a runtime field.
|
||||||
|
- `StorageKind { Implicit, Columnar, Packed, Sparse }` — how it's
|
||||||
|
stored; only meaningful for `D` that actually carry data (`Layer<()>`
|
||||||
|
has neither this nor `LayerContent` — it's a write-time-only state,
|
||||||
|
never a queryable content: once a layer is closed, "no matrix file"
|
||||||
|
reads back as `Presence`/`Implicit` via `PersistentBitMatrix::open`'s
|
||||||
|
own fallback, not as some third "empty" content).
|
||||||
|
- `EvidenceKind { Exact, Approx, Hybrid }` — from `MphfLayer`'s own
|
||||||
|
already-in-memory `LayerEvidence` discriminant.
|
||||||
|
- Not all `(LayerContent, StorageKind)` pairs are legal: `Count` never
|
||||||
|
has `Implicit` or `Sparse`.
|
||||||
|
|
||||||
|
**Implemented (2026-08-20).** `LayerContent`/`StorageKind`/`EvidenceKind`
|
||||||
|
now exist, each with two forms:
|
||||||
|
- A runtime accessor on an already-open value (`Layer<D>::content()`/
|
||||||
|
`storage_kind()`/`evidence_kind()`, `PersistentBitMatrix::storage_kind()`,
|
||||||
|
`PersistentCompactIntMatrix::storage_kind()`, `MphfLayer::evidence_kind()`)
|
||||||
|
— reads a discriminant already in memory, zero disk access.
|
||||||
|
- A lightweight `detect()`/`detect_storage()` disk probe that mirrors the
|
||||||
|
corresponding `open()`'s own priority order by hand (file-existence
|
||||||
|
checks only, no mmap) — usable *before* committing to a `D`, unlike the
|
||||||
|
runtime accessors. Exposed per-layer on `LayeredMap<D>` as
|
||||||
|
`detect_layer_content`/`detect_layer_storage`/`detect_layer_evidence`
|
||||||
|
(work regardless of `D`, since they only use `self.root` + the layer
|
||||||
|
index).
|
||||||
|
|
||||||
|
`StorageKind` lives in `obicompactvec` (owner of `PersistentBitMatrix`/
|
||||||
|
`PersistentCompactIntMatrix`); `LayerContent`/`EvidenceKind` live in
|
||||||
|
`obilayeredmap`. `HasLayerContent`/`HasStorageKind` gate `Layer<()>` out of
|
||||||
|
`content()`/`storage_kind()` (no matrix, nothing to report), matching the
|
||||||
|
"empty is transitional" conclusion above. 42 new tests across
|
||||||
|
`obilayeredmap`'s `tests/layer.rs` and `tests/map.rs`; full workspace
|
||||||
|
suite green (0 failed) after.
|
||||||
|
|
||||||
|
Not done: these `detect()` probes don't yet replace `Mat::open`'s or
|
||||||
|
`QueryLayer::open`'s own hand-rolled equivalents (still duplicated content/
|
||||||
|
storage decisions, now a *third* copy of the same logic to keep in sync)
|
||||||
|
— that consolidation is (1)/(2)'s job, not this prep step's.
|
||||||
|
|
||||||
|
## One bug found while reading around this (signalled, not fixed); one earlier claim retracted
|
||||||
|
|
||||||
|
- `obicompactvec::bitmatrix::sparse.rs`'s module doc says
|
||||||
|
"Not used by any production code path yet" — false since `obikmer pack
|
||||||
|
--sparse` (`cmd/pack/mod.rs`) is wired to `pack_sparse_bit_matrix` and
|
||||||
|
`Mat::open` already reads the result back in the sibling-annex path.
|
||||||
|
Stale comment, not corrected.
|
||||||
|
- **Retracted (2026-08-20)**: an earlier pass through this doc claimed
|
||||||
|
`obikpartitionner::query_layer::QueryLayer::open` had no sparse-format
|
||||||
|
detection and would silently corrupt reads on a `pack --sparse`d layer.
|
||||||
|
False — `PersistentBitMatrix` (`obicompactvec::bitmatrix::persistent`)
|
||||||
|
is a 4-way enum (`Columnar`/`Packed`/`Sparse`/`Implicit`), not 3-way as
|
||||||
|
first read; its `open()` already detects `Sparse` via
|
||||||
|
`presence/sparse_meta.json`, and every method on the type (`row`,
|
||||||
|
`fill_row`, `nonzero_iter`, …) already dispatches all 4 arms.
|
||||||
|
`QueryLayer::open`'s `PersistentBitMatrix::open(layer_dir)` call was
|
||||||
|
never the bug. Root cause of the false claim: a `grep -n
|
||||||
|
"Implicit\|Columnar\|Packed"` used to read the enum definition silently
|
||||||
|
skipped the `Sparse(...)` line because it matched none of those three
|
||||||
|
words — a self-inflicted blind spot from a filtered read, not a fact
|
||||||
|
about the code. Lesson: for a `pub enum` whose variant list matters,
|
||||||
|
read the definition unfiltered, don't grep for the variant names you
|
||||||
|
expect to find.
|
||||||
|
- One real consequence of that same correction: `obikphylo::siblings::
|
||||||
|
cache::Mat::SparsePresence(Layer<PersistentSparseBitMatrix>)` looks
|
||||||
|
redundant now — `Mat::Presence(Layer<PersistentBitMatrix>)` alone
|
||||||
|
would already handle sparse layers transparently, since
|
||||||
|
`PersistentBitMatrix` absorbs `Sparse` internally. Likely `Mat` predates
|
||||||
|
`PersistentBitMatrix` growing native sparse support. Signalled, not
|
||||||
|
removed — no mandate to touch `obikphylo` for this.
|
||||||
|
|
||||||
|
## Remaining instance of the PartitionMeta-encapsulation problem
|
||||||
|
|
||||||
|
`obikphylo::siblings::family_scan::scan_layer_families` still re-derives
|
||||||
|
`index_dir` from `layer_dir.parent()` and calls `PartitionMeta::load`
|
||||||
|
itself, purely to get `.mode` for `Mat::open`. Fixing it the way the 21
|
||||||
|
other call sites were fixed needs more than a 1:1 swap: `scan_layer_families`
|
||||||
|
only receives a bare `layer_dir: &Path`, not a `(partition, part, layer)`
|
||||||
|
triple, and its single upstream source of layer paths,
|
||||||
|
`sibling_layer_dirs`, returns a flat `Vec<PathBuf>` with the partition/layer
|
||||||
|
indices already discarded. Fixing it properly means either having
|
||||||
|
`sibling_layer_dirs` return `(PathBuf, IndexMode)` (or `(part, layer)`)
|
||||||
|
pairs, or threading `&KmerPartition` + indices through instead of paths —
|
||||||
|
and touching every one of `scan_layer_families`'s 8 callers (`distance.rs`,
|
||||||
|
`alignment.rs`, `cardinality.rs`, `entropy.rs` ×2, `sankoff_bundle.rs` ×2,
|
||||||
|
`stats.rs`). Left alone this round; worth doing as part of the same pass
|
||||||
|
that builds (1)/(2), since those callers are exactly the sibling-annex
|
||||||
|
consumers (2) is meant to serve.
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ use ndarray::{Array1, Array2};
|
|||||||
use crate::bitvec::{PersistentBitVec, PersistentBitVecBuilder};
|
use crate::bitvec::{PersistentBitVec, PersistentBitVecBuilder};
|
||||||
use crate::layer_meta::LayerMeta;
|
use crate::layer_meta::LayerMeta;
|
||||||
use crate::meta::MatrixMeta;
|
use crate::meta::MatrixMeta;
|
||||||
|
use crate::storage_kind::StorageKind;
|
||||||
use crate::traits::{BitPartials, ColumnWeights};
|
use crate::traits::{BitPartials, ColumnWeights};
|
||||||
use crate::views::{nonzero_triples, BitSliceView};
|
use crate::views::{nonzero_triples, BitSliceView};
|
||||||
|
|
||||||
@@ -63,6 +64,42 @@ impl PersistentBitMatrix {
|
|||||||
Ok(Self::Implicit { n_rows: meta.n, n_cols: 1 })
|
Ok(Self::Implicit { n_rows: meta.n, n_cols: 1 })
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// This already-open matrix's storage format — reads the discriminant
|
||||||
|
/// already in memory, no disk access.
|
||||||
|
#[inline]
|
||||||
|
pub fn storage_kind(&self) -> StorageKind {
|
||||||
|
match self {
|
||||||
|
Self::Columnar(_) => StorageKind::Columnar,
|
||||||
|
Self::Packed(_) => StorageKind::Packed,
|
||||||
|
Self::Sparse(_) => StorageKind::Sparse,
|
||||||
|
Self::Implicit { .. } => StorageKind::Implicit,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Lightweight disk probe: which format `open` would pick, without
|
||||||
|
/// opening (mmap'ing) anything. Mirrors `open`'s own priority order by
|
||||||
|
/// hand — keep the two in sync if that order ever changes.
|
||||||
|
pub fn detect_storage(layer_dir: &Path) -> io::Result<StorageKind> {
|
||||||
|
let presence_dir = layer_dir.join("presence");
|
||||||
|
|
||||||
|
if presence_dir.join("matrix.pbmx").exists() {
|
||||||
|
return Ok(StorageKind::Packed);
|
||||||
|
}
|
||||||
|
if MatrixMeta::load(&presence_dir).is_ok() {
|
||||||
|
return Ok(StorageKind::Columnar);
|
||||||
|
}
|
||||||
|
if presence_dir.join("sparse_meta.json").exists() {
|
||||||
|
return Ok(StorageKind::Sparse);
|
||||||
|
}
|
||||||
|
if LayerMeta::load(layer_dir).is_ok() {
|
||||||
|
return Ok(StorageKind::Implicit);
|
||||||
|
}
|
||||||
|
Err(io::Error::new(
|
||||||
|
io::ErrorKind::NotFound,
|
||||||
|
format!("cannot detect presence storage format in {}", layer_dir.display()),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
#[inline]
|
#[inline]
|
||||||
pub fn n(&self) -> usize {
|
pub fn n(&self) -> usize {
|
||||||
match self {
|
match self {
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ use crate::colgroup::{chunked_presence_count, ColGroup, MatrixGroupOps};
|
|||||||
use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE};
|
use crate::format::{HEADER_SIZE, OVERFLOW_ENTRY_SIZE};
|
||||||
use crate::meta::MatrixMeta;
|
use crate::meta::MatrixMeta;
|
||||||
use crate::reader::PersistentCompactIntVec;
|
use crate::reader::PersistentCompactIntVec;
|
||||||
|
use crate::storage_kind::StorageKind;
|
||||||
use crate::tempbitvec::{TempBitVec, TempBitVecBuilder};
|
use crate::tempbitvec::{TempBitVec, TempBitVecBuilder};
|
||||||
use crate::tempintvec::{TempCompactIntVec, TempCompactIntVecBuilder};
|
use crate::tempintvec::{TempCompactIntVec, TempCompactIntVecBuilder};
|
||||||
use crate::views::{nonzero_triples, IntSliceView};
|
use crate::views::{nonzero_triples, IntSliceView};
|
||||||
@@ -287,6 +288,33 @@ impl PersistentCompactIntMatrix {
|
|||||||
))
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// This already-open matrix's storage format — reads the discriminant
|
||||||
|
/// already in memory, no disk access.
|
||||||
|
#[inline]
|
||||||
|
pub fn storage_kind(&self) -> StorageKind {
|
||||||
|
match self {
|
||||||
|
Self::Columnar(_) => StorageKind::Columnar,
|
||||||
|
Self::Packed(_) => StorageKind::Packed,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Lightweight disk probe: which format `open` would pick, without
|
||||||
|
/// opening (mmap'ing) anything. Mirrors `open`'s own priority order by
|
||||||
|
/// hand — keep the two in sync if that order ever changes.
|
||||||
|
pub fn detect_storage(layer_dir: &Path) -> io::Result<StorageKind> {
|
||||||
|
let counts_dir = layer_dir.join("counts");
|
||||||
|
if counts_dir.join("matrix.pcmx").exists() {
|
||||||
|
return Ok(StorageKind::Packed);
|
||||||
|
}
|
||||||
|
if MatrixMeta::load(&counts_dir).is_ok() {
|
||||||
|
return Ok(StorageKind::Columnar);
|
||||||
|
}
|
||||||
|
Err(io::Error::new(
|
||||||
|
io::ErrorKind::NotFound,
|
||||||
|
format!("no count matrix found in {}", layer_dir.display()),
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
#[inline]
|
#[inline]
|
||||||
pub fn n(&self) -> usize {
|
pub fn n(&self) -> usize {
|
||||||
match self { Self::Columnar(m) => m.n(), Self::Packed(m) => m.n_rows }
|
match self { Self::Columnar(m) => m.n(), Self::Packed(m) => m.n_rows }
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ mod intmatrix;
|
|||||||
mod layer_meta;
|
mod layer_meta;
|
||||||
mod meta;
|
mod meta;
|
||||||
mod reader;
|
mod reader;
|
||||||
|
mod storage_kind;
|
||||||
mod tempbitvec;
|
mod tempbitvec;
|
||||||
mod tempintvec;
|
mod tempintvec;
|
||||||
mod views;
|
mod views;
|
||||||
@@ -25,6 +26,7 @@ pub use colgroup::{ColGroup, FilterMask, MatrixGroupOps, eval_filter_mask};
|
|||||||
pub use intmatrix::{PersistentCompactIntMatrix, PersistentCompactIntMatrixBuilder, pack_compact_int_matrix};
|
pub use intmatrix::{PersistentCompactIntMatrix, PersistentCompactIntMatrixBuilder, pack_compact_int_matrix};
|
||||||
pub use layer_meta::LayerMeta;
|
pub use layer_meta::LayerMeta;
|
||||||
pub use reader::{PersistentCompactIntVec, Iter as CompactIntVecIter};
|
pub use reader::{PersistentCompactIntVec, Iter as CompactIntVecIter};
|
||||||
|
pub use storage_kind::StorageKind;
|
||||||
pub use tempbitvec::{TempBitVec, TempBitVecBuilder};
|
pub use tempbitvec::{TempBitVec, TempBitVecBuilder};
|
||||||
pub use tempintvec::{TempCompactIntVec, TempCompactIntVecBuilder};
|
pub use tempintvec::{TempCompactIntVec, TempCompactIntVecBuilder};
|
||||||
pub use traits::{BinaryMatrix, BitPartials, ColumnWeights, CountPartials};
|
pub use traits::{BinaryMatrix, BitPartials, ColumnWeights, CountPartials};
|
||||||
|
|||||||
@@ -0,0 +1,19 @@
|
|||||||
|
//! How a persistent matrix's data is physically laid out on disk —
|
||||||
|
//! orthogonal to *what* it stores (count vs. presence, `obilayeredmap`'s
|
||||||
|
//! `LayerContent` concern, one crate up). `PersistentCompactIntMatrix` only
|
||||||
|
//! ever reports `Columnar`/`Packed`; `PersistentBitMatrix` is the only type
|
||||||
|
//! that can also report `Sparse`/`Implicit`.
|
||||||
|
|
||||||
|
/// On-disk storage format of a persistent matrix.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum StorageKind {
|
||||||
|
/// Per-column files (`.pbiv`/`.pciv`), the build-time format.
|
||||||
|
Columnar,
|
||||||
|
/// Single mmap'd file (`matrix.pbmx`/`matrix.pcmx`), query-optimised.
|
||||||
|
Packed,
|
||||||
|
/// Row-major deduplicated sparse format (`PersistentBitMatrix` only).
|
||||||
|
Sparse,
|
||||||
|
/// No file at all — mono-genome, all values present
|
||||||
|
/// (`PersistentBitMatrix` only).
|
||||||
|
Implicit,
|
||||||
|
}
|
||||||
@@ -3,7 +3,6 @@ use std::fs;
|
|||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use obikpartitionner::{KmerPartition, KmerSpectrum, PARTITIONS_SUBDIR};
|
use obikpartitionner::{KmerPartition, KmerSpectrum, PARTITIONS_SUBDIR};
|
||||||
use obilayeredmap;
|
|
||||||
use obisys::{Reporter, Stage, progress_bar};
|
use obisys::{Reporter, Stage, progress_bar};
|
||||||
use rayon::prelude::*;
|
use rayon::prelude::*;
|
||||||
use tracing::info;
|
use tracing::info;
|
||||||
@@ -148,11 +147,7 @@ impl KmerIndex {
|
|||||||
/// homogeneous across all partitions — reading it off partition 0
|
/// homogeneous across all partitions — reading it off partition 0
|
||||||
/// is enough, no need to scan every partition.
|
/// is enough, no need to scan every partition.
|
||||||
pub fn n_layers_per_partition(&self) -> OKIResult<usize> {
|
pub fn n_layers_per_partition(&self) -> OKIResult<usize> {
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
Ok(self.partition.n_layers(0)?)
|
||||||
let index_dir = self.partition.index_dir(0);
|
|
||||||
let meta = PartitionMeta::load(&index_dir)
|
|
||||||
.map_err(|e| OKIError::Io(std::io::Error::new(std::io::ErrorKind::Other, e.to_string())))?;
|
|
||||||
Ok(meta.n_layers)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Expose the inner partition so the caller can run scatter into it.
|
/// Expose the inner partition so the caller can run scatter into it.
|
||||||
@@ -279,7 +274,6 @@ impl KmerIndex {
|
|||||||
/// (see the sparse-matrix design plan's "Explicitly deferred").
|
/// (see the sparse-matrix design plan's "Explicitly deferred").
|
||||||
pub fn pack_matrices(&self, sparse: bool) -> OKIResult<()> {
|
pub fn pack_matrices(&self, sparse: bool) -> OKIResult<()> {
|
||||||
use obicompactvec::{pack_bit_matrix, pack_compact_int_matrix, pack_sparse_bit_matrix};
|
use obicompactvec::{pack_bit_matrix, pack_compact_int_matrix, pack_sparse_bit_matrix};
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
|
|
||||||
let n = self.n_partitions();
|
let n = self.n_partitions();
|
||||||
let order: Vec<usize> = (0..n).collect();
|
let order: Vec<usize> = (0..n).collect();
|
||||||
@@ -289,9 +283,8 @@ impl KmerIndex {
|
|||||||
|i| -> OKIResult<()> {
|
|i| -> OKIResult<()> {
|
||||||
let index_dir = self.partition.index_dir(i);
|
let index_dir = self.partition.index_dir(i);
|
||||||
if !index_dir.exists() { return Ok(()); }
|
if !index_dir.exists() { return Ok(()); }
|
||||||
let meta = PartitionMeta::load(&index_dir)
|
let n_layers = self.partition.n_layers(i)?;
|
||||||
.map_err(|e| OKIError::Io(std::io::Error::new(std::io::ErrorKind::Other, e.to_string())))?;
|
for l in 0..n_layers {
|
||||||
for l in 0..meta.n_layers {
|
|
||||||
let layer_dir = self.partition.layer_dir(i, l);
|
let layer_dir = self.partition.layer_dir(i, l);
|
||||||
let presence_dir = layer_dir.join("presence");
|
let presence_dir = layer_dir.join("presence");
|
||||||
let counts_dir = layer_dir.join("counts");
|
let counts_dir = layer_dir.join("counts");
|
||||||
@@ -319,7 +312,6 @@ impl KmerIndex {
|
|||||||
pub fn upgrade_layer_meta(&self) -> OKIResult<()> {
|
pub fn upgrade_layer_meta(&self) -> OKIResult<()> {
|
||||||
use obicompactvec::LayerMeta;
|
use obicompactvec::LayerMeta;
|
||||||
use obiskio::UnitigFileReader;
|
use obiskio::UnitigFileReader;
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
|
|
||||||
let n = self.n_partitions();
|
let n = self.n_partitions();
|
||||||
let errors: Vec<_> = (0..n)
|
let errors: Vec<_> = (0..n)
|
||||||
@@ -327,11 +319,11 @@ impl KmerIndex {
|
|||||||
.filter_map(|i| {
|
.filter_map(|i| {
|
||||||
let index_dir = self.partition.index_dir(i);
|
let index_dir = self.partition.index_dir(i);
|
||||||
if !index_dir.exists() { return None; }
|
if !index_dir.exists() { return None; }
|
||||||
let meta = match PartitionMeta::load(&index_dir) {
|
let n_layers = match self.partition.n_layers(i) {
|
||||||
Ok(m) => m,
|
Ok(n) => n,
|
||||||
Err(e) => return Some(OKIError::Io(std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))),
|
Err(e) => return Some(OKIError::Io(std::io::Error::new(std::io::ErrorKind::Other, e.to_string()))),
|
||||||
};
|
};
|
||||||
for l in 0..meta.n_layers {
|
for l in 0..n_layers {
|
||||||
let layer_dir = self.partition.layer_dir(i, l);
|
let layer_dir = self.partition.layer_dir(i, l);
|
||||||
let meta_path = layer_dir.join(LayerMeta::FILENAME);
|
let meta_path = layer_dir.join(LayerMeta::FILENAME);
|
||||||
if meta_path.exists() { continue; }
|
if meta_path.exists() { continue; }
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
use std::fs;
|
use std::fs;
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
use obilayeredmap::{layer_dir, IndexMode, layer::Layer};
|
use obikpartitionner::KmerPartition;
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
use obilayeredmap::{IndexMode, layer::Layer};
|
||||||
use obisys::{Reporter, Stage, progress_bar};
|
use obisys::{Reporter, Stage, progress_bar};
|
||||||
use tracing::info;
|
use tracing::info;
|
||||||
|
|
||||||
@@ -48,7 +48,7 @@ impl KmerIndex {
|
|||||||
let runner = crate::numa::PartitionRunner::new();
|
let runner = crate::numa::PartitionRunner::new();
|
||||||
runner.run(
|
runner.run(
|
||||||
&order,
|
&order,
|
||||||
|i| reindex_partition(&self.partition.index_dir(i), &target, block_bits)
|
|i| reindex_partition(&self.partition, i, &target, block_bits)
|
||||||
.map_err(|e| OKIError::InvalidInput(format!("partition {i}: {e}"))),
|
.map_err(|e| OKIError::InvalidInput(format!("partition {i}: {e}"))),
|
||||||
|_, _, _| { pb.inc(1); },
|
|_, _, _| { pb.inc(1); },
|
||||||
)?;
|
)?;
|
||||||
@@ -66,13 +66,13 @@ impl KmerIndex {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Process all layers of one partition's index directory.
|
/// Process all layers of one partition's index directory.
|
||||||
fn reindex_partition(index_dir: &Path, target: &IndexMode, block_bits: u8) -> OKIResult<()> {
|
fn reindex_partition(partition: &KmerPartition, i: usize, target: &IndexMode, block_bits: u8) -> OKIResult<()> {
|
||||||
if !index_dir.exists() {
|
if !partition.index_dir(i).exists() {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
let pm = PartitionMeta::load(index_dir).map_err(olm_to_oki)?;
|
let n_layers = partition.n_layers(i).map_err(|e| OKIError::InvalidInput(e.to_string()))?;
|
||||||
for layer_idx in 0..pm.n_layers {
|
for layer_idx in 0..n_layers {
|
||||||
reindex_layer(&layer_dir(index_dir, layer_idx), target, block_bits)?;
|
reindex_layer(&partition.layer_dir(i, layer_idx), target, block_bits)?;
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ use std::path::Path;
|
|||||||
|
|
||||||
use obicompactvec::{LayerMeta, PersistentBitMatrix, PersistentCompactIntMatrix};
|
use obicompactvec::{LayerMeta, PersistentBitMatrix, PersistentCompactIntMatrix};
|
||||||
use obicompactvec::traits::ColumnWeights;
|
use obicompactvec::traits::ColumnWeights;
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
use rayon::prelude::*;
|
use rayon::prelude::*;
|
||||||
|
|
||||||
use crate::error::OKIResult;
|
use crate::error::OKIResult;
|
||||||
@@ -93,9 +92,7 @@ impl KmerIndex {
|
|||||||
let index_dir = self.partition.index_dir(i);
|
let index_dir = self.partition.index_dir(i);
|
||||||
if !index_dir.exists() { return (0usize, 0u64, 0u64, 0u64); }
|
if !index_dir.exists() { return (0usize, 0u64, 0u64, 0u64); }
|
||||||
|
|
||||||
let n_layers = PartitionMeta::load(&index_dir)
|
let n_layers = self.partition.n_layers(i).unwrap_or(0);
|
||||||
.map(|m| m.n_layers)
|
|
||||||
.unwrap_or(0);
|
|
||||||
|
|
||||||
(0..n_layers).fold((0usize, 0u64, 0u64, 0u64), |acc, l| {
|
(0..n_layers).fold((0usize, 0u64, 0u64, 0u64), |acc, l| {
|
||||||
let lb = layer_bytes(&self.partition.layer_dir(i, l));
|
let lb = layer_bytes(&self.partition.layer_dir(i, l));
|
||||||
@@ -144,9 +141,7 @@ impl KmerIndex {
|
|||||||
let index_dir = self.partition.index_dir(i);
|
let index_dir = self.partition.index_dir(i);
|
||||||
if !index_dir.exists() { return (0, counts); }
|
if !index_dir.exists() { return (0, counts); }
|
||||||
|
|
||||||
let n_layers = PartitionMeta::load(&index_dir)
|
let n_layers = self.partition.n_layers(i).unwrap_or(0);
|
||||||
.map(|m| m.n_layers)
|
|
||||||
.unwrap_or(0);
|
|
||||||
|
|
||||||
for l in 0..n_layers {
|
for l in 0..n_layers {
|
||||||
let this_layer_dir = self.partition.layer_dir(i, l);
|
let this_layer_dir = self.partition.layer_dir(i, l);
|
||||||
|
|||||||
@@ -2,7 +2,6 @@ use obicompactvec::{PersistentBitMatrix, PersistentCompactIntMatrix};
|
|||||||
use obikseq::CanonicalKmer;
|
use obikseq::CanonicalKmer;
|
||||||
use obiskio::{SKError, SKResult, UnitigFileReader};
|
use obiskio::{SKError, SKResult, UnitigFileReader};
|
||||||
use obilayeredmap::{IndexMode, MphfLayer, OLMError};
|
use obilayeredmap::{IndexMode, MphfLayer, OLMError};
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
|
|
||||||
use crate::filter::{KmerFilter, passes_all};
|
use crate::filter::{KmerFilter, passes_all};
|
||||||
use crate::partition::KmerPartition;
|
use crate::partition::KmerPartition;
|
||||||
@@ -39,9 +38,7 @@ impl KmerPartition {
|
|||||||
return Ok(true);
|
return Ok(true);
|
||||||
}
|
}
|
||||||
|
|
||||||
let index_mode = PartitionMeta::load(&index_dir)
|
let index_mode = self.index_mode(part).unwrap_or(IndexMode::Exact);
|
||||||
.map(|m| m.mode)
|
|
||||||
.unwrap_or(IndexMode::Exact);
|
|
||||||
|
|
||||||
let mut l = 0;
|
let mut l = 0;
|
||||||
loop {
|
loop {
|
||||||
@@ -119,9 +116,7 @@ impl KmerPartition {
|
|||||||
return Ok(true);
|
return Ok(true);
|
||||||
}
|
}
|
||||||
|
|
||||||
let index_mode = PartitionMeta::load(&index_dir)
|
let index_mode = self.index_mode(part).unwrap_or(IndexMode::Exact);
|
||||||
.map(|m| m.mode)
|
|
||||||
.unwrap_or(IndexMode::Exact);
|
|
||||||
|
|
||||||
let mut layer = 0;
|
let mut layer = 0;
|
||||||
loop {
|
loop {
|
||||||
|
|||||||
@@ -21,7 +21,6 @@ use obipipeline::{
|
|||||||
|
|
||||||
use obicompactvec::{PersistentBitMatrixBuilder, PersistentCompactIntMatrixBuilder};
|
use obicompactvec::{PersistentBitMatrixBuilder, PersistentCompactIntMatrixBuilder};
|
||||||
use obikseq::CanonicalKmer;
|
use obikseq::CanonicalKmer;
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
use obilayeredmap::{layer_dir, IndexMode, Layer, LayeredMap, MphfOnly};
|
use obilayeredmap::{layer_dir, IndexMode, Layer, LayeredMap, MphfOnly};
|
||||||
use obiskio::{SKError, SKResult, UnitigFileReader};
|
use obiskio::{SKError, SKResult, UnitigFileReader};
|
||||||
|
|
||||||
@@ -526,7 +525,7 @@ impl KmerPartition {
|
|||||||
if let Some(mb) = new_mb {
|
if let Some(mb) = new_mb {
|
||||||
mb.close().map_err(SKError::Io)?;
|
mb.close().map_err(SKError::Io)?;
|
||||||
|
|
||||||
let mut part_meta = PartitionMeta::load(&dst_index_dir).map_err(|e| olm_to_sk(e, "merge"))?;
|
let mut part_meta = self.partition_meta(i)?;
|
||||||
part_meta.n_layers = new_layer_idx + 1;
|
part_meta.n_layers = new_layer_idx + 1;
|
||||||
part_meta.save(&dst_index_dir).map_err(|e| olm_to_sk(e, "merge"))?;
|
part_meta.save(&dst_index_dir).map_err(|e| olm_to_sk(e, "merge"))?;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,6 +8,8 @@ use obisys::progress_bar;
|
|||||||
|
|
||||||
use obikseq::RoutableSuperKmer;
|
use obikseq::RoutableSuperKmer;
|
||||||
use obiskio::SKResult;
|
use obiskio::SKResult;
|
||||||
|
use obilayeredmap::IndexMode;
|
||||||
|
use obilayeredmap::meta::PartitionMeta;
|
||||||
use rayon::prelude::*;
|
use rayon::prelude::*;
|
||||||
use remove_dir_all::remove_dir_all;
|
use remove_dir_all::remove_dir_all;
|
||||||
use sysinfo::System;
|
use sysinfo::System;
|
||||||
@@ -16,6 +18,7 @@ use niffler::Level;
|
|||||||
use niffler::send::compression::Format;
|
use niffler::send::compression::Format;
|
||||||
use obiskio::SKFileWriter;
|
use obiskio::SKFileWriter;
|
||||||
|
|
||||||
|
use crate::common::load_meta;
|
||||||
use crate::kmer_sort::chunk_size_from_ram;
|
use crate::kmer_sort::chunk_size_from_ram;
|
||||||
|
|
||||||
use super::count::count_partition;
|
use super::count::count_partition;
|
||||||
@@ -191,6 +194,26 @@ impl KmerPartition {
|
|||||||
obilayeredmap::layer_dir(&self.index_dir(i), l)
|
obilayeredmap::layer_dir(&self.index_dir(i), l)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Partition `i`'s metadata (layer count, evidence mode) — the single
|
||||||
|
/// entry point for this, so that callers outside `obikpartitionner`
|
||||||
|
/// never need to know it's a `meta.json` loaded via
|
||||||
|
/// `obilayeredmap::meta::PartitionMeta`, nor handle its own recovery
|
||||||
|
/// path for indexes built before that file existed (see
|
||||||
|
/// [`crate::common::load_meta`]).
|
||||||
|
pub fn partition_meta(&self, i: usize) -> SKResult<PartitionMeta> {
|
||||||
|
load_meta(&self.index_dir(i), "partition_meta")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Number of layers in partition `i` — see [`partition_meta`](Self::partition_meta).
|
||||||
|
pub fn n_layers(&self, i: usize) -> SKResult<usize> {
|
||||||
|
self.partition_meta(i).map(|m| m.n_layers)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Evidence mode of partition `i` — see [`partition_meta`](Self::partition_meta).
|
||||||
|
pub fn index_mode(&self, i: usize) -> SKResult<IndexMode> {
|
||||||
|
self.partition_meta(i).map(|m| m.mode)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn kmer_size(&self) -> usize {
|
pub fn kmer_size(&self) -> usize {
|
||||||
self.kmer_size
|
self.kmer_size
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,6 @@ use obicompactvec::{PersistentBitMatrix, PersistentCompactIntMatrix};
|
|||||||
use obikseq::CanonicalKmer;
|
use obikseq::CanonicalKmer;
|
||||||
use obiskio::{SKError, SKResult};
|
use obiskio::{SKError, SKResult};
|
||||||
use obilayeredmap::{IndexMode, MphfLayer, OLMError};
|
use obilayeredmap::{IndexMode, MphfLayer, OLMError};
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
|
|
||||||
use crate::partition::KmerPartition;
|
use crate::partition::KmerPartition;
|
||||||
|
|
||||||
@@ -178,7 +177,7 @@ impl KmerPartition {
|
|||||||
return Ok(stats);
|
return Ok(stats);
|
||||||
}
|
}
|
||||||
|
|
||||||
let meta = PartitionMeta::load(&index_dir).map_err(olm_to_sk)?;
|
let meta = self.partition_meta(part_idx)?;
|
||||||
let layers: Vec<QueryLayer> = (0..meta.n_layers)
|
let layers: Vec<QueryLayer> = (0..meta.n_layers)
|
||||||
.map(|i| QueryLayer::open(&self.layer_dir(part_idx, i), with_counts, &meta.mode))
|
.map(|i| QueryLayer::open(&self.layer_dir(part_idx, i), with_counts, &meta.mode))
|
||||||
.collect::<SKResult<_>>()?;
|
.collect::<SKResult<_>>()?;
|
||||||
|
|||||||
@@ -7,7 +7,6 @@ use obicompactvec::{
|
|||||||
PersistentBitMatrix, PersistentBitMatrixBuilder,
|
PersistentBitMatrix, PersistentBitMatrixBuilder,
|
||||||
PersistentCompactIntMatrix, PersistentCompactIntMatrixBuilder,
|
PersistentCompactIntMatrix, PersistentCompactIntMatrixBuilder,
|
||||||
};
|
};
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
use obilayeredmap::OLMError;
|
use obilayeredmap::OLMError;
|
||||||
use obiskio::{SKError, SKResult};
|
use obiskio::{SKError, SKResult};
|
||||||
|
|
||||||
@@ -150,8 +149,8 @@ impl KmerPartition {
|
|||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
|
|
||||||
let src_meta = PartitionMeta::load(&src_index_dir).map_err(olm_to_sk)?;
|
let n_src_layers = src.n_layers(i)?;
|
||||||
if src_meta.n_layers == 0 {
|
if n_src_layers == 0 {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -162,7 +161,7 @@ impl KmerPartition {
|
|||||||
|
|
||||||
let data_subdir = if output_presence { "presence" } else { "counts" };
|
let data_subdir = if output_presence { "presence" } else { "counts" };
|
||||||
|
|
||||||
for l in 0..src_meta.n_layers {
|
for l in 0..n_src_layers {
|
||||||
let src_layer_dir = src.layer_dir(i, l);
|
let src_layer_dir = src.layer_dir(i, l);
|
||||||
if !src_layer_dir.exists() { continue; }
|
if !src_layer_dir.exists() { continue; }
|
||||||
|
|
||||||
@@ -234,8 +233,7 @@ impl KmerPartition {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if !in_place {
|
if !in_place {
|
||||||
PartitionMeta::load(&src_index_dir).map_err(olm_to_sk)?
|
src.partition_meta(i)?.save(&dst_index_dir).map_err(olm_to_sk)?;
|
||||||
.save(&dst_index_dir).map_err(olm_to_sk)?;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ use obikpartitionner::KmerPartition;
|
|||||||
use obipipeline::ThrottleGuard;
|
use obipipeline::ThrottleGuard;
|
||||||
use obikseq::CanonicalKmer;
|
use obikseq::CanonicalKmer;
|
||||||
use obilayeredmap::MphfLayer;
|
use obilayeredmap::MphfLayer;
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
use obilayeredmap::meta::IndexMode;
|
||||||
use obisys::progress_bar;
|
use obisys::progress_bar;
|
||||||
|
|
||||||
use obikindex::{OKIError, OKIResult};
|
use obikindex::{OKIError, OKIResult};
|
||||||
@@ -97,11 +97,13 @@ impl SiblingAnnexBuildExt for KmerIndex {
|
|||||||
pb.inc(1);
|
pb.inc(1);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let meta = PartitionMeta::load(&index_dir).map_err(olm_to_ok)?;
|
let meta = self.partition().partition_meta(part)?;
|
||||||
|
|
||||||
let mut part_slots: u64 = 0;
|
let mut part_slots: u64 = 0;
|
||||||
for l in 0..meta.n_layers {
|
for l in 0..meta.n_layers {
|
||||||
part_slots += build_layer_sibling_annex(self, &self.partition().layer_dir(part, l), n_parts, l, &cache)?;
|
part_slots += build_layer_sibling_annex(
|
||||||
|
self, &self.partition().layer_dir(part, l), &meta.mode, n_parts, l, &cache,
|
||||||
|
)?;
|
||||||
}
|
}
|
||||||
total_slots += part_slots;
|
total_slots += part_slots;
|
||||||
pb.inc(1);
|
pb.inc(1);
|
||||||
@@ -120,13 +122,12 @@ impl SiblingAnnexBuildExt for KmerIndex {
|
|||||||
fn build_layer_sibling_annex(
|
fn build_layer_sibling_annex(
|
||||||
index: &KmerIndex,
|
index: &KmerIndex,
|
||||||
layer_dir: &Path,
|
layer_dir: &Path,
|
||||||
|
mode: &IndexMode,
|
||||||
n_parts: usize,
|
n_parts: usize,
|
||||||
l: usize,
|
l: usize,
|
||||||
cache: &Arc<PartitionCache>,
|
cache: &Arc<PartitionCache>,
|
||||||
) -> OKIResult<u64> {
|
) -> OKIResult<u64> {
|
||||||
let index_dir = layer_dir.parent().expect("layer_dir has a parent index dir");
|
let mphf = MphfLayer::open(layer_dir, mode).map_err(olm_to_ok)?;
|
||||||
let meta = PartitionMeta::load(index_dir).map_err(olm_to_ok)?;
|
|
||||||
let mphf = MphfLayer::open(layer_dir, &meta.mode).map_err(olm_to_ok)?;
|
|
||||||
let k = index.kmer_size();
|
let k = index.kmer_size();
|
||||||
let n = mphf.n();
|
let n = mphf.n();
|
||||||
|
|
||||||
|
|||||||
@@ -6,13 +6,13 @@ use obicompactvec::{PersistentBitMatrix, PersistentCompactIntMatrix, PersistentS
|
|||||||
use obikpartitionner::KmerPartition;
|
use obikpartitionner::KmerPartition;
|
||||||
use obikseq::CanonicalKmer;
|
use obikseq::CanonicalKmer;
|
||||||
use obilayeredmap::{Layer, OLMResult};
|
use obilayeredmap::{Layer, OLMResult};
|
||||||
use obilayeredmap::meta::{IndexMode, PartitionMeta};
|
use obilayeredmap::meta::IndexMode;
|
||||||
use obisys::progress_bar;
|
use obisys::progress_bar;
|
||||||
|
|
||||||
use obikindex::OKIResult;
|
use obikindex::OKIResult;
|
||||||
|
|
||||||
use super::iter::SiblingLayerExt;
|
use super::iter::SiblingLayerExt;
|
||||||
use super::{olm_to_ok, SiblingAnnex};
|
use super::SiblingAnnex;
|
||||||
|
|
||||||
/// Every partition's already-open layers, built **once** for the whole
|
/// Every partition's already-open layers, built **once** for the whole
|
||||||
/// `build_sibling_annex` run and shared (read-only) across every lookup, in
|
/// `build_sibling_annex` run and shared (read-only) across every lookup, in
|
||||||
@@ -179,7 +179,7 @@ impl PartitionCache {
|
|||||||
pb.inc(1);
|
pb.inc(1);
|
||||||
return Ok((Vec::new(), 0));
|
return Ok((Vec::new(), 0));
|
||||||
}
|
}
|
||||||
let meta = PartitionMeta::load(&index_dir).map_err(olm_to_ok)?;
|
let meta = partition.partition_meta(part)?;
|
||||||
let mut mats = Vec::with_capacity(meta.n_layers);
|
let mut mats = Vec::with_capacity(meta.n_layers);
|
||||||
for l in 0..meta.n_layers {
|
for l in 0..meta.n_layers {
|
||||||
let Ok(mat) = Mat::open(&partition.layer_dir(part, l), &meta.mode, with_counts) else { continue };
|
let Ok(mat) = Mat::open(&partition.layer_dir(part, l), &meta.mode, with_counts) else { continue };
|
||||||
|
|||||||
@@ -109,8 +109,8 @@ pub(crate) fn sibling_layer_dirs(index: &KmerIndex) -> OKIResult<Vec<PathBuf>> {
|
|||||||
if !index_dir.exists() {
|
if !index_dir.exists() {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
let meta = PartitionMeta::load(&index_dir).map_err(olm_to_ok)?;
|
let n_layers = index.partition().n_layers(part)?;
|
||||||
for l in 0..meta.n_layers {
|
for l in 0..n_layers {
|
||||||
let this_layer_dir = index.partition().layer_dir(part, l);
|
let this_layer_dir = index.partition().layer_dir(part, l);
|
||||||
let annex_path = this_layer_dir.join(ANNEX_FILE_NAME);
|
let annex_path = this_layer_dir.join(ANNEX_FILE_NAME);
|
||||||
if !annex_path.exists() {
|
if !annex_path.exists() {
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ use std::path::Path;
|
|||||||
|
|
||||||
use obikseq::{CanonicalKmer, Kmer, Sequence};
|
use obikseq::{CanonicalKmer, Kmer, Sequence};
|
||||||
use obilayeredmap::MphfLayer;
|
use obilayeredmap::MphfLayer;
|
||||||
use obilayeredmap::meta::PartitionMeta;
|
|
||||||
use obisys::Reporter;
|
use obisys::Reporter;
|
||||||
use tempfile::tempdir;
|
use tempfile::tempdir;
|
||||||
|
|
||||||
@@ -88,8 +87,7 @@ fn canonical(ascii: &[u8]) -> CanonicalKmer {
|
|||||||
/// Read back the annex entry for a given canonical k-mer from the merged
|
/// Read back the annex entry for a given canonical k-mer from the merged
|
||||||
/// index's (single) partition/layer, asserting it was found at all.
|
/// index's (single) partition/layer, asserting it was found at all.
|
||||||
fn annex_info_for(idx: &KmerIndex, kmer: CanonicalKmer) -> FamilyMask {
|
fn annex_info_for(idx: &KmerIndex, kmer: CanonicalKmer) -> FamilyMask {
|
||||||
let index_dir = idx.partition().index_dir(0);
|
let meta = idx.partition().partition_meta(0).unwrap();
|
||||||
let meta = PartitionMeta::load(&index_dir).unwrap();
|
|
||||||
for l in 0..meta.n_layers {
|
for l in 0..meta.n_layers {
|
||||||
let layer_dir = idx.partition().layer_dir(0, l);
|
let layer_dir = idx.partition().layer_dir(0, l);
|
||||||
let mphf = MphfLayer::open(&layer_dir, &meta.mode).unwrap();
|
let mphf = MphfLayer::open(&layer_dir, &meta.mode).unwrap();
|
||||||
@@ -310,9 +308,8 @@ fn sibling_annex_no_empty_masks_after_build() {
|
|||||||
let g1 = build_single_genome_index(dir.path(), "g1", &seq);
|
let g1 = build_single_genome_index(dir.path(), "g1", &seq);
|
||||||
g1.build_sibling_annex().expect("build_sibling_annex");
|
g1.build_sibling_annex().expect("build_sibling_annex");
|
||||||
|
|
||||||
let index_dir = g1.partition().index_dir(0);
|
let n_layers = g1.partition().n_layers(0).expect("partition meta");
|
||||||
let meta = PartitionMeta::load(&index_dir).expect("partition meta");
|
for l in 0..n_layers {
|
||||||
for l in 0..meta.n_layers {
|
|
||||||
let layer_dir = g1.partition().layer_dir(0, l);
|
let layer_dir = g1.partition().layer_dir(0, l);
|
||||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).expect("annex open");
|
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).expect("annex open");
|
||||||
for slot in 0..annex.len() {
|
for slot in 0..annex.len() {
|
||||||
@@ -361,9 +358,8 @@ fn sibling_annex_records_the_real_layer_of_each_family_member() {
|
|||||||
let merged = merge_two(dir.path(), &g1, &g2);
|
let merged = merge_two(dir.path(), &g1, &g2);
|
||||||
merged.build_sibling_annex().expect("build_sibling_annex");
|
merged.build_sibling_annex().expect("build_sibling_annex");
|
||||||
|
|
||||||
let index_dir = merged.partition().index_dir(0);
|
let n_layers = merged.partition().n_layers(0).unwrap();
|
||||||
let meta = PartitionMeta::load(&index_dir).unwrap();
|
assert_eq!(n_layers, 2, "fixture assumption: one merge, one new layer");
|
||||||
assert_eq!(meta.n_layers, 2, "fixture assumption: one merge, one new layer");
|
|
||||||
|
|
||||||
let g1_kmer = canonical(b"AACCGCTTAAG"); // own base C (1), sibling base G (2) — lives in layer 1
|
let g1_kmer = canonical(b"AACCGCTTAAG"); // own base C (1), sibling base G (2) — lives in layer 1
|
||||||
let g2_kmer = canonical(b"AACCGGTTAAG"); // own base G (2), sibling base C (1) — lives in layer 0
|
let g2_kmer = canonical(b"AACCGGTTAAG"); // own base G (2), sibling base C (1) — lives in layer 0
|
||||||
|
|||||||
@@ -85,6 +85,76 @@ impl LayerData for PersistentSparseBitMatrix {
|
|||||||
fn read(&self, slot: usize) -> Box<[bool]> { self.row(slot) }
|
fn read(&self, slot: usize) -> Box<[bool]> { self.row(slot) }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── LayerContent ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// What a layer's data matrix represents — orthogonal to *how* it's stored
|
||||||
|
/// on disk (`obicompactvec::StorageKind`'s concern). Only meaningful for
|
||||||
|
/// `D` that actually carry a matrix: `Layer<()>` (mode 1, set membership,
|
||||||
|
/// no matrix at all) has neither a `LayerContent` nor a `content()` method
|
||||||
|
/// — it's a write-time-only state, never a queryable content. Once a layer
|
||||||
|
/// is closed, "no matrix file" reads back as `Presence` via
|
||||||
|
/// `PersistentBitMatrix::open`'s own `Implicit` fallback, not as some third
|
||||||
|
/// "empty" content — see `DevDocMD/implementation/partition_layer_cache.md`.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum LayerContent {
|
||||||
|
Count,
|
||||||
|
Presence,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LayerContent {
|
||||||
|
/// Lightweight disk probe, no MPHF/matrix opened: `counts/` present →
|
||||||
|
/// `Count`, otherwise `Presence` — an absent/`Implicit` presence
|
||||||
|
/// matrix still reads as `Presence`, never a separate content.
|
||||||
|
pub fn detect(layer_dir: &Path) -> LayerContent {
|
||||||
|
if layer_dir.join(COUNTS_DIR).exists() {
|
||||||
|
LayerContent::Count
|
||||||
|
} else {
|
||||||
|
LayerContent::Presence
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Implemented only by `D` that carry a real matrix — gates
|
||||||
|
/// [`Layer::content`](Layer::content) so `Layer<()>` doesn't get it.
|
||||||
|
pub trait HasLayerContent {
|
||||||
|
const CONTENT: LayerContent;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HasLayerContent for PersistentCompactIntMatrix {
|
||||||
|
const CONTENT: LayerContent = LayerContent::Count;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HasLayerContent for PersistentBitMatrix {
|
||||||
|
const CONTENT: LayerContent = LayerContent::Presence;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HasLayerContent for PersistentSparseBitMatrix {
|
||||||
|
const CONTENT: LayerContent = LayerContent::Presence;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── StorageKind passthrough ────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Implemented only by `D` that expose their own `storage_kind()` — gates
|
||||||
|
/// [`Layer::storage_kind`]. `PersistentSparseBitMatrix` has a single,
|
||||||
|
/// fixed on-disk format (no `Columnar`/`Packed`/`Implicit` variants of its
|
||||||
|
/// own), so it's deliberately not given an impl: there's nothing to report
|
||||||
|
/// beyond what `content()` already says.
|
||||||
|
pub trait HasStorageKind {
|
||||||
|
fn storage_kind(&self) -> obicompactvec::StorageKind;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HasStorageKind for PersistentCompactIntMatrix {
|
||||||
|
fn storage_kind(&self) -> obicompactvec::StorageKind {
|
||||||
|
PersistentCompactIntMatrix::storage_kind(self)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl HasStorageKind for PersistentBitMatrix {
|
||||||
|
fn storage_kind(&self) -> obicompactvec::StorageKind {
|
||||||
|
PersistentBitMatrix::storage_kind(self)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ── Structures ────────────────────────────────────────────────────────────────
|
// ── Structures ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
pub struct Layer<D: LayerData = ()> {
|
pub struct Layer<D: LayerData = ()> {
|
||||||
@@ -171,6 +241,28 @@ impl<D: LayerData> Layer<D> {
|
|||||||
MphfLayer::build_approx_evidence(layer_dir, b, z)
|
MphfLayer::build_approx_evidence(layer_dir, b, z)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// This already-open layer's evidence mode — reads the discriminant
|
||||||
|
/// already in memory (`MphfLayer`'s own `LayerEvidence`), no disk
|
||||||
|
/// access. Available regardless of `D`, unlike `content`/`storage_kind`.
|
||||||
|
pub fn evidence_kind(&self) -> crate::mphf_layer::EvidenceKind {
|
||||||
|
self.mphf.evidence_kind()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<D: LayerData + HasLayerContent> Layer<D> {
|
||||||
|
/// This already-open layer's content (`Count`/`Presence`) — a
|
||||||
|
/// compile-time fact about `D`, not a runtime read.
|
||||||
|
pub const fn content(&self) -> LayerContent {
|
||||||
|
D::CONTENT
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<D: LayerData + HasStorageKind> Layer<D> {
|
||||||
|
/// This already-open layer's storage format — delegates to `D`'s own
|
||||||
|
/// `storage_kind()`, reading a discriminant already in memory.
|
||||||
|
pub fn storage_kind(&self) -> obicompactvec::StorageKind {
|
||||||
|
self.data.storage_kind()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ── Mode 1 — set membership ───────────────────────────────────────────────────
|
// ── Mode 1 — set membership ───────────────────────────────────────────────────
|
||||||
|
|||||||
@@ -8,8 +8,8 @@ pub mod meta;
|
|||||||
pub(crate) mod mphf_layer;
|
pub(crate) mod mphf_layer;
|
||||||
|
|
||||||
pub use error::{OLMError, OLMResult};
|
pub use error::{OLMError, OLMResult};
|
||||||
pub use layer::{layer_dir, open_data, Hit, Layer, LayerData};
|
pub use layer::{layer_dir, open_data, HasLayerContent, HasStorageKind, Hit, Layer, LayerContent, LayerData};
|
||||||
pub use layered_store::LayeredStore;
|
pub use layered_store::LayeredStore;
|
||||||
pub use map::LayeredMap;
|
pub use map::LayeredMap;
|
||||||
pub use meta::{IndexMode, PartitionMeta};
|
pub use meta::{IndexMode, PartitionMeta};
|
||||||
pub use mphf_layer::{KmerBatchIter, KmerIter, MphfLayer, MphfOnly};
|
pub use mphf_layer::{EvidenceKind, KmerBatchIter, KmerIter, MphfLayer, MphfOnly};
|
||||||
|
|||||||
@@ -2,13 +2,14 @@ use std::collections::HashMap;
|
|||||||
use std::fs;
|
use std::fs;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use obicompactvec::PersistentCompactIntMatrix;
|
use obicompactvec::{PersistentBitMatrix, PersistentCompactIntMatrix, StorageKind};
|
||||||
use obikseq::CanonicalKmer;
|
use obikseq::CanonicalKmer;
|
||||||
use obiskio::{UnitigFileWriter, DEFAULT_BLOCK_BITS};
|
use obiskio::{UnitigFileWriter, DEFAULT_BLOCK_BITS};
|
||||||
|
|
||||||
use crate::error::OLMResult;
|
use crate::error::{OLMError, OLMResult};
|
||||||
use crate::layer::{layer_dir, Hit, Layer, LayerData};
|
use crate::layer::{layer_dir, Hit, Layer, LayerContent, LayerData};
|
||||||
use crate::meta::{IndexMode, PartitionMeta};
|
use crate::meta::{IndexMode, PartitionMeta};
|
||||||
|
use crate::mphf_layer::EvidenceKind;
|
||||||
|
|
||||||
/// Layered kmer index for a single partition.
|
/// Layered kmer index for a single partition.
|
||||||
///
|
///
|
||||||
@@ -46,6 +47,30 @@ impl<D: LayerData> LayeredMap<D> {
|
|||||||
pub fn layer(&self, i: usize) -> &Layer<D> { &self.layers[i] }
|
pub fn layer(&self, i: usize) -> &Layer<D> { &self.layers[i] }
|
||||||
pub fn mode(&self) -> &IndexMode { &self.meta.mode }
|
pub fn mode(&self) -> &IndexMode { &self.meta.mode }
|
||||||
|
|
||||||
|
/// Layer `i`'s content (`Count`/`Presence`) — a lightweight disk probe,
|
||||||
|
/// no MPHF/matrix opened. For picking a `D` before committing to
|
||||||
|
/// [`Layer::open`], or for reporting/diagnostics over an already-open
|
||||||
|
/// `LayeredMap` without re-opening every layer under a different `D`.
|
||||||
|
pub fn detect_layer_content(&self, i: usize) -> LayerContent {
|
||||||
|
LayerContent::detect(&layer_dir(&self.root, i))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Layer `i`'s storage format — a lightweight disk probe, no
|
||||||
|
/// MPHF/matrix opened. See [`detect_layer_content`](Self::detect_layer_content).
|
||||||
|
pub fn detect_layer_storage(&self, i: usize) -> OLMResult<StorageKind> {
|
||||||
|
let dir = layer_dir(&self.root, i);
|
||||||
|
match LayerContent::detect(&dir) {
|
||||||
|
LayerContent::Count => PersistentCompactIntMatrix::detect_storage(&dir).map_err(OLMError::Io),
|
||||||
|
LayerContent::Presence => PersistentBitMatrix::detect_storage(&dir).map_err(OLMError::Io),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Layer `i`'s evidence mode — a lightweight disk probe, no
|
||||||
|
/// MPHF/matrix opened. See [`detect_layer_content`](Self::detect_layer_content).
|
||||||
|
pub fn detect_layer_evidence(&self, i: usize) -> OLMResult<EvidenceKind> {
|
||||||
|
EvidenceKind::detect(&layer_dir(&self.root, i))
|
||||||
|
}
|
||||||
|
|
||||||
/// Query `kmer` across all layers. Returns `(layer_index, Hit)` on match.
|
/// Query `kmer` across all layers. Returns `(layer_index, Hit)` on match.
|
||||||
pub fn query(&self, kmer: CanonicalKmer) -> Option<(usize, Hit<D::Item>)> {
|
pub fn query(&self, kmer: CanonicalKmer) -> Option<(usize, Hit<D::Item>)> {
|
||||||
self.layers
|
self.layers
|
||||||
|
|||||||
@@ -35,6 +35,35 @@ enum LayerEvidence {
|
|||||||
Hybrid { evidence: Evidence, unitigs: Arc<UnitigFileReader>, fingerprint: FingerprintVec },
|
Hybrid { evidence: Evidence, unitigs: Arc<UnitigFileReader>, fingerprint: FingerprintVec },
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── EvidenceKind ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Coarse evidence mode of a layer — the `IndexMode` a layer was opened
|
||||||
|
/// with, without the `Approx`/`Hybrid` `b`/`z` parameters.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum EvidenceKind {
|
||||||
|
Exact,
|
||||||
|
Approx,
|
||||||
|
Hybrid,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EvidenceKind {
|
||||||
|
/// Lightweight disk probe, no MPHF/evidence opened: presence of
|
||||||
|
/// `evidence.bin`/`fingerprint.bin` alone determines the mode — same
|
||||||
|
/// signal `MphfLayer::open` uses, just without opening either file.
|
||||||
|
pub fn detect(layer_dir: &Path) -> OLMResult<EvidenceKind> {
|
||||||
|
let has_evidence = layer_dir.join(EVIDENCE_FILE).exists();
|
||||||
|
let has_fingerprint = layer_dir.join(FINGERPRINT_FILE).exists();
|
||||||
|
match (has_evidence, has_fingerprint) {
|
||||||
|
(true, false) => Ok(EvidenceKind::Exact),
|
||||||
|
(false, true) => Ok(EvidenceKind::Approx),
|
||||||
|
(true, true) => Ok(EvidenceKind::Hybrid),
|
||||||
|
(false, false) => Err(OLMError::InvalidLayer(format!(
|
||||||
|
"no evidence.bin or fingerprint.bin in {}", layer_dir.display()
|
||||||
|
))),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ── MphfLayer ─────────────────────────────────────────────────────────────────
|
// ── MphfLayer ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
/// Autonomous kmer → slot mapping for one layer.
|
/// Autonomous kmer → slot mapping for one layer.
|
||||||
@@ -163,6 +192,16 @@ impl MphfLayer {
|
|||||||
|
|
||||||
pub fn n(&self) -> usize { self.n }
|
pub fn n(&self) -> usize { self.n }
|
||||||
|
|
||||||
|
/// This already-open layer's evidence mode — reads the discriminant
|
||||||
|
/// already in memory, no disk access.
|
||||||
|
pub fn evidence_kind(&self) -> EvidenceKind {
|
||||||
|
match &self.ev {
|
||||||
|
LayerEvidence::Exact { .. } => EvidenceKind::Exact,
|
||||||
|
LayerEvidence::Approx { .. } => EvidenceKind::Approx,
|
||||||
|
LayerEvidence::Hybrid { .. } => EvidenceKind::Hybrid,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Raw MPHF lookup: kmer → slot.
|
/// Raw MPHF lookup: kmer → slot.
|
||||||
///
|
///
|
||||||
/// Returns the slot assigned by the MPHF without performing any membership
|
/// Returns the slot assigned by the MPHF without performing any membership
|
||||||
|
|||||||
@@ -98,6 +98,36 @@ fn presence_layer_generic_over_sparse_matches_dense() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── content / storage_kind / evidence_kind (runtime introspection) ─────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn count_layer_reports_count_content_and_columnar_storage() {
|
||||||
|
set_k(4);
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
write_unitigs(dir.path(), &[b"AAAACGT"]);
|
||||||
|
Layer::<PersistentCompactIntMatrix>::build(dir.path(), DEFAULT_BLOCK_BITS, &IndexMode::Exact, |_| 1)
|
||||||
|
.unwrap();
|
||||||
|
let layer = Layer::<PersistentCompactIntMatrix>::open(dir.path(), &IndexMode::Exact).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(layer.content(), LayerContent::Count);
|
||||||
|
assert_eq!(layer.storage_kind(), obicompactvec::StorageKind::Columnar);
|
||||||
|
assert_eq!(layer.evidence_kind(), crate::mphf_layer::EvidenceKind::Exact);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn presence_layer_reports_presence_content_and_columnar_storage() {
|
||||||
|
set_k(4);
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
write_unitigs(dir.path(), &[b"AAATCTA", b"CTTCGCC", b"TGATACG"]);
|
||||||
|
Layer::<PersistentBitMatrix>::build_presence(dir.path(), DEFAULT_BLOCK_BITS, &IndexMode::Exact, 2, |kmer, g| {
|
||||||
|
(kmer.raw().wrapping_add(g as u64)) % 2 == 0
|
||||||
|
}).unwrap();
|
||||||
|
let layer = Layer::<PersistentBitMatrix>::open(dir.path(), &IndexMode::Exact).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(layer.content(), LayerContent::Presence);
|
||||||
|
assert_eq!(layer.storage_kind(), obicompactvec::StorageKind::Columnar);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn enumerate_kmers_is_stable_across_calls() {
|
fn enumerate_kmers_is_stable_across_calls() {
|
||||||
set_k(4);
|
set_k(4);
|
||||||
|
|||||||
@@ -1,7 +1,10 @@
|
|||||||
use super::*;
|
use super::*;
|
||||||
use obicompactvec::PersistentCompactIntMatrix;
|
use obicompactvec::{pack_bit_matrix, pack_sparse_bit_matrix, PersistentBitMatrix, PersistentCompactIntMatrix, StorageKind};
|
||||||
use obikseq::{set_k, Sequence as _, Unitig};
|
use obikseq::{set_k, Sequence as _, Unitig};
|
||||||
|
use obiskio::DEFAULT_BLOCK_BITS;
|
||||||
|
use crate::layer::LayerContent;
|
||||||
use crate::meta::IndexMode;
|
use crate::meta::IndexMode;
|
||||||
|
use crate::mphf_layer::EvidenceKind;
|
||||||
use tempfile::tempdir;
|
use tempfile::tempdir;
|
||||||
|
|
||||||
fn push_unitigs_and_layer(
|
fn push_unitigs_and_layer(
|
||||||
@@ -96,3 +99,121 @@ fn push_layer_from_map_convenience() {
|
|||||||
let (_, hit) = map.query(canonical(b"AAAA")).unwrap();
|
let (_, hit) = map.query(canonical(b"AAAA")).unwrap();
|
||||||
assert_eq!(hit.data[0], 10);
|
assert_eq!(hit.data[0], 10);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── detect_layer_content / detect_layer_storage / detect_layer_evidence ────
|
||||||
|
//
|
||||||
|
// All built directly on disk (bypassing `LayeredMap<D>::push_layer`, which
|
||||||
|
// only exists for modes 1 and 2 — mode 3/presence has no `push_layer`),
|
||||||
|
// then reopened as `LayeredMap<()>` — `()`'s `LayerData::open` never reads
|
||||||
|
// the matrix, so it's the right handle for exercising the disk-probing
|
||||||
|
// `detect_layer_*` methods regardless of what content/storage the layer
|
||||||
|
// underneath actually is.
|
||||||
|
|
||||||
|
fn write_unitigs_at(dir: &Path, seqs: &[&[u8]]) {
|
||||||
|
fs::create_dir_all(dir).unwrap();
|
||||||
|
let mut w = UnitigFileWriter::create(&dir.join(crate::layer::UNITIGS_FILE)).unwrap();
|
||||||
|
for s in seqs {
|
||||||
|
w.write(&Unitig::from_ascii(s)).unwrap();
|
||||||
|
}
|
||||||
|
w.close().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build a one-layer partition root with a presence layer at `layer_0`,
|
||||||
|
/// save `meta.json`, and reopen it as `LayeredMap<()>`.
|
||||||
|
fn presence_partition(seqs: &[&[u8]], n_genomes: usize) -> (tempfile::TempDir, LayeredMap<()>) {
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let mode = IndexMode::Exact;
|
||||||
|
let layer0 = layer_dir(dir.path(), 0);
|
||||||
|
write_unitigs_at(&layer0, seqs);
|
||||||
|
Layer::<PersistentBitMatrix>::build_presence(&layer0, DEFAULT_BLOCK_BITS, &mode, n_genomes, |kmer, g| {
|
||||||
|
(kmer.raw().wrapping_add(g as u64)) % 2 == 0
|
||||||
|
}).unwrap();
|
||||||
|
PartitionMeta { n_layers: 1, mode }.save(dir.path()).unwrap();
|
||||||
|
let map = LayeredMap::<()>::open(dir.path()).unwrap();
|
||||||
|
(dir, map)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn detect_layer_content_count() {
|
||||||
|
set_k(4);
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let mut map = LayeredMap::<PersistentCompactIntMatrix>::create(dir.path(), IndexMode::Exact).unwrap();
|
||||||
|
push_unitigs_and_layer(&mut map, &[b"AAAACGT"], 3);
|
||||||
|
assert_eq!(map.detect_layer_content(0), LayerContent::Count);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn detect_layer_content_presence() {
|
||||||
|
set_k(4);
|
||||||
|
let (_dir, map) = presence_partition(&[b"AAATCTA", b"CTTCGCC", b"TGATACG"], 3);
|
||||||
|
assert_eq!(map.detect_layer_content(0), LayerContent::Presence);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn detect_layer_storage_columnar_then_packed_then_sparse() {
|
||||||
|
set_k(4);
|
||||||
|
let (dir, map) = presence_partition(&[b"AAATCTA", b"CTTCGCC", b"TGATACG"], 3);
|
||||||
|
let presence_dir = layer_dir(dir.path(), 0).join("presence");
|
||||||
|
|
||||||
|
assert_eq!(map.detect_layer_storage(0).unwrap(), StorageKind::Columnar);
|
||||||
|
|
||||||
|
pack_bit_matrix(&presence_dir).unwrap();
|
||||||
|
assert_eq!(map.detect_layer_storage(0).unwrap(), StorageKind::Packed);
|
||||||
|
|
||||||
|
pack_sparse_bit_matrix(&presence_dir).unwrap();
|
||||||
|
assert_eq!(map.detect_layer_storage(0).unwrap(), StorageKind::Sparse);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn detect_layer_storage_implicit() {
|
||||||
|
set_k(4);
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let mode = IndexMode::Exact;
|
||||||
|
let layer0 = layer_dir(dir.path(), 0);
|
||||||
|
write_unitigs_at(&layer0, &[b"AAAACGT"]);
|
||||||
|
// Mode-1 build: MPHF + evidence + `layer_meta.json`, no matrix at all —
|
||||||
|
// must read back as `Presence`/`Implicit`, never a third "empty" state.
|
||||||
|
Layer::<()>::build(&layer0, DEFAULT_BLOCK_BITS, &mode).unwrap();
|
||||||
|
PartitionMeta { n_layers: 1, mode }.save(dir.path()).unwrap();
|
||||||
|
let map = LayeredMap::<()>::open(dir.path()).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(map.detect_layer_content(0), LayerContent::Presence);
|
||||||
|
assert_eq!(map.detect_layer_storage(0).unwrap(), StorageKind::Implicit);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn detect_layer_storage_count_columnar_then_packed() {
|
||||||
|
set_k(4);
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let mut map = LayeredMap::<PersistentCompactIntMatrix>::create(dir.path(), IndexMode::Exact).unwrap();
|
||||||
|
push_unitigs_and_layer(&mut map, &[b"AAAACGT"], 3);
|
||||||
|
drop(map);
|
||||||
|
|
||||||
|
let map = LayeredMap::<()>::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(map.detect_layer_storage(0).unwrap(), StorageKind::Columnar);
|
||||||
|
|
||||||
|
obicompactvec::pack_compact_int_matrix(&layer_dir(dir.path(), 0).join("counts")).unwrap();
|
||||||
|
assert_eq!(map.detect_layer_storage(0).unwrap(), StorageKind::Packed);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn detect_layer_evidence_exact_vs_approx() {
|
||||||
|
set_k(4);
|
||||||
|
|
||||||
|
let dir = tempdir().unwrap();
|
||||||
|
let layer0 = layer_dir(dir.path(), 0);
|
||||||
|
write_unitigs_at(&layer0, &[b"AAAACGT"]);
|
||||||
|
Layer::<()>::build(&layer0, DEFAULT_BLOCK_BITS, &IndexMode::Exact).unwrap();
|
||||||
|
PartitionMeta { n_layers: 1, mode: IndexMode::Exact }.save(dir.path()).unwrap();
|
||||||
|
let map = LayeredMap::<()>::open(dir.path()).unwrap();
|
||||||
|
assert_eq!(map.detect_layer_evidence(0).unwrap(), EvidenceKind::Exact);
|
||||||
|
|
||||||
|
let approx = IndexMode::Approx { b: 8, z: 1 };
|
||||||
|
let dir2 = tempdir().unwrap();
|
||||||
|
let layer0b = layer_dir(dir2.path(), 0);
|
||||||
|
write_unitigs_at(&layer0b, &[b"AAAACGT"]);
|
||||||
|
Layer::<()>::build(&layer0b, DEFAULT_BLOCK_BITS, &approx).unwrap();
|
||||||
|
PartitionMeta { n_layers: 1, mode: approx }.save(dir2.path()).unwrap();
|
||||||
|
let map2 = LayeredMap::<()>::open(dir2.path()).unwrap();
|
||||||
|
assert_eq!(map2.detect_layer_evidence(0).unwrap(), EvidenceKind::Approx);
|
||||||
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user