feat: add convert command for in-place index evidence modification
Introduce a new CLI command that modifies existing indices to support exact, approximate, or hybrid evidence modes. This change enables the reindex module in obikrebuild, exposing an IndexReindex trait for evidence persistence and layer processing. It also refactors obikindexer to publicly expose an IndexBuilder trait for post-build evidence construction, while adding a set_evidence API in obikindex to safely update configuration fields without altering core parameters.
This commit is contained in:
1 parent
bd7729b095
commit
4ea3cd32ba
10 files changed
+562
-340
No files matched your search
@@ -0,0 +1,68 @@
|
||||
//! `IndexBuilder` — public, idempotent evidence-construction primitives
|
||||
//! for an already-built `KmerLayer` (`unitigs.bin` + `mphf.bin` already on
|
||||
//! disk; only the evidence bundle — `evidence.bin`/`unitigs.bin.idx` and/or
|
||||
//! `fingerprint.bin` — is touched). Unlike `PrivateBuilder` (this module's
|
||||
//! private sibling), these are meant for callers outside this crate that
|
||||
//! need to add or convert a layer's evidence after the fact — today,
|
||||
//! `obikrebuild::reindex`.
|
||||
//!
|
||||
//! `KmerLayer` itself (`obikindex`) stays a pure data-model type: it
|
||||
//! exposes paths and reads content, it never builds anything. Construction
|
||||
//! lives here instead, in the crate whose whole purpose is building index
|
||||
//! content — same reasoning as `PrivateBuilder`, just public.
|
||||
|
||||
use obikindex::OKIResult;
|
||||
use obikindex::layer::fingerprint::FingerprintVec;
|
||||
use obikindex::layer::{KmerLayer, MphfLayer};
|
||||
|
||||
pub trait IndexBuilder {
|
||||
/// Build this layer's exact evidence bundle (`evidence.bin` +
|
||||
/// `unitigs.bin.idx`) if not already present.
|
||||
///
|
||||
/// No-op (`Ok(0)`) if exact evidence already exists — both `Exact` and
|
||||
/// `Hybrid` layers already carry it.
|
||||
fn build_exact_evidence(&self, block_bits: u8) -> OKIResult<usize>;
|
||||
|
||||
/// Build this layer's approximate (fingerprint) evidence with `b` bits
|
||||
/// per slot and Findere parameter `z`, if not already present with a
|
||||
/// matching `b`.
|
||||
///
|
||||
/// No-op (`Ok(0)`) if `Approx`/`Hybrid` evidence with the same `b`
|
||||
/// already exists — rebuilds it (same file, same `b`, whatever `z`)
|
||||
/// otherwise. `z` is never persisted (see `IndexMode::Approx`'s own
|
||||
/// doc): it only governs how many consecutive stored k-mers a query
|
||||
/// must match, so changing it alone is not something this method has
|
||||
/// any file-level evidence to detect — the caller decides.
|
||||
fn build_approx_evidence(&self, b: u8, z: u8) -> OKIResult<usize>;
|
||||
|
||||
/// Build whichever of the exact/approximate bundles this layer is
|
||||
/// still missing, so it ends up carrying both (`Hybrid`).
|
||||
///
|
||||
/// Just the composition of `build_exact_evidence` and
|
||||
/// `build_approx_evidence` — each is already independently idempotent,
|
||||
/// so there is no extra "what's already there" decision to make here.
|
||||
fn build_hybrid_evidence(&self, b: u8, z: u8, block_bits: u8) -> OKIResult<usize>;
|
||||
}
|
||||
|
||||
impl IndexBuilder for KmerLayer {
|
||||
fn build_exact_evidence(&self, block_bits: u8) -> OKIResult<usize> {
|
||||
if self.evidence_path().exists() {
|
||||
return Ok(0);
|
||||
}
|
||||
MphfLayer::build_exact_evidence(self.dir(), block_bits)
|
||||
}
|
||||
|
||||
fn build_approx_evidence(&self, b: u8, z: u8) -> OKIResult<usize> {
|
||||
let fingerprint_path = self.fingerprint_path();
|
||||
if fingerprint_path.exists() && FingerprintVec::open(&fingerprint_path)?.b() == b {
|
||||
return Ok(0);
|
||||
}
|
||||
MphfLayer::build_approx_evidence(self.dir(), b, z)
|
||||
}
|
||||
|
||||
fn build_hybrid_evidence(&self, b: u8, z: u8, block_bits: u8) -> OKIResult<usize> {
|
||||
let n_exact = self.build_exact_evidence(block_bits)?;
|
||||
let n_approx = self.build_approx_evidence(b, z)?;
|
||||
Ok(n_exact.max(n_approx))
|
||||
}
|
||||
}
|
||||
@@ -1,245 +1,20 @@
|
||||
//! Construction-only `KmerIndex` operations — semantically meaningless
|
||||
//! once an index is built and being used for reading (query, dump, phylo,
|
||||
//! or as a `merge`/`select`/`rebuild` *source*): sentinel marking, the
|
||||
//! per-partition layer-build primitive, and cleanup of build-time scratch
|
||||
//! files. Private to this crate — no consumer outside the
|
||||
//! indexing-pipeline algorithms in `obikindexer::algorithms` should ever
|
||||
//! need them. See `DevDocMD/implementation/partition_layer_cache.md`,
|
||||
//! "(8)".
|
||||
//! Extension traits over `obikindex`'s foreign types (`KmerIndex`,
|
||||
//! `KmerLayer`) — Rust's orphan rule requires such traits to be defined in
|
||||
//! the crate that implements them for a foreign type, not in `obikindex`
|
||||
//! itself; `obikindex` has no way to make a trait "visible to only some
|
||||
//! crates" from its own side.
|
||||
//!
|
||||
//! Defined here, not in `obikindex`, so it can be genuinely `pub(crate)`
|
||||
//! while extending a foreign type (`KmerIndex`) — Rust's orphan rule
|
||||
//! requires the trait itself to be local to the crate that implements it
|
||||
//! for a foreign type; `obikindex` has no way to make a trait "private
|
||||
//! except to `obikindexer`" from its own side.
|
||||
//!
|
||||
//! Scoped to the six methods exclusive to this crate's own pipeline
|
||||
//! (`partitionner`/`dereplicator`/`counter`/`layer_builder`).
|
||||
//! `clear_output_for_create`/`create_skeleton`/`finalize_indexed`/`state`
|
||||
//! stay inherent on `KmerIndex`, deliberately not moved here: they're also
|
||||
//! called from `obikindex`'s own `merge`/`select`/`rebuild`/`reindex`
|
||||
//! modules, which — living inside `obikindex` itself — could never reach a
|
||||
//! trait defined in `obikindexer` (the dependency only runs
|
||||
//! `obikindexer → obikindex`, never the other way). Moving those four
|
||||
//! would require extracting `merge`/`select`/`rebuild`/`reindex` into
|
||||
//! algorithms first — separate, larger future work, not this one.
|
||||
//! Two traits, two visibilities, one per submodule:
|
||||
//! - [`private_builder::PrivateBuilder`] (`pub(crate)`) — construction-only
|
||||
//! `KmerIndex` operations exclusive to this crate's own indexing
|
||||
//! pipeline (`algorithms::{partitionner,dereplicator,counter,layer_builder}`).
|
||||
//! - [`index_builder::IndexBuilder`] (`pub`, re-exported at the crate
|
||||
//! root) — idempotent per-layer evidence-construction primitives, for
|
||||
//! any external caller that needs to add/convert a layer's evidence
|
||||
//! bundle after the fact (`obikrebuild::reindex`).
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::io;
|
||||
use std::path::Path;
|
||||
mod index_builder;
|
||||
mod private_builder;
|
||||
|
||||
use cacheline_ef::{CachelineEf, CachelineEfVec};
|
||||
use epserde::prelude::*;
|
||||
use obicompactvec::{PersistentCompactIntMatrix, PersistentCompactIntVec};
|
||||
use obidebruinj::GraphDeBruijn;
|
||||
use obikindex::layer::IndexMode;
|
||||
use obikindex::layer::{KmerLayer, TypedLayer};
|
||||
use obikindex::{KmerIndex, OKIError, OKIResult};
|
||||
use crate::graph_pipeline::{materialize_layer, write_graph_as_unitigs};
|
||||
use obiskio::{SKError, SKFileMeta, SKFileReader};
|
||||
use ptr_hash::{PtrHash, bucket_fn::CubicEps, hash::Xx64};
|
||||
|
||||
type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xx64, Vec<u8>>;
|
||||
|
||||
pub(crate) trait PrivateBuilder {
|
||||
/// Mark scatter as complete (`IndexState::Scattered`).
|
||||
///
|
||||
/// If no genome label was set at creation time, one is derived from
|
||||
/// the index root directory name (stripped of all extensions). `&self`
|
||||
/// — not `&mut self` anymore: nothing is cached in memory to mutate,
|
||||
/// see `DevDocMD/implementation/partition_layer_cache.md`, "(11)".
|
||||
fn mark_scattered(&self) -> OKIResult<()>;
|
||||
|
||||
/// Mark dereplicate+count as complete (`IndexState::Counted`).
|
||||
fn mark_counted(&self) -> OKIResult<()>;
|
||||
|
||||
/// Partition `i`'s layer 0 — the one and only layer every pipeline
|
||||
/// algorithm in this crate (`partitionner`/`dereplicator`/`counter`/
|
||||
/// `layer_builder`) reads from or writes to; later layers only exist
|
||||
/// after a `merge`, which lives in `obikindex`, not here.
|
||||
fn layer0(&self, i: usize) -> OKIResult<KmerLayer>;
|
||||
|
||||
/// Mark layer construction as complete (`IndexState::Indexed`).
|
||||
fn mark_indexed(&self) -> OKIResult<()>;
|
||||
|
||||
/// Write `spectrums/{label}.json` from an already-computed kmer
|
||||
/// spectrum (`f0`/`f1`/abundance histogram). Takes plain values rather
|
||||
/// than `algorithms::counter::KmerSpectrum` — `KmerIndex` is the data
|
||||
/// model, `Counter` the algorithm that depends on it, not the other
|
||||
/// way around, and this is the only field of that type it actually
|
||||
/// uses.
|
||||
fn write_spectrum(&self, f0: u64, f1: u64, counts: &BTreeMap<u32, u64>) -> OKIResult<()>;
|
||||
|
||||
/// Build the layered MPHF index for partition `i`.
|
||||
///
|
||||
/// Returns the number of canonical k-mers indexed, or 0 if the
|
||||
/// partition has no data or its layer was already built
|
||||
/// (resume-safe).
|
||||
///
|
||||
/// Abundance filtering is applied when `min_ab > 1` or
|
||||
/// `max_ab.is_some()`, using `mphf1.bin` + `counts1.bin` if they
|
||||
/// exist. Count payload is stored iff `with_counts` is true.
|
||||
fn build_index_layer(
|
||||
&self,
|
||||
i: usize,
|
||||
min_ab: u32,
|
||||
max_ab: Option<u32>,
|
||||
with_counts: bool,
|
||||
mode: &IndexMode,
|
||||
block_bits: u8,
|
||||
) -> Result<usize, SKError>;
|
||||
|
||||
/// Remove intermediate build artifacts for partition `i`: dereplicated
|
||||
/// superkmers (+ sidecar), `mphf1.bin`, `counts1.bin`.
|
||||
fn remove_build_artifacts(&self, i: usize);
|
||||
}
|
||||
|
||||
impl PrivateBuilder for KmerIndex {
|
||||
fn mark_scattered(&self) -> OKIResult<()> {
|
||||
self.meta().mark_scattered().map_err(OKIError::Io)
|
||||
}
|
||||
|
||||
fn mark_counted(&self) -> OKIResult<()> {
|
||||
self.meta().mark_counted().map_err(OKIError::Io)
|
||||
}
|
||||
|
||||
fn layer0(&self, i: usize) -> OKIResult<KmerLayer> {
|
||||
self.partition(i)?.layer(0)
|
||||
}
|
||||
|
||||
fn mark_indexed(&self) -> OKIResult<()> {
|
||||
self.meta().mark_indexed().map_err(OKIError::Io)
|
||||
}
|
||||
|
||||
fn write_spectrum(&self, f0: u64, f1: u64, counts: &BTreeMap<u32, u64>) -> OKIResult<()> {
|
||||
let genomes = self.meta().genomes().map_err(OKIError::Io)?;
|
||||
let label = genomes
|
||||
.first()
|
||||
.map(|g| g.label.as_str())
|
||||
.unwrap_or("unknown")
|
||||
.to_owned();
|
||||
let spectrums_dir = self.dir().join("spectrums");
|
||||
fs::create_dir_all(&spectrums_dir)?;
|
||||
let path = spectrums_dir.join(format!("{label}.json"));
|
||||
let spectrum_map: BTreeMap<String, u64> = counts
|
||||
.iter()
|
||||
.map(|(&c, &f)| (format!("{c:010}"), f))
|
||||
.collect();
|
||||
let f = fs::File::create(&path)?;
|
||||
serde_json::to_writer_pretty(
|
||||
f,
|
||||
&serde_json::json!({ "f0": f0, "f1": f1, "spectrum": spectrum_map }),
|
||||
)
|
||||
.map_err(OKIError::Json)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn build_index_layer(
|
||||
&self,
|
||||
i: usize,
|
||||
min_ab: u32,
|
||||
max_ab: Option<u32>,
|
||||
with_counts: bool,
|
||||
mode: &IndexMode,
|
||||
block_bits: u8,
|
||||
) -> Result<usize, SKError> {
|
||||
let layer0 = self.layer0(i).map_err(|e| io::Error::other(e.to_string()))?;
|
||||
let layer0_dir = layer0.dir();
|
||||
let dedup_path = layer0.dereplicated_superkmers_path();
|
||||
if !dedup_path.exists() {
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
if layer0_dir.join("mphf.bin").exists() {
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
let filter_active = min_ab > 1 || max_ab.is_some();
|
||||
let need_counts = filter_active || with_counts;
|
||||
|
||||
let mphf1_opt: Option<Mphf> = if need_counts {
|
||||
let p = layer0_dir.join("mphf1.bin");
|
||||
p.exists().then(|| Mphf::load_full(&p).ok()).flatten()
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let counts1_opt: Option<PersistentCompactIntVec> = if need_counts {
|
||||
let p = layer0_dir.join("counts1.bin");
|
||||
p.exists()
|
||||
.then(|| PersistentCompactIntVec::open(&p).ok())
|
||||
.flatten()
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let mut g = GraphDeBruijn::new();
|
||||
let mut reader = SKFileReader::open(&dedup_path)?;
|
||||
for sk in reader.iter() {
|
||||
for kmer in sk.iter_canonical_kmers() {
|
||||
let accept = if filter_active {
|
||||
match (&mphf1_opt, &counts1_opt) {
|
||||
(Some(mphf), Some(counts)) => {
|
||||
let ab = counts.get(mphf.index(&kmer.raw()));
|
||||
ab >= min_ab && max_ab.map_or(true, |max| ab <= max)
|
||||
}
|
||||
_ => true,
|
||||
}
|
||||
} else {
|
||||
true
|
||||
};
|
||||
if accept {
|
||||
g.push(kmer);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let n_kmers = if with_counts {
|
||||
let n = write_graph_as_unitigs(g, layer0_dir)
|
||||
.map_err(|e| io::Error::other(e.to_string()))?;
|
||||
TypedLayer::<PersistentCompactIntMatrix>::build(
|
||||
layer0_dir,
|
||||
block_bits,
|
||||
mode,
|
||||
|kmer| match (&mphf1_opt, &counts1_opt) {
|
||||
(Some(mphf), Some(counts)) => counts.get(mphf.index(&kmer.raw())),
|
||||
_ => 1,
|
||||
},
|
||||
)
|
||||
.map_err(|e| io::Error::other(e.to_string()))?;
|
||||
n
|
||||
} else {
|
||||
materialize_layer(g, layer0_dir, block_bits, mode)
|
||||
.map_err(|e| io::Error::other(e.to_string()))?
|
||||
};
|
||||
|
||||
Ok(n_kmers)
|
||||
}
|
||||
|
||||
fn remove_build_artifacts(&self, i: usize) {
|
||||
let layer0 = match self.layer0(i) {
|
||||
Ok(layer) => layer,
|
||||
Err(e) => {
|
||||
eprintln!("warning: could not locate layer 0 of partition {i}: {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let layer0_dir = layer0.dir();
|
||||
let dedup = layer0.dereplicated_superkmers_path();
|
||||
remove_if_exists(&SKFileMeta::sidecar_path(&dedup));
|
||||
remove_if_exists(&dedup);
|
||||
remove_if_exists(&layer0_dir.join("mphf1.bin"));
|
||||
remove_if_exists(&layer0_dir.join("counts1.bin"));
|
||||
}
|
||||
}
|
||||
|
||||
// ── private helpers ──────────────────────────────────────────────────────
|
||||
|
||||
fn remove_if_exists(path: &Path) {
|
||||
if let Err(e) = fs::remove_file(path) {
|
||||
if e.kind() != io::ErrorKind::NotFound {
|
||||
eprintln!("warning: could not remove {}: {e}", path.display());
|
||||
}
|
||||
}
|
||||
}
|
||||
pub use index_builder::IndexBuilder;
|
||||
pub(crate) use private_builder::PrivateBuilder;
|
||||
@@ -0,0 +1,240 @@
|
||||
//! `PrivateBuilder` — construction-only `KmerIndex` operations:
|
||||
//! sentinel marking, the per-partition layer-build primitive, and cleanup
|
||||
//! of build-time scratch files. Semantically meaningless once an index is
|
||||
//! built and being used for reading (query, dump, phylo, or as a
|
||||
//! `merge`/`select`/`rebuild` *source*).
|
||||
//!
|
||||
//! Scoped to the six methods exclusive to this crate's own pipeline
|
||||
//! (`partitionner`/`dereplicator`/`counter`/`layer_builder`) — no
|
||||
//! consumer outside those algorithms should ever need them, hence
|
||||
//! `pub(crate)` (see the module-level orphan-rule note in `extensions`'s
|
||||
//! `mod.rs`). See `DevDocMD/implementation/partition_layer_cache.md`, "(8)".
|
||||
//!
|
||||
//! `clear_output_for_create`/`create_skeleton`/`finalize_indexed`/`state`
|
||||
//! stay inherent on `KmerIndex`, deliberately not moved here: they're also
|
||||
//! called from `obikindex`'s own `merge`/`select`/`rebuild`/`reindex`
|
||||
//! modules, which — living inside `obikindex` itself — could never reach a
|
||||
//! trait defined in `obikindexer` (the dependency only runs
|
||||
//! `obikindexer → obikindex`, never the other way). Moving those four
|
||||
//! would require extracting `merge`/`select`/`rebuild`/`reindex` into
|
||||
//! algorithms first — separate, larger future work, not this one.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::fs;
|
||||
use std::io;
|
||||
use std::path::Path;
|
||||
|
||||
use cacheline_ef::{CachelineEf, CachelineEfVec};
|
||||
use epserde::prelude::*;
|
||||
use obicompactvec::{PersistentCompactIntMatrix, PersistentCompactIntVec};
|
||||
use obidebruinj::GraphDeBruijn;
|
||||
use obikindex::layer::IndexMode;
|
||||
use obikindex::layer::{KmerLayer, TypedLayer};
|
||||
use obikindex::{KmerIndex, OKIError, OKIResult};
|
||||
use crate::graph_pipeline::{materialize_layer, write_graph_as_unitigs};
|
||||
use obiskio::{SKError, SKFileMeta, SKFileReader};
|
||||
use ptr_hash::{PtrHash, bucket_fn::CubicEps, hash::Xx64};
|
||||
|
||||
type Mphf = PtrHash<u64, CubicEps, CachelineEfVec<Vec<CachelineEf>>, Xx64, Vec<u8>>;
|
||||
|
||||
pub(crate) trait PrivateBuilder {
|
||||
/// Mark scatter as complete (`IndexState::Scattered`).
|
||||
///
|
||||
/// If no genome label was set at creation time, one is derived from
|
||||
/// the index root directory name (stripped of all extensions). `&self`
|
||||
/// — not `&mut self` anymore: nothing is cached in memory to mutate,
|
||||
/// see `DevDocMD/implementation/partition_layer_cache.md`, "(11)".
|
||||
fn mark_scattered(&self) -> OKIResult<()>;
|
||||
|
||||
/// Mark dereplicate+count as complete (`IndexState::Counted`).
|
||||
fn mark_counted(&self) -> OKIResult<()>;
|
||||
|
||||
/// Partition `i`'s layer 0 — the one and only layer every pipeline
|
||||
/// algorithm in this crate (`partitionner`/`dereplicator`/`counter`/
|
||||
/// `layer_builder`) reads from or writes to; later layers only exist
|
||||
/// after a `merge`, which lives in `obikindex`, not here.
|
||||
fn layer0(&self, i: usize) -> OKIResult<KmerLayer>;
|
||||
|
||||
/// Mark layer construction as complete (`IndexState::Indexed`).
|
||||
fn mark_indexed(&self) -> OKIResult<()>;
|
||||
|
||||
/// Write `spectrums/{label}.json` from an already-computed kmer
|
||||
/// spectrum (`f0`/`f1`/abundance histogram). Takes plain values rather
|
||||
/// than `algorithms::counter::KmerSpectrum` — `KmerIndex` is the data
|
||||
/// model, `Counter` the algorithm that depends on it, not the other
|
||||
/// way around, and this is the only field of that type it actually
|
||||
/// uses.
|
||||
fn write_spectrum(&self, f0: u64, f1: u64, counts: &BTreeMap<u32, u64>) -> OKIResult<()>;
|
||||
|
||||
/// Build the layered MPHF index for partition `i`.
|
||||
///
|
||||
/// Returns the number of canonical k-mers indexed, or 0 if the
|
||||
/// partition has no data or its layer was already built
|
||||
/// (resume-safe).
|
||||
///
|
||||
/// Abundance filtering is applied when `min_ab > 1` or
|
||||
/// `max_ab.is_some()`, using `mphf1.bin` + `counts1.bin` if they
|
||||
/// exist. Count payload is stored iff `with_counts` is true.
|
||||
fn build_index_layer(
|
||||
&self,
|
||||
i: usize,
|
||||
min_ab: u32,
|
||||
max_ab: Option<u32>,
|
||||
with_counts: bool,
|
||||
mode: &IndexMode,
|
||||
block_bits: u8,
|
||||
) -> Result<usize, SKError>;
|
||||
|
||||
/// Remove intermediate build artifacts for partition `i`: dereplicated
|
||||
/// superkmers (+ sidecar), `mphf1.bin`, `counts1.bin`.
|
||||
fn remove_build_artifacts(&self, i: usize);
|
||||
}
|
||||
|
||||
impl PrivateBuilder for KmerIndex {
|
||||
fn mark_scattered(&self) -> OKIResult<()> {
|
||||
self.meta().mark_scattered().map_err(OKIError::Io)
|
||||
}
|
||||
|
||||
fn mark_counted(&self) -> OKIResult<()> {
|
||||
self.meta().mark_counted().map_err(OKIError::Io)
|
||||
}
|
||||
|
||||
fn layer0(&self, i: usize) -> OKIResult<KmerLayer> {
|
||||
self.partition(i)?.layer(0)
|
||||
}
|
||||
|
||||
fn mark_indexed(&self) -> OKIResult<()> {
|
||||
self.meta().mark_indexed().map_err(OKIError::Io)
|
||||
}
|
||||
|
||||
fn write_spectrum(&self, f0: u64, f1: u64, counts: &BTreeMap<u32, u64>) -> OKIResult<()> {
|
||||
let genomes = self.meta().genomes().map_err(OKIError::Io)?;
|
||||
let label = genomes
|
||||
.first()
|
||||
.map(|g| g.label.as_str())
|
||||
.unwrap_or("unknown")
|
||||
.to_owned();
|
||||
let spectrums_dir = self.dir().join("spectrums");
|
||||
fs::create_dir_all(&spectrums_dir)?;
|
||||
let path = spectrums_dir.join(format!("{label}.json"));
|
||||
let spectrum_map: BTreeMap<String, u64> = counts
|
||||
.iter()
|
||||
.map(|(&c, &f)| (format!("{c:010}"), f))
|
||||
.collect();
|
||||
let f = fs::File::create(&path)?;
|
||||
serde_json::to_writer_pretty(
|
||||
f,
|
||||
&serde_json::json!({ "f0": f0, "f1": f1, "spectrum": spectrum_map }),
|
||||
)
|
||||
.map_err(OKIError::Json)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn build_index_layer(
|
||||
&self,
|
||||
i: usize,
|
||||
min_ab: u32,
|
||||
max_ab: Option<u32>,
|
||||
with_counts: bool,
|
||||
mode: &IndexMode,
|
||||
block_bits: u8,
|
||||
) -> Result<usize, SKError> {
|
||||
let layer0 = self.layer0(i).map_err(|e| io::Error::other(e.to_string()))?;
|
||||
let layer0_dir = layer0.dir();
|
||||
let dedup_path = layer0.dereplicated_superkmers_path();
|
||||
if !dedup_path.exists() {
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
if layer0_dir.join("mphf.bin").exists() {
|
||||
return Ok(0);
|
||||
}
|
||||
|
||||
let filter_active = min_ab > 1 || max_ab.is_some();
|
||||
let need_counts = filter_active || with_counts;
|
||||
|
||||
let mphf1_opt: Option<Mphf> = if need_counts {
|
||||
let p = layer0_dir.join("mphf1.bin");
|
||||
p.exists().then(|| Mphf::load_full(&p).ok()).flatten()
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let counts1_opt: Option<PersistentCompactIntVec> = if need_counts {
|
||||
let p = layer0_dir.join("counts1.bin");
|
||||
p.exists()
|
||||
.then(|| PersistentCompactIntVec::open(&p).ok())
|
||||
.flatten()
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
let mut g = GraphDeBruijn::new();
|
||||
let mut reader = SKFileReader::open(&dedup_path)?;
|
||||
for sk in reader.iter() {
|
||||
for kmer in sk.iter_canonical_kmers() {
|
||||
let accept = if filter_active {
|
||||
match (&mphf1_opt, &counts1_opt) {
|
||||
(Some(mphf), Some(counts)) => {
|
||||
let ab = counts.get(mphf.index(&kmer.raw()));
|
||||
ab >= min_ab && max_ab.map_or(true, |max| ab <= max)
|
||||
}
|
||||
_ => true,
|
||||
}
|
||||
} else {
|
||||
true
|
||||
};
|
||||
if accept {
|
||||
g.push(kmer);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let n_kmers = if with_counts {
|
||||
let n = write_graph_as_unitigs(g, layer0_dir)
|
||||
.map_err(|e| io::Error::other(e.to_string()))?;
|
||||
TypedLayer::<PersistentCompactIntMatrix>::build(
|
||||
layer0_dir,
|
||||
block_bits,
|
||||
mode,
|
||||
|kmer| match (&mphf1_opt, &counts1_opt) {
|
||||
(Some(mphf), Some(counts)) => counts.get(mphf.index(&kmer.raw())),
|
||||
_ => 1,
|
||||
},
|
||||
)
|
||||
.map_err(|e| io::Error::other(e.to_string()))?;
|
||||
n
|
||||
} else {
|
||||
materialize_layer(g, layer0_dir, block_bits, mode)
|
||||
.map_err(|e| io::Error::other(e.to_string()))?
|
||||
};
|
||||
|
||||
Ok(n_kmers)
|
||||
}
|
||||
|
||||
fn remove_build_artifacts(&self, i: usize) {
|
||||
let layer0 = match self.layer0(i) {
|
||||
Ok(layer) => layer,
|
||||
Err(e) => {
|
||||
eprintln!("warning: could not locate layer 0 of partition {i}: {e}");
|
||||
return;
|
||||
}
|
||||
};
|
||||
let layer0_dir = layer0.dir();
|
||||
let dedup = layer0.dereplicated_superkmers_path();
|
||||
remove_if_exists(&SKFileMeta::sidecar_path(&dedup));
|
||||
remove_if_exists(&dedup);
|
||||
remove_if_exists(&layer0_dir.join("mphf1.bin"));
|
||||
remove_if_exists(&layer0_dir.join("counts1.bin"));
|
||||
}
|
||||
}
|
||||
|
||||
// ── private helpers ──────────────────────────────────────────────────────
|
||||
|
||||
fn remove_if_exists(path: &Path) {
|
||||
if let Err(e) = fs::remove_file(path) {
|
||||
if e.kind() != io::ErrorKind::NotFound {
|
||||
eprintln!("warning: could not remove {}: {e}", path.display());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,11 +9,14 @@
|
||||
//! (routing raw super-kmers into partitions), `dereplicator` (deduplicating
|
||||
//! a partition's raw super-kmers), `counter` (counting unique canonical
|
||||
//! kmers), `layer_builder` (turning dereplicated super-kmers + counts into
|
||||
//! the real layer 0). [`extensions`] (private — see its own module docs)
|
||||
//! holds construction-only `KmerIndex` operations these algorithms share.
|
||||
//! the real layer 0). `extensions` (module itself private — see its own
|
||||
//! docs) holds `KmerIndex`/`KmerLayer` extension traits: some exclusive to
|
||||
//! this crate's own pipeline, one ([`IndexBuilder`]) public and re-exported
|
||||
//! here for external callers.
|
||||
|
||||
pub mod algorithms;
|
||||
pub(crate) mod extensions;
|
||||
mod extensions;
|
||||
pub mod graph_pipeline;
|
||||
|
||||
pub use extensions::IndexBuilder;
|
||||
pub use graph_pipeline::{build_graph, build_layer_from_kmers, materialize_layer, write_graph_as_unitigs};
|
||||
Reference in new issue
Block a user