Refactor k-mer indexing, enhance session features, and update documentation

This commit introduces significant refinements across the codebase. Core indexing logic is updated with hard constraints on k-mer length, statistical modeling clarifications, and the implementation of the Karlin & Ost theory for chance-collision statistics. Session management features are enhanced with robust memoization capabilities, atomic file operations, per-key locking, and memory mapping support. Additionally, documentation is updated, including a new citation style, and new mathematical notebook support is added.
This commit is contained in:
Eric Coissac committed 2026-10-13 08:41:42 +02:00
1 parent e715528d34
commit 0d1ecac233
13 files changed
+1683 -18

No files matched your search

+11
View File
@@ -98,6 +98,15 @@ version = "1.0.104"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
[[package]]
name = "approx"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cab112f0a86d568ea0e627cc1d6be74a1e9cd55214684db5561995f6dad897c6"
dependencies = [
"num-traits",
]
[[package]]
name = "arbitrary-chunks"
version = "0.4.1"
@@ -1665,8 +1674,10 @@ dependencies = [
name = "obiksession"
version = "0.1.0"
dependencies = [
"approx",
"memmap2",
"obisys",
"serde",
"tempfile",
"tracing",
"xxhash-rust",
+331
View File
@@ -0,0 +1,331 @@
//! Chance-collision statistics for exact k-mer matches shared by `r` of `s`
//! independent random sequences, after Karlin & Ost's asymptotic theory of
//! the longest common word among random letter sequences.
//!
//! # The question this answers
//!
//! Given `s` sequences of length `N` (e.g. `s` genomes), each drawn from an
//! i.i.d. equiprobable `m`-letter alphabet (`m = 4` for DNA), what is the
//! probability that a k-mer of length `k` is shared, purely by chance, by
//! at least `r` of those `s` sequences at unrelated, unaligned positions?
//! This is the null-model baseline against which an *observed* shared
//! k-mer (e.g. a [`crate::siblings`] family mask spanning `r` layers) can be
//! judged significant rather than a statistical accident.
//!
//! # Why not just `G / 4^k`
//!
//! The naive estimate used elsewhere for a single pair of sequences,
//! `P_chance(k, G) ~= G / 4^k` (see `UserDocMD/theory/kmer_indexing/kmers.md`),
//! treats every pair of positions as an independent trial. It ignores that
//! overlapping candidate matches (a match of length `L` contains a match of
//! every shorter length at the same start) are not independent events, and
//! it has no notion of "shared by `r` of `s` sequences" at all — only
//! pairwise. Karlin & Ost's asymptotic theory answers exactly this
//! generalized question, with a rigorous de-clumping correction instead of
//! the naive independence assumption.
//!
//! # Derivation
//!
//! For `s` independent i.i.d. equiprobable sequences over an alphabet of
//! size `m`, define the characteristic local match parameter for an
//! `r`-fold match (Karlin & Ost 1988, eq. 2.5-2.6):
//!
//! ```text
//! lambda^[r] = m^(1-r)
//! ```
//!
//! (`r = 2`: a plain pairwise match, `lambda = 1/m`; larger `r` shrinks
//! `lambda` geometrically, since all `r` sequences must agree on the same
//! word simultaneously).
//!
//! The expected number of `r`-fold common k-words among the `C(s, r)`
//! relevant groups of sequences is (eq. 2.9, with the i.i.d. equiprobable
//! simplification `gamma = 1` derived below):
//!
//! ```text
//! S_1(N, k) = N^r * C(s, r) * lambda^k
//! ```
//!
//! `gamma` (eq. 2.8) is defined as `lim_{k->oo} F(k) / lambda^k`, where
//! `F(k) = sum_w Pr{w}^r` over all `m^k` possible words `w`. For the i.i.d.
//! equiprobable case, `Pr{w} = m^-k` for every word, so `F(k) = m^k *
//! m^(-k*r) = m^(k*(1-r)) = lambda^k` **exactly**, for every `k` — not just
//! asymptotically. Hence `gamma = 1` exactly and the `(1 + o(1))` error term
//! in the general formula vanishes identically in this case; no correction
//! is dropped.
//!
//! Theorem 2.2 (eq. 2.12) gives the limiting distribution of the longest
//! `r`-fold common word length `K_{r,s}(N)` as `N -> oo`:
//!
//! ```text
//! Pr{K_{r,s}(N) < k} ~= exp{ -(1 - lambda) * S_1(N, k) }
//! ```
//!
//! so the probability of at least one chance `r`-fold match of length `k`
//! is
//!
//! ```text
//! P(k) = Pr{K_{r,s}(N) >= k} ~= 1 - exp{ -(1 - lambda) * S_1(N, k) }
//! ```
//!
//! The `(1 - lambda)` factor is the de-clumping correction: it accounts for
//! the fact that a single long chance match of length `L` is otherwise
//! counted once for every `k <= L` it contains, which would overstate the
//! rate of genuinely distinct matches.
//!
//! # Unequal genome lengths
//!
//! The `N^r` factor in `S_1(N, k)` is a shortcut for a more basic fact: for
//! one fixed group of `r` sequences, the number of candidate position
//! combinations is the *product* of that group's individual lengths (one
//! independent position choice per sequence in the group), not `N^r` — the
//! latter only holds because all `s` sequences share the same length `N`.
//! Dropping that assumption and summing over all `C(s, r)` groups of `s`
//! sequences with individual lengths `N_1, ..., N_s` gives
//!
//! ```text
//! S_1(k) = lambda^k * e_r(N_1, ..., N_s)
//! ```
//!
//! where `e_r` is the degree-`r` [elementary symmetric polynomial] in the
//! `s` lengths, `e_r(N_1,...,N_s) = sum_{i_1 < ... < i_r} N_i1 * N_i2 * ...
//! * N_ir`. Setting all `N_i = N` collapses `e_r` to `C(s, r) * N^r`,
//! recovering the equal-length formula above exactly — this is a strict
//! generalization, not a different approximation.
//!
//! [elementary symmetric polynomial]: https://en.wikipedia.org/wiki/Elementary_symmetric_polynomial
//!
//! The Remark following Theorem 2.2 in Karlin & Ost (1988) notes that for
//! the *limiting distribution* itself (not just its `S_1` parameter) to
//! converge, the `N_i` must stay of comparable order as they all grow
//! (formally, `(N_{sigma_1} * ... * N_{sigma_r}) / N^r` must converge to
//! some `alpha(sigma)` for a common reference scale `N`) — in practice this
//! is a non-issue unless genome sizes differ by several orders of
//! magnitude.
//!
//! `N_i` here is the sequence length proper; as a first approximation it
//! can be taken as the genome's k-mer set cardinality (or total k-mer
//! occurrence count), which is already what OBIKmer computes per genome
//! and tracks the true sequence length closely enough for this purpose.
//!
//! # Validity
//!
//! - The `exp{-(1-lambda) S_1}` form (Poisson approximation) is valid
//! whenever the per-position match probability `lambda^k` is small, which
//! holds throughout OBIKmer's supported range (`k in [11, 31]`) — it does
//! **not** require `S_1` itself to be small.
//! - The further linearisation `1 - exp(-x) ~= x` used by callers that want
//! `P(k) ~= (1 - lambda) * S_1(N, k)` **does** require `(1-lambda) *
//! S_1(N, k) << 1`; callers must check this themselves (or just use
//! [`p_at_least_one`], which always uses the exponential form and is
//! valid regardless of scale).
//! - This module only covers the *unrestricted-position* case (`K_{r,s}`,
//! matches anywhere in each sequence, not aligned at a common start) and
//! *distinct* sequences (`s` independent realizations). The single
//! sequence self-repeat case (a genome compared to itself, positions
//! drawn from the same realization) needs the reverse-complement variant
//! `R(N)` treated separately in Karlin & Ost (1987); it is not
//! implemented here.
//!
//! # References
//!
//! - Karlin, S. and Ost, F. (1988). "Maximal length of common words among
//! random letter sequences." *The Annals of Probability*, 16(2):535-563.
//! [`Theorem 2.2`] — the unrestricted-position `K_{r,s}(N)` result
//! implemented here. DOI: 10.1214/aop/1176991772.
//! - Karlin, S. and Ost, F. (1987). "Counts of long aligned word matches
//! among random letter sequences." *Advances in Applied Probability*,
//! 19(2):293-351. Companion paper for the *aligned* variant `C_{r,s}(N)`
//! (fixed common start position across sequences, relevant to local
//! alignment significance, not to unrestricted repeat-finding) — not
//! implemented here.
//! - Karlin, S. and Ost, F. (1985). "Maximal segmental match length among
//! random sequences from a finite alphabet." In *Proceedings of the
//! Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer*, vol. 1,
//! pp. 225-243. Wadsworth. No DOI found (pre-DOI conference proceedings).
//! Evaluates the reverse-complement match parameter needed for the
//! self-repeat case `R(N)` mentioned above.
/// DNA alphabet size (`A`, `C`, `G`, `T`), assumed equiprobable and
/// independent — the same null-model assumption used throughout
/// `UserDocMD/theory/kmer_indexing/kmers.md`.
const DNA_ALPHABET_SIZE: f64 = 4.0;
/// `C(n, k)`, the binomial coefficient, computed as `f64` since `S_1` is
/// itself only ever an asymptotic approximation — exact integer precision
/// is not meaningful here.
fn binomial(n: u32, k: u32) -> f64 {
if k > n {
return 0.0;
}
let k = k.min(n - k);
let mut result = 1.0f64;
for i in 0..k {
result *= (n - i) as f64 / (i + 1) as f64;
}
result
}
/// The Karlin-Ost characteristic local match parameter `lambda^[r] = m^(1-r)`
/// for `r` i.i.d. equiprobable sequences over an alphabet of size `m`
/// (eq. 2.5-2.6 of Karlin & Ost 1988).
///
/// `alphabet_size` defaults to [`DNA_ALPHABET_SIZE`] for the DNA case via
/// [`dna_lambda`]; this general form is kept for documentation fidelity to
/// the source formula and for non-DNA alphabets (e.g. amino acids).
fn lambda(alphabet_size: f64, r: u32) -> f64 {
alphabet_size.powi(1 - r as i32)
}
/// [`lambda`] specialized to the DNA alphabet (`m = 4`).
pub fn dna_lambda(r: u32) -> f64 {
lambda(DNA_ALPHABET_SIZE, r)
}
/// Degree-`r` elementary symmetric polynomial `e_r(lengths)` — see module
/// docs, "Unequal genome lengths". Computed by the standard `O(r * s)` DP:
/// `e[j]` accumulates the sum over all `j`-subsets of the lengths seen so
/// far, updated one length at a time from high `j` to low so each length is
/// used at most once per subset.
///
/// Returns `0.0` if `r > lengths.len()`.
fn elementary_symmetric(lengths: &[f64], r: u32) -> f64 {
let r = r as usize;
if r > lengths.len() {
return 0.0;
}
let mut e = vec![0.0f64; r + 1];
e[0] = 1.0;
for &length in lengths {
for j in (1..=r).rev() {
e[j] += length * e[j - 1];
}
}
e[r]
}
/// Expected number of chance `r`-fold common k-words among `C(s, r)` groups
/// of `s` independent i.i.d. equiprobable DNA sequences of individual
/// lengths `lengths` (Karlin & Ost 1988, eq. 2.9, generalized to unequal
/// lengths via the degree-`r` elementary symmetric polynomial — see module
/// docs, "Unequal genome lengths" — with the i.i.d. `gamma = 1`
/// simplification derived in the module docs).
///
/// `lengths` can be exact sequence lengths, or, as a first approximation,
/// each genome's k-mer set cardinality (or total k-mer occurrence count).
///
/// # Panics
///
/// Does not panic; returns `0.0` if `r > lengths.len()`.
pub fn expected_shared_kmers_varying(lengths: &[f64], k: u32, r: u32) -> f64 {
elementary_symmetric(lengths, r) * dna_lambda(r).powi(k as i32)
}
/// Expected number of chance `r`-fold common k-words among `C(s, r)` groups
/// of `s` independent i.i.d. equiprobable DNA sequences of length `n`
/// (Karlin & Ost 1988, eq. 2.9, with the i.i.d. `gamma = 1` simplification
/// derived in the module docs).
///
/// `r = 2, s = 2` recovers the plain pairwise chance-collision count for
/// two sequences; `r = s` recovers Karlin-Ost's within-a-single-genome
/// growth rate scaled by the number of independent sequences considered.
///
/// Closed-form fast path for the equal-length case — equivalent to
/// [`expected_shared_kmers_varying`] called with `s` copies of `n`, since
/// `e_r(n,...,n) = C(s, r) * n^r` (see module docs, "Unequal genome
/// lengths").
///
/// # Panics
///
/// Does not panic; returns `0.0` if `r > s`.
pub fn expected_shared_kmers(n: f64, k: u32, r: u32, s: u32) -> f64 {
n.powi(r as i32) * binomial(s, r) * dna_lambda(r).powi(k as i32)
}
/// Probability that at least one k-mer of length `k` is shared, purely by
/// chance, by at least `r` of `s` independent i.i.d. equiprobable DNA
/// sequences of individual lengths `lengths` (Karlin & Ost 1988, Theorem
/// 2.2 / eq. 2.12, exponential form, generalized to unequal lengths).
///
/// Valid across the full range relevant to OBIKmer (`k` in `[11, 31]`)
/// regardless of whether [`expected_shared_kmers_varying`] itself is small
/// or large — unlike the linear approximation `P ~= (1 - lambda) *
/// expected_shared_kmers_varying(..)`, which additionally requires the
/// latter to be `<< 1` (see module docs, "Validity").
pub fn p_at_least_one_varying(lengths: &[f64], k: u32, r: u32) -> f64 {
let s1 = expected_shared_kmers_varying(lengths, k, r);
let lambda = dna_lambda(r);
-f64::exp_m1(-(1.0 - lambda) * s1)
}
/// Probability that at least one k-mer of length `k` is shared, purely by
/// chance, by at least `r` of `s` independent i.i.d. equiprobable DNA
/// sequences of length `n` (Karlin & Ost 1988, Theorem 2.2 / eq. 2.12,
/// exponential form).
///
/// Valid across the full range relevant to OBIKmer (`k` in `[11, 31]`)
/// regardless of whether [`expected_shared_kmers`] itself is small or
/// large — unlike the linear approximation `P ~= (1 - lambda) *
/// expected_shared_kmers(..)`, which additionally requires the latter to
/// be `<< 1` (see module docs, "Validity").
pub fn p_at_least_one(n: f64, k: u32, r: u32, s: u32) -> f64 {
let s1 = expected_shared_kmers(n, k, r, s);
let lambda = dna_lambda(r);
-f64::exp_m1(-(1.0 - lambda) * s1)
}
#[cfg(test)]
mod tests {
use super::*;
/// Reproduces the r=2, s=2, N=1Gb, k=31 example worked out by hand:
/// S_1 ~= 0.217, P ~= 0.150.
#[test]
fn two_genomes_one_gb_k31() {
let s1 = expected_shared_kmers(1e9, 31, 2, 2);
assert!((s1 - 0.217).abs() < 0.01, "S_1 = {s1}");
let p = p_at_least_one(1e9, 31, 2, 2);
assert!((p - 0.150).abs() < 0.01, "P = {p}");
}
/// Reproduces the r=5, s=10, N=10Gb, k=31 example: S_1 ~= 5.6e-23.
#[test]
fn five_of_ten_genomes_ten_gb_k31() {
let s1 = expected_shared_kmers(1e10, 31, 5, 10);
assert!(s1 < 1e-20, "S_1 = {s1}");
assert!(s1 > 1e-25, "S_1 = {s1}");
}
/// Reproduces the r=3, s=10, N=10Gb, k=31 example: S_1 ~= 5.6e-6.
#[test]
fn three_of_ten_genomes_ten_gb_k31() {
let s1 = expected_shared_kmers(1e10, 31, 3, 10);
assert!((s1 - 5.6e-6).abs() < 1e-6, "S_1 = {s1}");
}
/// `expected_shared_kmers_varying` with `s` equal lengths must agree
/// with the closed-form `expected_shared_kmers`, since `e_r(n,...,n) =
/// C(s, r) * n^r`.
#[test]
fn varying_matches_closed_form_when_lengths_equal() {
let lengths = vec![1e10; 10];
let general = expected_shared_kmers_varying(&lengths, 31, 3);
let closed_form = expected_shared_kmers(1e10, 31, 3, 10);
assert!(
(general - closed_form).abs() / closed_form < 1e-9,
"general = {general}, closed_form = {closed_form}"
);
}
/// `e_2(a, b) = a * b` for two sequences — direct check of the
/// unequal-length case against hand computation, using an 8 Gb and a
/// 12 Gb genome at k=21.
#[test]
fn two_genomes_unequal_lengths_k21() {
let lengths = [8e9, 12e9];
let s1 = expected_shared_kmers_varying(&lengths, 21, 2);
let expected = 8e9 * 12e9 * dna_lambda(2).powi(21);
assert!((s1 - expected).abs() / expected < 1e-9, "S_1 = {s1}");
}
}
+8
View File
@@ -7,6 +7,13 @@
//! cardinality, pseudo-alignment); further phylo-domain functionality
//! (currently [`distance`]) moves here incrementally.
//!
//! [`karlin`] holds standalone chance-collision statistics (Karlin & Ost)
//! for judging whether a k-mer shared by several genomes/layers is
//! statistically significant; kept here for now as it has no other
//! consumer yet, but it depends on nothing else in this crate and may move
//! to a lower-level crate if an indexing-side consumer (e.g. default-`k`
//! guidance) appears.
//!
//! `siblings` is being reconnected piece by piece while `obikindex`'s
//! stateless audit is caught up with — see the project memory on this. So
//! far it only carries the annex-construction path
@@ -15,6 +22,7 @@
//! still disconnected.
mod distance;
pub mod karlin;
mod tree;
pub mod siblings;
+2
View File
@@ -8,6 +8,8 @@ obisys = { path = "../obisys", default-features = false }
memmap2 = "0.9.11"
xxhash-rust = { version = "0.8.18", features = ["xxh3"] }
tracing = "0.1"
serde = { version = "1", optional = true }
[dev-dependencies]
approx = "0.5.1"
tempfile = "3"
+15
View File
@@ -13,7 +13,22 @@
//! already used elsewhere in this project between a generic mechanism
//! crate and the domain crate that plugs into it (`obisys`/`obikindex`,
//! `obicompactvec`/`obikindexer`).
//!
//! ## Memoization
//!
//! [`Session::memoize`] returns a [`Memoize`] handle that lets callers
//! memoize expensive computations on top of the same checksummed,
//! persist-on-disk artifact store. Two API styles are provided:
//!
//! * **Manual** — [`Memoize::get_or_compute_with_key`] with an explicit
//! string key, or [`Memoize::get_or_compute`] with auto-derived keys
//! via the [`MemoKey`] trait.
//! * **Declarative** — the `session_memoize!` macro wraps a function so
//! that recursive calls transparently consult the cache, similar to
//! `kash`/`cached`.
mod memoize;
mod session;
pub use memoize::{MemoKey, Memoize, MemoizeValue};
pub use session::{ParamsConflict, Session, SessionOutcome};
+511
View File
@@ -0,0 +1,511 @@
use std::fmt;
use std::io;
use std::path::PathBuf;
use memmap2::Mmap;
use crate::Session;
use crate::session::{MEMO_DIR, checksum_path_for, verify_mmap, write_checked};
/// Trait for types that can be used as memoization keys.
///
/// The default implementation uses `Debug` + `xxh3` to produce a
/// stable string key from any `fmt::Debug` type. Callers can
/// override this to supply a custom key (e.g. a pre-hashed value
/// or a semantic key).
pub trait MemoKey: fmt::Debug {
fn memo_key(&self, function_name: &str) -> String;
}
impl<T: fmt::Debug> MemoKey for T {
fn memo_key(&self, function_name: &str) -> String {
let payload = format!("{}:{:?}", function_name, self);
xxhash_rust::xxh3::xxh3_64(payload.as_bytes()).to_string()
}
}
/// Marker trait for types that can be stored in the memoization
/// cache. The default blanket impl covers all `Vec<u8>` values.
pub trait MemoizeValue: Sized {
fn to_bytes(&self) -> Vec<u8>;
fn from_bytes(bytes: &[u8]) -> io::Result<Self>;
}
impl MemoizeValue for Vec<u8> {
fn to_bytes(&self) -> Vec<u8> {
self.clone()
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
Ok(bytes.to_vec())
}
}
// Blanket impl for serde types when the serde feature is enabled
#[cfg(feature = "serde")]
impl<T> MemoizeValue for T
where
T: serde::Serialize + serde::de::DeserializeOwned,
{
fn to_bytes(&self) -> Vec<u8> {
bincode::serialize(self).expect("serialization failed")
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
bincode::deserialize(bytes).map_err(io::Error::other)
}
}
// Blanket impls for common primitive types (no serde needed)
impl MemoizeValue for u8 {
fn to_bytes(&self) -> Vec<u8> {
vec![*self]
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
bytes.first().copied().ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidData, "invalid u8 bytes")
})
}
}
impl MemoizeValue for u16 {
fn to_bytes(&self) -> Vec<u8> {
self.to_le_bytes().to_vec()
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
let arr = <[u8; 2]>::try_from(bytes)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "invalid u16 bytes"))?;
Ok(u16::from_le_bytes(arr))
}
}
impl MemoizeValue for u32 {
fn to_bytes(&self) -> Vec<u8> {
self.to_le_bytes().to_vec()
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
let arr = <[u8; 4]>::try_from(bytes)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "invalid u32 bytes"))?;
Ok(u32::from_le_bytes(arr))
}
}
impl MemoizeValue for u64 {
fn to_bytes(&self) -> Vec<u8> {
self.to_le_bytes().to_vec()
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
let arr = <[u8; 8]>::try_from(bytes)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "invalid u64 bytes"))?;
Ok(u64::from_le_bytes(arr))
}
}
impl MemoizeValue for i32 {
fn to_bytes(&self) -> Vec<u8> {
self.to_le_bytes().to_vec()
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
let arr = <[u8; 4]>::try_from(bytes)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "invalid i32 bytes"))?;
Ok(i32::from_le_bytes(arr))
}
}
impl MemoizeValue for i64 {
fn to_bytes(&self) -> Vec<u8> {
self.to_le_bytes().to_vec()
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
let arr = <[u8; 8]>::try_from(bytes)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "invalid i64 bytes"))?;
Ok(i64::from_le_bytes(arr))
}
}
impl MemoizeValue for f64 {
fn to_bytes(&self) -> Vec<u8> {
self.to_le_bytes().to_vec()
}
fn from_bytes(bytes: &[u8]) -> io::Result<Self> {
let arr = <[u8; 8]>::try_from(bytes)
.map_err(|_| io::Error::new(io::ErrorKind::InvalidData, "invalid f64 bytes"))?;
Ok(f64::from_le_bytes(arr))
}
}
/// A memoization handle backed by a `Session` directory.
#[derive(Debug)]
pub struct Memoize<'a> {
session: &'a Session,
}
impl<'a> Memoize<'a> {
pub(crate) fn new(session: &'a Session) -> Self {
Self { session }
}
/// Retrieve a cached value by its explicit key, or compute and
/// store it via `compute_fn`.
///
/// Concurrent calls for the *same* `key` (from different threads
/// sharing this session) don't race to recompute: the first caller
/// runs `compute_fn` while the others block on
/// [`Session::with_key_lock`], then each re-checks the cache — which
/// the first caller has by then populated — instead of recomputing.
pub fn get_or_compute_with_key(
&self,
key: &str,
compute_fn: impl FnOnce() -> io::Result<Vec<u8>>,
) -> io::Result<Vec<u8>> {
if let Some(cached) = self.restore_raw(key)? {
return Ok(cached.to_vec());
}
self.session.with_key_lock(key, || {
if let Some(cached) = self.restore_raw(key)? {
return Ok(cached.to_vec());
}
let value = compute_fn()?;
self.store_raw(key, &value)?;
Ok(value)
})
}
/// Retrieve a cached value whose key is auto-derived from
/// `args` via [`MemoKey`], or compute and store it.
pub fn get_or_compute<K, V, F>(
&self,
function_name: &str,
args: &K,
compute_fn: F,
) -> io::Result<Vec<u8>>
where
K: MemoKey,
V: MemoizeValue,
F: FnOnce() -> io::Result<V>,
{
let key = args.memo_key(function_name);
let raw = self.get_or_compute_with_key(&key, || {
compute_fn().map(|v| v.to_bytes())
})?;
Ok(raw)
}
/// Restore a raw (byte) artifact by name, `mmap`ed read-only.
fn restore_raw(&self, name: &str) -> io::Result<Option<Mmap>> {
let path = self.memo_path(name);
verify_mmap(&path)
}
/// Store raw bytes under `name`, atomically with a checksum sidecar.
fn store_raw(&self, name: &str, bytes: &[u8]) -> io::Result<()> {
let path = self.memo_path(name);
write_checked(&path, bytes)
}
/// Remove a single memoized entry by key. Returns `true` if the
/// entry existed and was deleted.
pub fn invalidate(&self, key: &str) -> io::Result<bool> {
let path = self.memo_path(key);
let checksum_path = checksum_path_for(&path);
let existed = path.exists();
if path.exists() {
std::fs::remove_file(&path)?;
}
if checksum_path.exists() {
let _ = std::fs::remove_file(&checksum_path);
}
Ok(existed)
}
pub fn cached_keys(&self) -> io::Result<Vec<String>> {
let dir = self.memo_dir();
if !dir.exists() {
return Ok(Vec::new());
}
let mut keys = Vec::new();
for entry in std::fs::read_dir(dir)? {
let entry = entry?;
let name = entry.file_name();
let name = name.to_string_lossy().to_string();
if name.ends_with(".xxh3") || name.ends_with(".tmp") {
continue;
}
keys.push(name);
}
Ok(keys)
}
fn memo_dir(&self) -> PathBuf {
self.session.dir().join(MEMO_DIR)
}
fn memo_path(&self, name: &str) -> PathBuf {
self.memo_dir().join(name)
}
}
/// Declarative memoization macro, similar in spirit to `kash`/`cached`.
///
/// Wraps a function so that every call transparently consults the
/// session's persistent cache. On a cache miss the function body
/// is executed and its return value is stored; on a hit the stored
/// value is returned without re-executing the body.
///
/// # Syntax
///
/// ```ignore
/// session_memoize! {
/// fn my_func(session: &Session, a: u64, b: &str) -> MyResult {
/// // expensive computation
/// }
/// }
/// ```
///
/// After expansion, call `my_func(&session, 5, "hello")` — the function
/// returns `io::Result<MyResult>`. Check the result with `.unwrap()` or
/// `.expect()`.
///
/// Recursive calls within the body to `my_func` also pass through the
/// cache (they must also include the session as the first argument).
///
/// # Requirements
///
/// * The first parameter must be `session: &Session`.
/// * All argument types must implement [`MemoKey`] (provided for `Debug` types).
/// * The return type must implement [`MemoizeValue`] (provided for
/// `Vec<u8>`, primitive numeric types, and, when the `serde`
/// feature is enabled, for `Serialize + DeserializeOwned` types).
#[macro_export]
macro_rules! session_memoize {
(
fn $name:ident(session: &Session, $($arg:ident : $arg_ty:ty),*) -> $ret:ty $body:block
) => {
fn $name(session: &Session, $($arg: $arg_ty),*) -> ::std::io::Result<$ret> {
let memo = session.memoize();
let args = (&($($arg),*),);
let raw = memo.get_or_compute(
stringify!($name),
&args,
|| -> ::std::io::Result<$ret> {
let __result: $ret = $body;
Ok(__result)
},
)?;
<$ret as $crate::MemoizeValue>::from_bytes(&raw)
.map_err(|e| ::std::io::Error::new(::std::io::ErrorKind::Other, e))
}
};
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn memoize_get_or_compute_miss_then_hit() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
let memo = Memoize::new(&session);
let computed = memo.get_or_compute_with_key("key1", || Ok(b"value1".to_vec())).unwrap();
assert_eq!(&computed[..], b"value1");
let cached = memo.get_or_compute_with_key("key1", || panic!("should not be called")).unwrap();
assert_eq!(&cached[..], b"value1");
}
#[test]
fn memoize_invalidate() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
let memo = Memoize::new(&session);
memo.get_or_compute_with_key("key1", || Ok(b"value1".to_vec())).unwrap();
assert!(memo.invalidate("key1").unwrap());
assert!(memo.restore_raw("key1").unwrap().is_none());
assert!(!memo.invalidate("key1").unwrap());
}
#[test]
fn memoize_cached_keys() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
let memo = Memoize::new(&session);
assert!(memo.cached_keys().unwrap().is_empty());
memo.get_or_compute_with_key("a", || Ok(vec![1])).unwrap();
memo.get_or_compute_with_key("b", || Ok(vec![2])).unwrap();
let keys = memo.cached_keys().unwrap();
assert!(keys.contains(&"a".to_string()));
assert!(keys.contains(&"b".to_string()));
assert_eq!(keys.len(), 2);
}
#[test]
fn memo_key_from_debug() {
#[derive(Debug)]
struct Args {
x: u64,
y: String,
}
let args = Args { x: 42, y: "hello".into() };
let key1 = args.memo_key("my_fn");
let key2 = args.memo_key("my_fn");
assert_eq!(key1, key2);
let key_other = args.memo_key("other_fn");
assert_ne!(key1, key_other);
}
#[test]
fn session_memoize_macro_u64() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
session_memoize! {
fn double(session: &Session, x: u64) -> u64 {
x * 2
}
}
let result = double(&session, 7).unwrap();
assert_eq!(result, 14);
let result2 = double(&session, 7).unwrap();
assert_eq!(result2, 14);
}
#[test]
fn session_memoize_macro_i32() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
session_memoize! {
fn add(session: &Session, a: i32, b: i32) -> i32 {
a + b
}
}
let result = add(&session, 3, 4).unwrap();
assert_eq!(result, 7);
let result2 = add(&session, 3, 4).unwrap();
assert_eq!(result2, 7);
}
#[test]
fn session_memoize_macro_f64() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
session_memoize! {
fn square(session: &Session, x: f64) -> f64 {
x * x
}
}
let result = square(&session, 3.0).unwrap();
approx::assert_relative_eq!(result, 9.0);
let result2 = square(&session, 3.0).unwrap();
approx::assert_relative_eq!(result2, 9.0);
}
#[test]
fn session_memoize_macro_persists_across_sessions() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
// Define the memoized function once
session_memoize! {
fn cached_add(session: &Session, a: u64, b: u64) -> u64 {
a + b
}
}
// First session: compute and cache
{
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
assert_eq!(cached_add(&session, 3, 4).unwrap(), 7);
}
// Second session with same params: should hit cache
{
let session = Session::open(&dir, b"params", false).unwrap().unwrap();
assert_eq!(cached_add(&session, 3, 4).unwrap(), 7);
}
}
#[test]
fn session_memoize_macro_invalidated_by_force_reset() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
session_memoize! {
fn tagged(session: &Session, x: u64) -> u64 {
x
}
}
// First session: compute and cache under "params-v1".
{
let session = Session::open(&dir, b"params-v1", false).unwrap().unwrap();
assert_eq!(tagged(&session, 1).unwrap(), 1);
assert_eq!(session.memoize().cached_keys().unwrap().len(), 1);
}
// Reopen with different params and force: true — the whole
// session, including the memo cache, is reset, so the call
// below must recompute rather than reuse the stale entry.
{
let session = Session::open(&dir, b"params-v2", true).unwrap().unwrap();
assert!(session.memoize().cached_keys().unwrap().is_empty());
assert_eq!(tagged(&session, 1).unwrap(), 1);
}
}
#[test]
fn concurrent_calls_for_same_key_compute_once() {
let tmp = tempfile::tempdir().unwrap();
let dir = tmp.path().join("session");
let session = std::sync::Arc::new(Session::open(&dir, b"params", false).unwrap().unwrap());
let calls = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
// Forces both threads to reach `get_or_compute_with_key` at
// essentially the same instant, so the second one is guaranteed
// to hit the "wait, don't recompute" path instead of just
// happening to run after the first one already finished.
let barrier = std::sync::Arc::new(std::sync::Barrier::new(2));
let handles: Vec<_> = (0..2)
.map(|_| {
let session = session.clone();
let calls = calls.clone();
let barrier = barrier.clone();
std::thread::spawn(move || {
barrier.wait();
session
.memoize()
.get_or_compute_with_key("shared-key", || {
calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
std::thread::sleep(std::time::Duration::from_millis(50));
Ok(b"computed".to_vec())
})
.unwrap()
})
})
.collect();
for handle in handles {
assert_eq!(&handle.join().unwrap()[..], b"computed");
}
assert_eq!(calls.load(std::sync::atomic::Ordering::SeqCst), 1);
// The per-key lock must have been cleaned up once both threads
// are done with it, not left behind forever.
assert_eq!(session.memo_lock_table_len(), 0);
}
}
+93 -13
View File
@@ -1,12 +1,15 @@
use std::collections::HashMap;
use std::fs;
use std::io;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use memmap2::Mmap;
use obisys::DirLock;
const PARAMS_FILE: &str = "params.bin";
const ARTIFACTS_DIR: &str = "artifacts";
pub(crate) const MEMO_DIR: &str = "memo";
const CHECKSUM_EXT: &str = "xxh3";
/// Whether [`Session::open`] found a fresh (just-created, or just-reset by
@@ -48,6 +51,9 @@ pub struct Session {
dir: PathBuf,
outcome: SessionOutcome,
_lock: DirLock,
/// Per-key in-process locks backing [`Memoize`](crate::Memoize)'s
/// single-flight behavior — see [`Session::with_key_lock`].
memo_locks: Mutex<HashMap<String, Arc<Mutex<()>>>>,
}
impl Session {
@@ -66,24 +72,31 @@ impl Session {
dir: dir.to_path_buf(),
outcome: SessionOutcome::Reused,
_lock: lock,
memo_locks: Mutex::new(HashMap::new()),
})),
Some(saved) if !force => Ok(Err(ParamsConflict { saved, requested: params.to_vec() })),
_ => {
// No saved params yet, or a mismatch with `force: true`:
// start clean. Any artifact cached under different params
// is stale by construction — deleting the whole
// `artifacts/` directory is simpler and safer than trying
// to figure out which entries are still valid.
// start clean. Any artifact or memoized entry cached under
// different params is stale by construction — deleting the
// whole `artifacts/` and `memo/` directories is simpler and
// safer than trying to figure out which entries are still
// valid.
let artifacts_dir = dir.join(ARTIFACTS_DIR);
if artifacts_dir.exists() {
fs::remove_dir_all(&artifacts_dir)?;
}
fs::create_dir_all(&artifacts_dir)?;
let memo_dir = dir.join(MEMO_DIR);
if memo_dir.exists() {
fs::remove_dir_all(&memo_dir)?;
}
write_checked(&params_path, params)?;
Ok(Ok(Session {
dir: dir.to_path_buf(),
outcome: SessionOutcome::Created,
_lock: lock,
memo_locks: Mutex::new(HashMap::new()),
}))
}
}
@@ -123,20 +136,87 @@ impl Session {
pub fn store(&self, name: &str, bytes: &[u8]) -> io::Result<()> {
write_checked(&self.artifact_path(name), bytes)
}
pub fn memoize(&self) -> crate::Memoize<'_> {
crate::Memoize::new(self)
}
/// Serialize concurrent calls that share the same `key`, within this
/// process: the first caller for a given `key` runs `f` while any
/// other caller for that *same* `key` blocks; once `f` returns, each
/// blocked caller runs in turn. Backs [`Memoize`](crate::Memoize)'s
/// single-flight behavior — combined with a cache re-check inside `f`,
/// it turns "N threads race to recompute and overwrite the same
/// artifact" into "one thread computes, the rest wait and then read
/// the cache".
///
/// Purely an in-process mechanism: cross-process exclusivity is
/// already provided by the `DirLock` held for the `Session`'s entire
/// lifetime, so contention here only ever happens between threads of
/// this one process sharing the same `Session`.
///
/// The per-key lock is garbage-collected: its entry is removed as
/// soon as no other thread is (or could still be) waiting on it, so a
/// long-running process doesn't accumulate one entry per distinct key
/// ever requested — only per key currently or recently contended.
pub(crate) fn with_key_lock<T>(&self, key: &str, f: impl FnOnce() -> T) -> T {
let entry = {
let mut table = self.memo_locks.lock().unwrap();
table.entry(key.to_string()).or_insert_with(|| Arc::new(Mutex::new(()))).clone()
};
let guard = entry.lock().unwrap();
let result = f();
// Still holding `guard`: any other thread that already cloned
// `entry` out of the table is necessarily still blocked trying to
// acquire it below — it cannot have run `f` and returned yet — so
// `entry`'s strong count here is exact. `table` holds one
// reference and this call holds the other (`entry` itself); if
// that is still all of them, nobody else is or ever will be
// waiting on this particular entry, so it is safe to drop it from
// the table. A later call for the same `key` just starts a fresh
// entry.
{
let mut table = self.memo_locks.lock().unwrap();
if Arc::strong_count(&entry) == 2 {
table.remove(key);
}
}
drop(guard);
result
}
/// Number of currently-contended per-key locks — exposed only for
/// tests, to check that [`with_key_lock`](Self::with_key_lock)'s
/// garbage collection actually removes entries once uncontended.
#[cfg(test)]
pub(crate) fn memo_lock_table_len(&self) -> usize {
self.memo_locks.lock().unwrap().len()
}
}
fn checksum_path_for(path: &Path) -> PathBuf {
pub(crate) fn checksum_path_for(path: &Path) -> PathBuf {
let mut os_str = path.as_os_str().to_owned();
os_str.push(".");
os_str.push(CHECKSUM_EXT);
PathBuf::from(os_str)
}
/// `mmap` `path` and verify it against its checksum sidecar. `Ok(None)`
/// when `path` doesn't exist, its checksum sidecar doesn't exist/parse, or
/// the checksum doesn't match — every case here means "nothing usable is
/// cached," never an error the caller has to handle specially.
fn verify_mmap(path: &Path) -> io::Result<Option<Mmap>> {
/// Append `.tmp` to `path`'s full file name (never replaces an existing
/// extension the way [`Path::with_extension`] would). Used to derive the
/// staging path for both the data file and the checksum sidecar in
/// [`write_checked`] — appending rather than replacing guarantees the two
/// staging paths can never collide with each other, nor with an unrelated
/// artifact whose name happens to contain a `.`.
fn tmp_path_for(path: &Path) -> PathBuf {
let mut os_str = path.as_os_str().to_owned();
os_str.push(".tmp");
PathBuf::from(os_str)
}
pub(crate) fn verify_mmap(path: &Path) -> io::Result<Option<Mmap>> {
if !path.exists() {
return Ok(None);
}
@@ -169,17 +249,17 @@ fn read_checked(path: &Path) -> io::Result<Option<Vec<u8>>> {
Ok(verify_mmap(path)?.map(|mmap| mmap[..].to_vec()))
}
fn write_checked(path: &Path, bytes: &[u8]) -> io::Result<()> {
pub(crate) fn write_checked(path: &Path, bytes: &[u8]) -> io::Result<()> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let tmp = path.with_extension("tmp");
let tmp = tmp_path_for(path);
fs::write(&tmp, bytes)?;
fs::rename(&tmp, path)?;
let checksum = xxhash_rust::xxh3::xxh3_64(bytes);
let checksum_path = checksum_path_for(path);
let checksum_tmp = checksum_path.with_extension("tmp");
let checksum_tmp = tmp_path_for(&checksum_path);
fs::write(&checksum_tmp, checksum.to_string())?;
fs::rename(&checksum_tmp, checksum_path)?;
Ok(())