diff --git a/DevDocMD/architecture/hot_resume_memoization.md b/DevDocMD/architecture/hot_resume_memoization.md new file mode 100644 index 00000000..d737c1a9 --- /dev/null +++ b/DevDocMD/architecture/hot_resume_memoization.md @@ -0,0 +1,61 @@ +# Hot resume and memoization of long computations (discussion, 2026-10-14) + +## Goal (user) + +1. **Hot resume**: a run killed midway (crash, `Ctrl-C`, scheduler walltime) restarts from the last finished unit of work, not from zero. +2. **No recomputation**: a long function called again with the same parameters (same command rerun, or another command needing the same intermediate) reuses its stored result. + +Not a goal: bit-identical resumed samples. Sampling uses an OS-seeded RNG on purpose, repeated runs must stay independent draws to measure variance; a resumed run being a *different* valid draw is not a defect. + +## Why it matters: one real run + +`phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance --family-overlap --sibling-stats` on the 180-genome Primulaceae index, from the stderr of the OAR job (2026-10-13), 256 partitions: + +| Stage | Wall | +|---|---| +| `sibling_stats` (full scan, `Selection::All`) | 8 235 s | +| `family_overlap` (sweep 1: overlap annex) | 5 582 s | +| `sampled_scan`: sweep 1 again (entropy + quartet annexes) | ≈ 5 347 s | +| `sampled_scan`: sweep 2 (pair-weight annex) | ≈ 5 296 s | +| `sampled_scan`: the draw itself | ≈ 9 573 s | +| `distance` (Jaccard, not asked for; older binary) | 53 s | +| Total | 34 085 s (9 h 28) | + +The three annex sweeps cost about the same although their per-family work differs, so the cost is the layer traversal. Part of it was addressed by restricting the matrix pass to selected families (`family_scan.rs`, same session); the sweeps were not fused. Nothing in this run could be resumed or reused: every stage restarts from its beginning. + +## What exists today (verified in the code, 2026-10-14) + +**Persisted per-layer annexes.** Entropy (`entropy*.pent`), quartet (`quartet*.pbiv`), pair weight, sibling annex: one file per layer, built by `ensure_layer_*` and skipped when `path.exists()` (`family_annexes.rs`). A rerun with the same conventions skips every sweep. This is already memoization, keyed by *file name only* (convention in the name, no fingerprint of the inputs; a rebuilt sibling annex deletes everything derived from it, see `pair_coverage_sampling.md`). + +**Not persisted.** The family-overlap accumulator (`FamilyOverlapAccumulator`) is written once at the end of sweep 1 (`sibling_builder.rs`). The draw (`sample_index`) and the `sibling_stats` scan keep their state in memory only. + +**Defect found, then fixed (2026-10-14): a killed annex build left a file that looked finished.** `create_mmap_file` (`obicompactvec/src/mmap_file.rs`) created the file at its *final path* with magic, header and the full length (`set_len`) before any data was written. A kill during a layer left a valid-looking file with unwritten data; the next run saw `path.exists()` and treated the annex as built, silently. + +Reproduced on the 1x testbed (20 genomes, 512 layers): derived annexes deleted, `phylo --family-overlap --quartet --pair-balance --entropy 1 …` killed by `timeout 6`, then rerun. Old binary: 141 finals after the kill, 2 differing from a reference build; the rerun exited 0 and kept both. Fixed binary: kill at 6 s left 499 finals, all identical to the reference, plus 2 `.building` files; the rerun completed 1 537 files identical to the reference, no `.building` left. (Kill at 3 s: nothing published, nothing broken.) + +**Fix.** `obicompactvec::building_path` / `publish_built`: a file is built under `.building` and moved to its final name (same-directory `rename`, atomic) by the builder's `close()`, after the flush. Applied to the builders that go through `create_mmap_file` or write their own mmap on the annex path: `PersistentBitVecBuilder` (all constructors; quartet annex), `PersistentFixedIntVecBuilder`, `PersistentRankSelectBitVecBuilder`, `FamilyScalarAnnexBuilder` (entropy, pair weight), `SiblingAnnexBuilder` (`siblings.psib`) and `FamilyOverlapAccumulator::write` (overlap matrix). A leftover `.building` is truncated by the next build of that file. A build that is dropped without `close()` no longer produces the final file (no caller did; workspace tests pass). Also reordered in `--sibling-annex`: derived annexes are now removed *before* the new sibling annex is built and published, so a kill between the two leaves the old annex with no derived files, never a new annex next to files computed against the old numbering. + +**Not covered.** The builders outside that path still write at their final name: `PersistentCompactIntVecBuilder`, the int/sparse matrix builders, `EliasFanoBuilder`, `TempBitVecBuilder`/`TempCompactIntVecBuilder` (temporary by nature) and the index-construction files that use them. The same reasoning applies to them as soon as a stage built on them has to be resumed; not audited. The in-memory overlap accumulator is still lost on a kill (a sweep restarts), now without leaving a corrupt file. + +**`obiksession` crate** (`src/obiksession/`): `Session` (named directory, opaque parameter blob checked on open, whole-lifetime `DirLock`, artifacts checksummed with xxh3 and mmap-restored) and `Memoize` (`get_or_compute[_with_key]`, `invalidate`, `cached_keys`, `MemoKey` from `Debug` + xxh3, the `session_memoize!` macro). Compiles, 15 tests pass, **no caller**: `obikmer` lists the dependency without using it. `phylo --session` exits with an error (`cmd/phylo/mod.rs:271`) since `session_cache.rs` (the consumer tied to `PairwiseTally`/`PartitionDispersion`/`SnpAlignment`) was removed. `sample_index(skip_layers)` remains as a seam, never called with a non-zero value. + +**Previous design and why it missed the goal** (`theory/evolutionary_distances.md`, section "Explicit `--session`"): one session per *exact* parameter tuple (changing `--n-sites` made another session), artifacts specific to the sampling result, resume only inside the draw (checkpoint every 8 layers or 30 s). Reuse across commands existed only for the sample. An exhaustive-sample cache from which smaller samples would be drawn was proposed and never done. User judgement: this version did not answer the objective. + +## Proposed direction (not implemented; to be validated) + +Start from the goal, not from the existing crate. + +1. **Unit of work = one `(partition, layer)` step that already produces files.** The annex builds, the draw per layer and the scans all iterate that way (`for part … for layer …`). Resume means "skip the units whose output is complete", exactly what `exists()` approximates today. +2. **Complete-or-absent outputs.** Done for the annex path (see the fix above); the remaining builders are to be covered as their stages become resumable. Prerequisite for everything below. The mechanism lives in `obicompactvec` (shared lower-level API), not in each consumer. +3. **Key = function identity + parameters + fingerprint of the inputs.** A result is reusable iff the function, every parameter that influences it and the identity of its inputs (index content, upstream artifacts) match. Today the key is a file name plus an invalidation convention. The fingerprint requirement is the cost: what identifies an index version cheaply is to be decided (meta file, rebuild counter, hash of annex headers). +4. **The cache holds results of long functions, not domain objects.** The crate must not know `PairwiseTally`. Callers serialize their own values (`rkyv` was already chosen for that). Whether `Memoize`'s API fits is unproven: it has no caller, no documentation and was written against the earlier design. +5. **Granularity matters more than the API.** A cached 9 h result helps only the exact rerun; a per-layer cache also helps a changed parameter that leaves earlier layers untouched. The overlap accumulator (end-of-sweep write) would need a per-layer partial (a sum of per-layer matrices, mergeable) to be resumable. +6. **Fuse before caching.** Three sweeps over the same layers where two suffice (overlap first, then entropy + quartet + pair weight together, since the pair weight needs only the finished overlap matrix) is a cost no cache removes on a first run; to do independently. + +## Open questions + +- Fingerprint of an index version: which cheap, stable identity? +- Overlap accumulator resume: per-layer partial matrices on disk (size n² × layers) or a checkpoint every N layers? +- `Memoize`/`Session`: keep (adapt to the points above), rewrite, or delete the crate until a caller exists (the "existing code is a hypothesis" rule applies; today it has no user). +- Scope of the next step: atomic writes are done for the annex path; extend to the other builders, or go to the cache. +- Draw resume: a layer's draw is independent of the others except through the quotas, which depend on every layer's mass; quotas can be recomputed from annexes on resume (cheap), so only the per-layer survivors need storing. diff --git a/mkdocs.yml b/mkdocs.yml index ee06c0fe..3b306ff5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -64,6 +64,7 @@ nav: - Sequences: architecture/sequences/invariant.md - Kmer index: architecture/index_architecture.md - Sibling annex: architecture/siblings.md + - Hot resume and memoization: architecture/hot_resume_memoization.md - NUMA-aware worker pools: architecture/numa_worker_pools.md - NUMA-aware partition runner: architecture/numa_partition_runner.md diff --git a/src/obicompactvec/src/bitvec.rs b/src/obicompactvec/src/bitvec.rs index 3cde1910..087e3527 100644 --- a/src/obicompactvec/src/bitvec.rs +++ b/src/obicompactvec/src/bitvec.rs @@ -4,7 +4,7 @@ use std::path::{Path, PathBuf}; use memmap2::{Mmap, MmapMut}; -use crate::mmap_file::{create_mmap_file, open_mmap_file}; +use crate::mmap_file::{building_path, create_mmap_file, open_mmap_file, publish_built}; use crate::reader::PersistentCompactIntVec; use crate::views::{BitSliceIter, BitSliceView, IntSliceView}; @@ -206,7 +206,7 @@ impl PersistentBitVecBuilder { .write(true) .create(true) .truncate(true) - .open(path)?; + .open(building_path(path))?; file.write_all(&MAGIC)?; file.write_all(&[0u8; 4])?; file.write_all(&(n as u64).to_le_bytes())?; @@ -229,8 +229,8 @@ impl PersistentBitVecBuilder { } pub fn build_from(source: &PersistentBitVec, path: &Path) -> io::Result { - fs::copy(source.path(), path)?; - let file = OpenOptions::new().read(true).write(true).open(path)?; + fs::copy(source.path(), building_path(path))?; + let file = OpenOptions::new().read(true).write(true).open(building_path(path))?; let mmap = unsafe { MmapMut::map_mut(&file)? }; let n = source.len(); Ok(Self { @@ -441,7 +441,8 @@ impl PersistentBitVecBuilder { } pub fn close(self) -> io::Result<()> { - self.mmap.flush() + self.mmap.flush()?; + publish_built(&self.path) } pub fn finish(self) -> io::Result { diff --git a/src/obicompactvec/src/fixedintvec.rs b/src/obicompactvec/src/fixedintvec.rs index 4645d82c..a1b74854 100644 --- a/src/obicompactvec/src/fixedintvec.rs +++ b/src/obicompactvec/src/fixedintvec.rs @@ -15,7 +15,7 @@ use std::path::{Path, PathBuf}; use memmap2::{Mmap, MmapMut}; -use crate::mmap_file::{create_mmap_file, open_mmap_file}; +use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built}; const MAGIC: [u8; 4] = *b"PFIV"; @@ -178,7 +178,8 @@ impl PersistentFixedIntVecBuilder { } pub fn close(self) -> io::Result<()> { - self.mmap.flush() + self.mmap.flush()?; + publish_built(&self.path) } pub fn finish(self) -> io::Result { diff --git a/src/obicompactvec/src/lib.rs b/src/obicompactvec/src/lib.rs index fb1ad1ed..07fb1c13 100644 --- a/src/obicompactvec/src/lib.rs +++ b/src/obicompactvec/src/lib.rs @@ -35,6 +35,7 @@ pub use intmatrix::{ }; pub use layer_meta::LayerMeta; +pub use mmap_file::{building_path, publish_built}; pub use matrix_builder::{ColBuilder, MatrixBuilder}; pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder}; pub use reader::{Iter as CompactIntVecIter, PersistentCompactIntVec}; diff --git a/src/obicompactvec/src/mmap_file.rs b/src/obicompactvec/src/mmap_file.rs index 02c56265..ece6c106 100644 --- a/src/obicompactvec/src/mmap_file.rs +++ b/src/obicompactvec/src/mmap_file.rs @@ -10,13 +10,35 @@ //! `block_ranks`) and its own higher-level API — those genuinely differ //! between formats and aren't forced into a shared shape. -use std::fs::{File, OpenOptions}; +use std::fs::{self, File, OpenOptions}; use std::io::{self, Seek, SeekFrom, Write as _}; -use std::path::Path; +use std::path::{Path, PathBuf}; use memmap2::{Mmap, MmapMut}; -/// Creates (or truncates) `path`, writes the standard header — `magic` + +/// Suffix of the name a file lives under while it is being built. +const BUILDING_SUFFIX: &str = ".building"; + +/// The name `path` is built under: a file only appears at `path` itself once +/// complete ([`publish_built`]). A process killed mid-build leaves at most a +/// `.building` file, never a `path` with a valid header and unwritten data +/// that `path.exists()` would take for a finished one — what every +/// "skip it, it already exists" caller (per-layer annexes) relies on. +pub fn building_path(path: &Path) -> PathBuf { + let mut name = path.as_os_str().to_owned(); + name.push(BUILDING_SUFFIX); + PathBuf::from(name) +} + +/// Atomically moves the file built under [`building_path`]`(path)` to `path` +/// (same directory, so a plain `rename`), replacing any previous one. To be +/// called once the data is complete and flushed. +pub fn publish_built(path: &Path) -> io::Result<()> { + fs::rename(building_path(path), path) +} + +/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it +/// with [`publish_built`] once complete — writes the standard header — `magic` + /// `pad` (4 bytes right after `magic`; usually all-zero, but /// `PersistentFixedIntVec` repurposes byte 4 to store its `width`) + /// `n as u64`, followed by `extra_header_zeros` more zero bytes for a @@ -37,7 +59,7 @@ pub(crate) fn create_mmap_file( .write(true) .create(true) .truncate(true) - .open(path)?; + .open(building_path(path))?; file.write_all(&magic)?; file.write_all(&pad)?; file.write_all(&(n as u64).to_le_bytes())?; diff --git a/src/obicompactvec/src/rankselect.rs b/src/obicompactvec/src/rankselect.rs index a3514ef1..0a8d9e1e 100644 --- a/src/obicompactvec/src/rankselect.rs +++ b/src/obicompactvec/src/rankselect.rs @@ -21,7 +21,7 @@ use std::path::{Path, PathBuf}; use common_traits::SelectInWord; use memmap2::{Mmap, MmapMut}; -use crate::mmap_file::{create_mmap_file, open_mmap_file}; +use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built}; const MAGIC: [u8; 4] = *b"PRSB"; @@ -268,7 +268,8 @@ impl PersistentRankSelectBitVecBuilder { let block_off = HEADER_SIZE + bits_bytes(self.n); let block_bytes = u64_slice_to_le_bytes(&block_ranks); self.mmap[block_off..block_off + block_bytes.len()].copy_from_slice(&block_bytes); - self.mmap.flush() + self.mmap.flush()?; + publish_built(&self.path) } pub fn finish(self) -> io::Result { diff --git a/src/obikphylo/src/siblings/extensions/sibling_ext.rs b/src/obikphylo/src/siblings/extensions/sibling_ext.rs index 3f7c1db8..d898de53 100644 --- a/src/obikphylo/src/siblings/extensions/sibling_ext.rs +++ b/src/obikphylo/src/siblings/extensions/sibling_ext.rs @@ -129,16 +129,19 @@ impl SiblingExt for IndexCache { let layer = self .get_layer(part, l) .expect("cache-consistent: n_layer(part) already bounds l"); - part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?; - // Invalidate this layer's entropy and quartet annexes, if // any — they were built against the sibling annex's // *previous* `family_idx` numbering // (`non_monomorphic_reindex_layer` reads the sibling - // annex), now stale. `ensure_family_annexes` below only - // builds what's *missing*, so the old files must be removed - // here for them to be rebuilt rather than silently left - // stale. + // annex), and are stale once the new one exists. + // `ensure_family_annexes` below only builds what's + // *missing*, so the old files must be removed here for + // them to be rebuilt rather than silently left stale. + // Removed *before* the new annex is built and published + // (atomically, at its end): a build killed in between + // leaves the previous annex with no derived files — still + // consistent, rebuilt on demand — never a new annex next to + // derived files computed against the old numbering. for name in ScalarKind::Entropy .file_names() .into_iter() @@ -150,6 +153,7 @@ impl SiblingExt for IndexCache { std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?; } } + part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?; } total_slots += part_slots; pb.inc(1); diff --git a/src/obikphylo/src/siblings/family_overlap.rs b/src/obikphylo/src/siblings/family_overlap.rs index 2f19159e..f371bc92 100644 --- a/src/obikphylo/src/siblings/family_overlap.rs +++ b/src/obikphylo/src/siblings/family_overlap.rs @@ -23,6 +23,7 @@ use std::io; use std::path::Path; use memmap2::Mmap; +use obicompactvec::{building_path, publish_built}; use crate::siblings::helpers::{is_carrier, triangle_index}; @@ -166,7 +167,8 @@ impl FamilyOverlapAccumulator { for &c in &self.counts { buf.extend_from_slice(&c.to_le_bytes()); } - std::fs::write(path, buf) + std::fs::write(building_path(path), buf)?; + publish_built(path) } } diff --git a/src/obikphylo/src/siblings/family_scalar_annex.rs b/src/obikphylo/src/siblings/family_scalar_annex.rs index c5679f9f..a61fe30b 100644 --- a/src/obikphylo/src/siblings/family_scalar_annex.rs +++ b/src/obikphylo/src/siblings/family_scalar_annex.rs @@ -20,9 +20,10 @@ use std::fs::{File, OpenOptions}; use std::io; -use std::path::Path; +use std::path::{Path, PathBuf}; use memmap2::{Mmap, MmapMut}; +use obicompactvec::{building_path, publish_built}; // Header: magic(4) + flags(4) + n(8) = 16 bytes. Data (4 bytes/entry) follows. // `flags` bit 0 = `no_ambiguity` (the counting convention the values were @@ -125,6 +126,7 @@ impl FamilyScalarAnnex { pub struct FamilyScalarAnnexBuilder { mmap: MmapMut, + path: PathBuf, } impl FamilyScalarAnnexBuilder { @@ -139,7 +141,7 @@ impl FamilyScalarAnnexBuilder { .write(true) .create(true) .truncate(true) - .open(path)?; + .open(building_path(path))?; file.set_len(file_size as u64)?; let mut mmap = unsafe { MmapMut::map_mut(&file)? }; mmap[0..4].copy_from_slice(&kind.magic()); @@ -150,7 +152,7 @@ impl FamilyScalarAnnexBuilder { let off = HEADER_SIZE + i * 4; mmap[off..off + 4].copy_from_slice(&SENTINEL.to_le_bytes()); } - Ok(Self { mmap }) + Ok(Self { mmap, path: path.to_path_buf() }) } pub(crate) fn set(&mut self, idx: usize, value: f32) { @@ -159,8 +161,12 @@ impl FamilyScalarAnnexBuilder { self.mmap[off..off + 4].copy_from_slice(&value.to_le_bytes()); } + /// Flushes, then publishes the file under its final name: until then it + /// lives under `building_path`, so a killed build never leaves a file + /// `path.exists()` would take for a finished annex. pub(crate) fn close(self) -> io::Result<()> { - self.mmap.flush() + self.mmap.flush()?; + publish_built(&self.path) } } diff --git a/src/obikphylo/src/siblings/siblingannex.rs b/src/obikphylo/src/siblings/siblingannex.rs index 0ec81d5e..4371e7e0 100644 --- a/src/obikphylo/src/siblings/siblingannex.rs +++ b/src/obikphylo/src/siblings/siblingannex.rs @@ -53,6 +53,7 @@ use std::path::{Path, PathBuf}; use std::sync::atomic::AtomicU16; use memmap2::{Mmap, MmapMut}; +use obicompactvec::{building_path, publish_built}; use obikseq::CanonicalKmer; @@ -283,6 +284,7 @@ impl SiblingAnnex { pub struct SiblingAnnexBuilder { mmap: MmapMut, + path: PathBuf, } impl SiblingAnnexBuilder { @@ -292,14 +294,14 @@ impl SiblingAnnexBuilder { let file_size = HEADER_SIZE + n * 2; let file = OpenOptions::new() .read(true).write(true).create(true).truncate(true) - .open(path)?; + .open(building_path(path))?; file.set_len(file_size as u64)?; let mut mmap = unsafe { MmapMut::map_mut(&file)? }; mmap[0..4].copy_from_slice(&MAGIC); mmap[4..8].copy_from_slice(&[0u8; 4]); mmap[8..16].copy_from_slice(&(n as u64).to_le_bytes()); // Data region left at 0 by `set_len`/mmap — the sentinel value. - Ok(Self { mmap }) + Ok(Self { mmap, path: path.to_path_buf() }) } /// `None` means the slot has not (yet) been computed — see module @@ -336,7 +338,13 @@ impl SiblingAnnexBuilder { self.mmap[off..off + 2].copy_from_slice(&mask.encode().to_le_bytes()); } - pub fn close(self) -> io::Result<()> { self.mmap.flush() } + /// Flushes, then publishes the annex under its final name — until then + /// it lives under `building_path`, so a killed build leaves no + /// `siblings.psib` for a reader to trust. + pub fn close(self) -> io::Result<()> { + self.mmap.flush()?; + publish_built(&self.path) + } } #[cfg(test)]