Push wpvskkyozppn #78

Merged
coissac merged 15 commits from push-wpvskkyozppn into main 2026-10-15 10:38:59 +00:00
11 changed files with 135 additions and 27 deletions
Showing only changes of commit 3bd7fd4f41 - Show all commits

No files matched your search

@@ -0,0 +1,61 @@
# Hot resume and memoization of long computations (discussion, 2026-10-14)
## Goal (user)
1. **Hot resume**: a run killed midway (crash, `Ctrl-C`, scheduler walltime) restarts from the last finished unit of work, not from zero.
2. **No recomputation**: a long function called again with the same parameters (same command rerun, or another command needing the same intermediate) reuses its stored result.
Not a goal: bit-identical resumed samples. Sampling uses an OS-seeded RNG on purpose, repeated runs must stay independent draws to measure variance; a resumed run being a *different* valid draw is not a defect.
## Why it matters: one real run
`phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance --family-overlap --sibling-stats` on the 180-genome Primulaceae index, from the stderr of the OAR job (2026-10-13), 256 partitions:
| Stage | Wall |
|---|---|
| `sibling_stats` (full scan, `Selection::All`) | 8 235 s |
| `family_overlap` (sweep 1: overlap annex) | 5 582 s |
| `sampled_scan`: sweep 1 again (entropy + quartet annexes) | ≈ 5 347 s |
| `sampled_scan`: sweep 2 (pair-weight annex) | ≈ 5 296 s |
| `sampled_scan`: the draw itself | ≈ 9 573 s |
| `distance` (Jaccard, not asked for; older binary) | 53 s |
| Total | 34 085 s (9 h 28) |
The three annex sweeps cost about the same although their per-family work differs, so the cost is the layer traversal. Part of it was addressed by restricting the matrix pass to selected families (`family_scan.rs`, same session); the sweeps were not fused. Nothing in this run could be resumed or reused: every stage restarts from its beginning.
## What exists today (verified in the code, 2026-10-14)
**Persisted per-layer annexes.** Entropy (`entropy*.pent`), quartet (`quartet*.pbiv`), pair weight, sibling annex: one file per layer, built by `ensure_layer_*` and skipped when `path.exists()` (`family_annexes.rs`). A rerun with the same conventions skips every sweep. This is already memoization, keyed by *file name only* (convention in the name, no fingerprint of the inputs; a rebuilt sibling annex deletes everything derived from it, see `pair_coverage_sampling.md`).
**Not persisted.** The family-overlap accumulator (`FamilyOverlapAccumulator`) is written once at the end of sweep 1 (`sibling_builder.rs`). The draw (`sample_index`) and the `sibling_stats` scan keep their state in memory only.
**Defect found, then fixed (2026-10-14): a killed annex build left a file that looked finished.** `create_mmap_file` (`obicompactvec/src/mmap_file.rs`) created the file at its *final path* with magic, header and the full length (`set_len`) before any data was written. A kill during a layer left a valid-looking file with unwritten data; the next run saw `path.exists()` and treated the annex as built, silently.
Reproduced on the 1x testbed (20 genomes, 512 layers): derived annexes deleted, `phylo --family-overlap --quartet --pair-balance --entropy 1 …` killed by `timeout 6`, then rerun. Old binary: 141 finals after the kill, 2 differing from a reference build; the rerun exited 0 and kept both. Fixed binary: kill at 6 s left 499 finals, all identical to the reference, plus 2 `.building` files; the rerun completed 1 537 files identical to the reference, no `.building` left. (Kill at 3 s: nothing published, nothing broken.)
**Fix.** `obicompactvec::building_path` / `publish_built`: a file is built under `<name>.building` and moved to its final name (same-directory `rename`, atomic) by the builder's `close()`, after the flush. Applied to the builders that go through `create_mmap_file` or write their own mmap on the annex path: `PersistentBitVecBuilder` (all constructors; quartet annex), `PersistentFixedIntVecBuilder`, `PersistentRankSelectBitVecBuilder`, `FamilyScalarAnnexBuilder` (entropy, pair weight), `SiblingAnnexBuilder` (`siblings.psib`) and `FamilyOverlapAccumulator::write` (overlap matrix). A leftover `.building` is truncated by the next build of that file. A build that is dropped without `close()` no longer produces the final file (no caller did; workspace tests pass). Also reordered in `--sibling-annex`: derived annexes are now removed *before* the new sibling annex is built and published, so a kill between the two leaves the old annex with no derived files, never a new annex next to files computed against the old numbering.
**Not covered.** The builders outside that path still write at their final name: `PersistentCompactIntVecBuilder`, the int/sparse matrix builders, `EliasFanoBuilder`, `TempBitVecBuilder`/`TempCompactIntVecBuilder` (temporary by nature) and the index-construction files that use them. The same reasoning applies to them as soon as a stage built on them has to be resumed; not audited. The in-memory overlap accumulator is still lost on a kill (a sweep restarts), now without leaving a corrupt file.
**`obiksession` crate** (`src/obiksession/`): `Session` (named directory, opaque parameter blob checked on open, whole-lifetime `DirLock`, artifacts checksummed with xxh3 and mmap-restored) and `Memoize` (`get_or_compute[_with_key]`, `invalidate`, `cached_keys`, `MemoKey` from `Debug` + xxh3, the `session_memoize!` macro). Compiles, 15 tests pass, **no caller**: `obikmer` lists the dependency without using it. `phylo --session` exits with an error (`cmd/phylo/mod.rs:271`) since `session_cache.rs` (the consumer tied to `PairwiseTally`/`PartitionDispersion`/`SnpAlignment`) was removed. `sample_index(skip_layers)` remains as a seam, never called with a non-zero value.
**Previous design and why it missed the goal** (`theory/evolutionary_distances.md`, section "Explicit `--session`"): one session per *exact* parameter tuple (changing `--n-sites` made another session), artifacts specific to the sampling result, resume only inside the draw (checkpoint every 8 layers or 30 s). Reuse across commands existed only for the sample. An exhaustive-sample cache from which smaller samples would be drawn was proposed and never done. User judgement: this version did not answer the objective.
## Proposed direction (not implemented; to be validated)
Start from the goal, not from the existing crate.
1. **Unit of work = one `(partition, layer)` step that already produces files.** The annex builds, the draw per layer and the scans all iterate that way (`for part … for layer …`). Resume means "skip the units whose output is complete", exactly what `exists()` approximates today.
2. **Complete-or-absent outputs.** Done for the annex path (see the fix above); the remaining builders are to be covered as their stages become resumable. Prerequisite for everything below. The mechanism lives in `obicompactvec` (shared lower-level API), not in each consumer.
3. **Key = function identity + parameters + fingerprint of the inputs.** A result is reusable iff the function, every parameter that influences it and the identity of its inputs (index content, upstream artifacts) match. Today the key is a file name plus an invalidation convention. The fingerprint requirement is the cost: what identifies an index version cheaply is to be decided (meta file, rebuild counter, hash of annex headers).
4. **The cache holds results of long functions, not domain objects.** The crate must not know `PairwiseTally`. Callers serialize their own values (`rkyv` was already chosen for that). Whether `Memoize`'s API fits is unproven: it has no caller, no documentation and was written against the earlier design.
5. **Granularity matters more than the API.** A cached 9 h result helps only the exact rerun; a per-layer cache also helps a changed parameter that leaves earlier layers untouched. The overlap accumulator (end-of-sweep write) would need a per-layer partial (a sum of per-layer matrices, mergeable) to be resumable.
6. **Fuse before caching.** Three sweeps over the same layers where two suffice (overlap first, then entropy + quartet + pair weight together, since the pair weight needs only the finished overlap matrix) is a cost no cache removes on a first run; to do independently.
## Open questions
- Fingerprint of an index version: which cheap, stable identity?
- Overlap accumulator resume: per-layer partial matrices on disk (size n² × layers) or a checkpoint every N layers?
- `Memoize`/`Session`: keep (adapt to the points above), rewrite, or delete the crate until a caller exists (the "existing code is a hypothesis" rule applies; today it has no user).
- Scope of the next step: atomic writes are done for the annex path; extend to the other builders, or go to the cache.
- Draw resume: a layer's draw is independent of the others except through the quotas, which depend on every layer's mass; quotas can be recomputed from annexes on resume (cheap), so only the per-layer survivors need storing.
+1
View File
@@ -64,6 +64,7 @@ nav:
- Sequences: architecture/sequences/invariant.md
- Kmer index: architecture/index_architecture.md
- Sibling annex: architecture/siblings.md
- Hot resume and memoization: architecture/hot_resume_memoization.md
- NUMA-aware worker pools: architecture/numa_worker_pools.md
- NUMA-aware partition runner: architecture/numa_partition_runner.md
+6 -5
View File
@@ -4,7 +4,7 @@ use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut};
use crate::mmap_file::{create_mmap_file, open_mmap_file};
use crate::mmap_file::{building_path, create_mmap_file, open_mmap_file, publish_built};
use crate::reader::PersistentCompactIntVec;
use crate::views::{BitSliceIter, BitSliceView, IntSliceView};
@@ -206,7 +206,7 @@ impl PersistentBitVecBuilder {
.write(true)
.create(true)
.truncate(true)
.open(path)?;
.open(building_path(path))?;
file.write_all(&MAGIC)?;
file.write_all(&[0u8; 4])?;
file.write_all(&(n as u64).to_le_bytes())?;
@@ -229,8 +229,8 @@ impl PersistentBitVecBuilder {
}
pub fn build_from(source: &PersistentBitVec, path: &Path) -> io::Result<Self> {
fs::copy(source.path(), path)?;
let file = OpenOptions::new().read(true).write(true).open(path)?;
fs::copy(source.path(), building_path(path))?;
let file = OpenOptions::new().read(true).write(true).open(building_path(path))?;
let mmap = unsafe { MmapMut::map_mut(&file)? };
let n = source.len();
Ok(Self {
@@ -441,7 +441,8 @@ impl PersistentBitVecBuilder {
}
pub fn close(self) -> io::Result<()> {
self.mmap.flush()
self.mmap.flush()?;
publish_built(&self.path)
}
pub fn finish(self) -> io::Result<PersistentBitVec> {
+3 -2
View File
@@ -15,7 +15,7 @@ use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut};
use crate::mmap_file::{create_mmap_file, open_mmap_file};
use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built};
const MAGIC: [u8; 4] = *b"PFIV";
@@ -178,7 +178,8 @@ impl PersistentFixedIntVecBuilder {
}
pub fn close(self) -> io::Result<()> {
self.mmap.flush()
self.mmap.flush()?;
publish_built(&self.path)
}
pub fn finish(self) -> io::Result<PersistentFixedIntVec> {
+1
View File
@@ -35,6 +35,7 @@ pub use intmatrix::{
};
pub use layer_meta::LayerMeta;
pub use mmap_file::{building_path, publish_built};
pub use matrix_builder::{ColBuilder, MatrixBuilder};
pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
pub use reader::{Iter as CompactIntVecIter, PersistentCompactIntVec};
+26 -4
View File
@@ -10,13 +10,35 @@
//! `block_ranks`) and its own higher-level API — those genuinely differ
//! between formats and aren't forced into a shared shape.
use std::fs::{File, OpenOptions};
use std::fs::{self, File, OpenOptions};
use std::io::{self, Seek, SeekFrom, Write as _};
use std::path::Path;
use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut};
/// Creates (or truncates) `path`, writes the standard header — `magic` +
/// Suffix of the name a file lives under while it is being built.
const BUILDING_SUFFIX: &str = ".building";
/// The name `path` is built under: a file only appears at `path` itself once
/// complete ([`publish_built`]). A process killed mid-build leaves at most a
/// `.building` file, never a `path` with a valid header and unwritten data
/// that `path.exists()` would take for a finished one — what every
/// "skip it, it already exists" caller (per-layer annexes) relies on.
pub fn building_path(path: &Path) -> PathBuf {
let mut name = path.as_os_str().to_owned();
name.push(BUILDING_SUFFIX);
PathBuf::from(name)
}
/// Atomically moves the file built under [`building_path`]`(path)` to `path`
/// (same directory, so a plain `rename`), replacing any previous one. To be
/// called once the data is complete and flushed.
pub fn publish_built(path: &Path) -> io::Result<()> {
fs::rename(building_path(path), path)
}
/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it
/// with [`publish_built`] once complete — writes the standard header — `magic` +
/// `pad` (4 bytes right after `magic`; usually all-zero, but
/// `PersistentFixedIntVec` repurposes byte 4 to store its `width`) +
/// `n as u64`, followed by `extra_header_zeros` more zero bytes for a
@@ -37,7 +59,7 @@ pub(crate) fn create_mmap_file(
.write(true)
.create(true)
.truncate(true)
.open(path)?;
.open(building_path(path))?;
file.write_all(&magic)?;
file.write_all(&pad)?;
file.write_all(&(n as u64).to_le_bytes())?;
+3 -2
View File
@@ -21,7 +21,7 @@ use std::path::{Path, PathBuf};
use common_traits::SelectInWord;
use memmap2::{Mmap, MmapMut};
use crate::mmap_file::{create_mmap_file, open_mmap_file};
use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built};
const MAGIC: [u8; 4] = *b"PRSB";
@@ -268,7 +268,8 @@ impl PersistentRankSelectBitVecBuilder {
let block_off = HEADER_SIZE + bits_bytes(self.n);
let block_bytes = u64_slice_to_le_bytes(&block_ranks);
self.mmap[block_off..block_off + block_bytes.len()].copy_from_slice(&block_bytes);
self.mmap.flush()
self.mmap.flush()?;
publish_built(&self.path)
}
pub fn finish(self) -> io::Result<PersistentRankSelectBitVec> {
@@ -129,16 +129,19 @@ impl SiblingExt for IndexCache {
let layer = self
.get_layer(part, l)
.expect("cache-consistent: n_layer(part) already bounds l");
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
// Invalidate this layer's entropy and quartet annexes, if
// any — they were built against the sibling annex's
// *previous* `family_idx` numbering
// (`non_monomorphic_reindex_layer` reads the sibling
// annex), now stale. `ensure_family_annexes` below only
// builds what's *missing*, so the old files must be removed
// here for them to be rebuilt rather than silently left
// stale.
// annex), and are stale once the new one exists.
// `ensure_family_annexes` below only builds what's
// *missing*, so the old files must be removed here for
// them to be rebuilt rather than silently left stale.
// Removed *before* the new annex is built and published
// (atomically, at its end): a build killed in between
// leaves the previous annex with no derived files — still
// consistent, rebuilt on demand — never a new annex next to
// derived files computed against the old numbering.
for name in ScalarKind::Entropy
.file_names()
.into_iter()
@@ -150,6 +153,7 @@ impl SiblingExt for IndexCache {
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
}
}
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
}
total_slots += part_slots;
pb.inc(1);
+3 -1
View File
@@ -23,6 +23,7 @@ use std::io;
use std::path::Path;
use memmap2::Mmap;
use obicompactvec::{building_path, publish_built};
use crate::siblings::helpers::{is_carrier, triangle_index};
@@ -166,7 +167,8 @@ impl FamilyOverlapAccumulator {
for &c in &self.counts {
buf.extend_from_slice(&c.to_le_bytes());
}
std::fs::write(path, buf)
std::fs::write(building_path(path), buf)?;
publish_built(path)
}
}
@@ -20,9 +20,10 @@
use std::fs::{File, OpenOptions};
use std::io;
use std::path::Path;
use std::path::{Path, PathBuf};
use memmap2::{Mmap, MmapMut};
use obicompactvec::{building_path, publish_built};
// Header: magic(4) + flags(4) + n(8) = 16 bytes. Data (4 bytes/entry) follows.
// `flags` bit 0 = `no_ambiguity` (the counting convention the values were
@@ -125,6 +126,7 @@ impl FamilyScalarAnnex {
pub struct FamilyScalarAnnexBuilder {
mmap: MmapMut,
path: PathBuf,
}
impl FamilyScalarAnnexBuilder {
@@ -139,7 +141,7 @@ impl FamilyScalarAnnexBuilder {
.write(true)
.create(true)
.truncate(true)
.open(path)?;
.open(building_path(path))?;
file.set_len(file_size as u64)?;
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
mmap[0..4].copy_from_slice(&kind.magic());
@@ -150,7 +152,7 @@ impl FamilyScalarAnnexBuilder {
let off = HEADER_SIZE + i * 4;
mmap[off..off + 4].copy_from_slice(&SENTINEL.to_le_bytes());
}
Ok(Self { mmap })
Ok(Self { mmap, path: path.to_path_buf() })
}
pub(crate) fn set(&mut self, idx: usize, value: f32) {
@@ -159,8 +161,12 @@ impl FamilyScalarAnnexBuilder {
self.mmap[off..off + 4].copy_from_slice(&value.to_le_bytes());
}
/// Flushes, then publishes the file under its final name: until then it
/// lives under `building_path`, so a killed build never leaves a file
/// `path.exists()` would take for a finished annex.
pub(crate) fn close(self) -> io::Result<()> {
self.mmap.flush()
self.mmap.flush()?;
publish_built(&self.path)
}
}
+11 -3
View File
@@ -53,6 +53,7 @@ use std::path::{Path, PathBuf};
use std::sync::atomic::AtomicU16;
use memmap2::{Mmap, MmapMut};
use obicompactvec::{building_path, publish_built};
use obikseq::CanonicalKmer;
@@ -283,6 +284,7 @@ impl SiblingAnnex {
pub struct SiblingAnnexBuilder {
mmap: MmapMut,
path: PathBuf,
}
impl SiblingAnnexBuilder {
@@ -292,14 +294,14 @@ impl SiblingAnnexBuilder {
let file_size = HEADER_SIZE + n * 2;
let file = OpenOptions::new()
.read(true).write(true).create(true).truncate(true)
.open(path)?;
.open(building_path(path))?;
file.set_len(file_size as u64)?;
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
mmap[0..4].copy_from_slice(&MAGIC);
mmap[4..8].copy_from_slice(&[0u8; 4]);
mmap[8..16].copy_from_slice(&(n as u64).to_le_bytes());
// Data region left at 0 by `set_len`/mmap — the sentinel value.
Ok(Self { mmap })
Ok(Self { mmap, path: path.to_path_buf() })
}
/// `None` means the slot has not (yet) been computed — see module
@@ -336,7 +338,13 @@ impl SiblingAnnexBuilder {
self.mmap[off..off + 2].copy_from_slice(&mask.encode().to_le_bytes());
}
pub fn close(self) -> io::Result<()> { self.mmap.flush() }
/// Flushes, then publishes the annex under its final name — until then
/// it lives under `building_path`, so a killed build leaves no
/// `siblings.psib` for a reader to trust.
pub fn close(self) -> io::Result<()> {
self.mmap.flush()?;
publish_built(&self.path)
}
}
#[cfg(test)]