Implement memory mapping for file construction
Refactors the building process to utilize memory mapping (`memmap2`) for file creation and introduces the necessary infrastructure for publishing built files atomically. This includes expanding the public API surface and cleaning up deprecated architectural components.
This commit is contained in:
1 parent
c226744405
commit
3bd7fd4f41
11 files changed
+135
-27
No files matched your search
@@ -0,0 +1,61 @@
|
||||
# Hot resume and memoization of long computations (discussion, 2026-10-14)
|
||||
|
||||
## Goal (user)
|
||||
|
||||
1. **Hot resume**: a run killed midway (crash, `Ctrl-C`, scheduler walltime) restarts from the last finished unit of work, not from zero.
|
||||
2. **No recomputation**: a long function called again with the same parameters (same command rerun, or another command needing the same intermediate) reuses its stored result.
|
||||
|
||||
Not a goal: bit-identical resumed samples. Sampling uses an OS-seeded RNG on purpose, repeated runs must stay independent draws to measure variance; a resumed run being a *different* valid draw is not a defect.
|
||||
|
||||
## Why it matters: one real run
|
||||
|
||||
`phylo --entropy 1 --pseudo-alignment --n-sites 5000000 --free-loss --no-ambiguity --quartet --pair-balance --family-overlap --sibling-stats` on the 180-genome Primulaceae index, from the stderr of the OAR job (2026-10-13), 256 partitions:
|
||||
|
||||
| Stage | Wall |
|
||||
|---|---|
|
||||
| `sibling_stats` (full scan, `Selection::All`) | 8 235 s |
|
||||
| `family_overlap` (sweep 1: overlap annex) | 5 582 s |
|
||||
| `sampled_scan`: sweep 1 again (entropy + quartet annexes) | ≈ 5 347 s |
|
||||
| `sampled_scan`: sweep 2 (pair-weight annex) | ≈ 5 296 s |
|
||||
| `sampled_scan`: the draw itself | ≈ 9 573 s |
|
||||
| `distance` (Jaccard, not asked for; older binary) | 53 s |
|
||||
| Total | 34 085 s (9 h 28) |
|
||||
|
||||
The three annex sweeps cost about the same although their per-family work differs, so the cost is the layer traversal. Part of it was addressed by restricting the matrix pass to selected families (`family_scan.rs`, same session); the sweeps were not fused. Nothing in this run could be resumed or reused: every stage restarts from its beginning.
|
||||
|
||||
## What exists today (verified in the code, 2026-10-14)
|
||||
|
||||
**Persisted per-layer annexes.** Entropy (`entropy*.pent`), quartet (`quartet*.pbiv`), pair weight, sibling annex: one file per layer, built by `ensure_layer_*` and skipped when `path.exists()` (`family_annexes.rs`). A rerun with the same conventions skips every sweep. This is already memoization, keyed by *file name only* (convention in the name, no fingerprint of the inputs; a rebuilt sibling annex deletes everything derived from it, see `pair_coverage_sampling.md`).
|
||||
|
||||
**Not persisted.** The family-overlap accumulator (`FamilyOverlapAccumulator`) is written once at the end of sweep 1 (`sibling_builder.rs`). The draw (`sample_index`) and the `sibling_stats` scan keep their state in memory only.
|
||||
|
||||
**Defect found, then fixed (2026-10-14): a killed annex build left a file that looked finished.** `create_mmap_file` (`obicompactvec/src/mmap_file.rs`) created the file at its *final path* with magic, header and the full length (`set_len`) before any data was written. A kill during a layer left a valid-looking file with unwritten data; the next run saw `path.exists()` and treated the annex as built, silently.
|
||||
|
||||
Reproduced on the 1x testbed (20 genomes, 512 layers): derived annexes deleted, `phylo --family-overlap --quartet --pair-balance --entropy 1 …` killed by `timeout 6`, then rerun. Old binary: 141 finals after the kill, 2 differing from a reference build; the rerun exited 0 and kept both. Fixed binary: kill at 6 s left 499 finals, all identical to the reference, plus 2 `.building` files; the rerun completed 1 537 files identical to the reference, no `.building` left. (Kill at 3 s: nothing published, nothing broken.)
|
||||
|
||||
**Fix.** `obicompactvec::building_path` / `publish_built`: a file is built under `<name>.building` and moved to its final name (same-directory `rename`, atomic) by the builder's `close()`, after the flush. Applied to the builders that go through `create_mmap_file` or write their own mmap on the annex path: `PersistentBitVecBuilder` (all constructors; quartet annex), `PersistentFixedIntVecBuilder`, `PersistentRankSelectBitVecBuilder`, `FamilyScalarAnnexBuilder` (entropy, pair weight), `SiblingAnnexBuilder` (`siblings.psib`) and `FamilyOverlapAccumulator::write` (overlap matrix). A leftover `.building` is truncated by the next build of that file. A build that is dropped without `close()` no longer produces the final file (no caller did; workspace tests pass). Also reordered in `--sibling-annex`: derived annexes are now removed *before* the new sibling annex is built and published, so a kill between the two leaves the old annex with no derived files, never a new annex next to files computed against the old numbering.
|
||||
|
||||
**Not covered.** The builders outside that path still write at their final name: `PersistentCompactIntVecBuilder`, the int/sparse matrix builders, `EliasFanoBuilder`, `TempBitVecBuilder`/`TempCompactIntVecBuilder` (temporary by nature) and the index-construction files that use them. The same reasoning applies to them as soon as a stage built on them has to be resumed; not audited. The in-memory overlap accumulator is still lost on a kill (a sweep restarts), now without leaving a corrupt file.
|
||||
|
||||
**`obiksession` crate** (`src/obiksession/`): `Session` (named directory, opaque parameter blob checked on open, whole-lifetime `DirLock`, artifacts checksummed with xxh3 and mmap-restored) and `Memoize` (`get_or_compute[_with_key]`, `invalidate`, `cached_keys`, `MemoKey` from `Debug` + xxh3, the `session_memoize!` macro). Compiles, 15 tests pass, **no caller**: `obikmer` lists the dependency without using it. `phylo --session` exits with an error (`cmd/phylo/mod.rs:271`) since `session_cache.rs` (the consumer tied to `PairwiseTally`/`PartitionDispersion`/`SnpAlignment`) was removed. `sample_index(skip_layers)` remains as a seam, never called with a non-zero value.
|
||||
|
||||
**Previous design and why it missed the goal** (`theory/evolutionary_distances.md`, section "Explicit `--session`"): one session per *exact* parameter tuple (changing `--n-sites` made another session), artifacts specific to the sampling result, resume only inside the draw (checkpoint every 8 layers or 30 s). Reuse across commands existed only for the sample. An exhaustive-sample cache from which smaller samples would be drawn was proposed and never done. User judgement: this version did not answer the objective.
|
||||
|
||||
## Proposed direction (not implemented; to be validated)
|
||||
|
||||
Start from the goal, not from the existing crate.
|
||||
|
||||
1. **Unit of work = one `(partition, layer)` step that already produces files.** The annex builds, the draw per layer and the scans all iterate that way (`for part … for layer …`). Resume means "skip the units whose output is complete", exactly what `exists()` approximates today.
|
||||
2. **Complete-or-absent outputs.** Done for the annex path (see the fix above); the remaining builders are to be covered as their stages become resumable. Prerequisite for everything below. The mechanism lives in `obicompactvec` (shared lower-level API), not in each consumer.
|
||||
3. **Key = function identity + parameters + fingerprint of the inputs.** A result is reusable iff the function, every parameter that influences it and the identity of its inputs (index content, upstream artifacts) match. Today the key is a file name plus an invalidation convention. The fingerprint requirement is the cost: what identifies an index version cheaply is to be decided (meta file, rebuild counter, hash of annex headers).
|
||||
4. **The cache holds results of long functions, not domain objects.** The crate must not know `PairwiseTally`. Callers serialize their own values (`rkyv` was already chosen for that). Whether `Memoize`'s API fits is unproven: it has no caller, no documentation and was written against the earlier design.
|
||||
5. **Granularity matters more than the API.** A cached 9 h result helps only the exact rerun; a per-layer cache also helps a changed parameter that leaves earlier layers untouched. The overlap accumulator (end-of-sweep write) would need a per-layer partial (a sum of per-layer matrices, mergeable) to be resumable.
|
||||
6. **Fuse before caching.** Three sweeps over the same layers where two suffice (overlap first, then entropy + quartet + pair weight together, since the pair weight needs only the finished overlap matrix) is a cost no cache removes on a first run; to do independently.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Fingerprint of an index version: which cheap, stable identity?
|
||||
- Overlap accumulator resume: per-layer partial matrices on disk (size n² × layers) or a checkpoint every N layers?
|
||||
- `Memoize`/`Session`: keep (adapt to the points above), rewrite, or delete the crate until a caller exists (the "existing code is a hypothesis" rule applies; today it has no user).
|
||||
- Scope of the next step: atomic writes are done for the annex path; extend to the other builders, or go to the cache.
|
||||
- Draw resume: a layer's draw is independent of the others except through the quotas, which depend on every layer's mass; quotas can be recomputed from annexes on resume (cheap), so only the per-layer survivors need storing.
|
||||
@@ -64,6 +64,7 @@ nav:
|
||||
- Sequences: architecture/sequences/invariant.md
|
||||
- Kmer index: architecture/index_architecture.md
|
||||
- Sibling annex: architecture/siblings.md
|
||||
- Hot resume and memoization: architecture/hot_resume_memoization.md
|
||||
- NUMA-aware worker pools: architecture/numa_worker_pools.md
|
||||
- NUMA-aware partition runner: architecture/numa_partition_runner.md
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ use std::path::{Path, PathBuf};
|
||||
|
||||
use memmap2::{Mmap, MmapMut};
|
||||
|
||||
use crate::mmap_file::{create_mmap_file, open_mmap_file};
|
||||
use crate::mmap_file::{building_path, create_mmap_file, open_mmap_file, publish_built};
|
||||
use crate::reader::PersistentCompactIntVec;
|
||||
use crate::views::{BitSliceIter, BitSliceView, IntSliceView};
|
||||
|
||||
@@ -206,7 +206,7 @@ impl PersistentBitVecBuilder {
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.open(path)?;
|
||||
.open(building_path(path))?;
|
||||
file.write_all(&MAGIC)?;
|
||||
file.write_all(&[0u8; 4])?;
|
||||
file.write_all(&(n as u64).to_le_bytes())?;
|
||||
@@ -229,8 +229,8 @@ impl PersistentBitVecBuilder {
|
||||
}
|
||||
|
||||
pub fn build_from(source: &PersistentBitVec, path: &Path) -> io::Result<Self> {
|
||||
fs::copy(source.path(), path)?;
|
||||
let file = OpenOptions::new().read(true).write(true).open(path)?;
|
||||
fs::copy(source.path(), building_path(path))?;
|
||||
let file = OpenOptions::new().read(true).write(true).open(building_path(path))?;
|
||||
let mmap = unsafe { MmapMut::map_mut(&file)? };
|
||||
let n = source.len();
|
||||
Ok(Self {
|
||||
@@ -441,7 +441,8 @@ impl PersistentBitVecBuilder {
|
||||
}
|
||||
|
||||
pub fn close(self) -> io::Result<()> {
|
||||
self.mmap.flush()
|
||||
self.mmap.flush()?;
|
||||
publish_built(&self.path)
|
||||
}
|
||||
|
||||
pub fn finish(self) -> io::Result<PersistentBitVec> {
|
||||
|
||||
@@ -15,7 +15,7 @@ use std::path::{Path, PathBuf};
|
||||
|
||||
use memmap2::{Mmap, MmapMut};
|
||||
|
||||
use crate::mmap_file::{create_mmap_file, open_mmap_file};
|
||||
use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built};
|
||||
|
||||
const MAGIC: [u8; 4] = *b"PFIV";
|
||||
|
||||
@@ -178,7 +178,8 @@ impl PersistentFixedIntVecBuilder {
|
||||
}
|
||||
|
||||
pub fn close(self) -> io::Result<()> {
|
||||
self.mmap.flush()
|
||||
self.mmap.flush()?;
|
||||
publish_built(&self.path)
|
||||
}
|
||||
|
||||
pub fn finish(self) -> io::Result<PersistentFixedIntVec> {
|
||||
|
||||
@@ -35,6 +35,7 @@ pub use intmatrix::{
|
||||
};
|
||||
|
||||
pub use layer_meta::LayerMeta;
|
||||
pub use mmap_file::{building_path, publish_built};
|
||||
pub use matrix_builder::{ColBuilder, MatrixBuilder};
|
||||
pub use rankselect::{PersistentRankSelectBitVec, PersistentRankSelectBitVecBuilder};
|
||||
pub use reader::{Iter as CompactIntVecIter, PersistentCompactIntVec};
|
||||
|
||||
@@ -10,13 +10,35 @@
|
||||
//! `block_ranks`) and its own higher-level API — those genuinely differ
|
||||
//! between formats and aren't forced into a shared shape.
|
||||
|
||||
use std::fs::{File, OpenOptions};
|
||||
use std::fs::{self, File, OpenOptions};
|
||||
use std::io::{self, Seek, SeekFrom, Write as _};
|
||||
use std::path::Path;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use memmap2::{Mmap, MmapMut};
|
||||
|
||||
/// Creates (or truncates) `path`, writes the standard header — `magic` +
|
||||
/// Suffix of the name a file lives under while it is being built.
|
||||
const BUILDING_SUFFIX: &str = ".building";
|
||||
|
||||
/// The name `path` is built under: a file only appears at `path` itself once
|
||||
/// complete ([`publish_built`]). A process killed mid-build leaves at most a
|
||||
/// `.building` file, never a `path` with a valid header and unwritten data
|
||||
/// that `path.exists()` would take for a finished one — what every
|
||||
/// "skip it, it already exists" caller (per-layer annexes) relies on.
|
||||
pub fn building_path(path: &Path) -> PathBuf {
|
||||
let mut name = path.as_os_str().to_owned();
|
||||
name.push(BUILDING_SUFFIX);
|
||||
PathBuf::from(name)
|
||||
}
|
||||
|
||||
/// Atomically moves the file built under [`building_path`]`(path)` to `path`
|
||||
/// (same directory, so a plain `rename`), replacing any previous one. To be
|
||||
/// called once the data is complete and flushed.
|
||||
pub fn publish_built(path: &Path) -> io::Result<()> {
|
||||
fs::rename(building_path(path), path)
|
||||
}
|
||||
|
||||
/// Creates (or truncates) [`building_path`]`(path)` — the caller publishes it
|
||||
/// with [`publish_built`] once complete — writes the standard header — `magic` +
|
||||
/// `pad` (4 bytes right after `magic`; usually all-zero, but
|
||||
/// `PersistentFixedIntVec` repurposes byte 4 to store its `width`) +
|
||||
/// `n as u64`, followed by `extra_header_zeros` more zero bytes for a
|
||||
@@ -37,7 +59,7 @@ pub(crate) fn create_mmap_file(
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.open(path)?;
|
||||
.open(building_path(path))?;
|
||||
file.write_all(&magic)?;
|
||||
file.write_all(&pad)?;
|
||||
file.write_all(&(n as u64).to_le_bytes())?;
|
||||
|
||||
@@ -21,7 +21,7 @@ use std::path::{Path, PathBuf};
|
||||
use common_traits::SelectInWord;
|
||||
use memmap2::{Mmap, MmapMut};
|
||||
|
||||
use crate::mmap_file::{create_mmap_file, open_mmap_file};
|
||||
use crate::mmap_file::{create_mmap_file, open_mmap_file, publish_built};
|
||||
|
||||
const MAGIC: [u8; 4] = *b"PRSB";
|
||||
|
||||
@@ -268,7 +268,8 @@ impl PersistentRankSelectBitVecBuilder {
|
||||
let block_off = HEADER_SIZE + bits_bytes(self.n);
|
||||
let block_bytes = u64_slice_to_le_bytes(&block_ranks);
|
||||
self.mmap[block_off..block_off + block_bytes.len()].copy_from_slice(&block_bytes);
|
||||
self.mmap.flush()
|
||||
self.mmap.flush()?;
|
||||
publish_built(&self.path)
|
||||
}
|
||||
|
||||
pub fn finish(self) -> io::Result<PersistentRankSelectBitVec> {
|
||||
|
||||
@@ -129,16 +129,19 @@ impl SiblingExt for IndexCache {
|
||||
let layer = self
|
||||
.get_layer(part, l)
|
||||
.expect("cache-consistent: n_layer(part) already bounds l");
|
||||
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
||||
|
||||
// Invalidate this layer's entropy and quartet annexes, if
|
||||
// any — they were built against the sibling annex's
|
||||
// *previous* `family_idx` numbering
|
||||
// (`non_monomorphic_reindex_layer` reads the sibling
|
||||
// annex), now stale. `ensure_family_annexes` below only
|
||||
// builds what's *missing*, so the old files must be removed
|
||||
// here for them to be rebuilt rather than silently left
|
||||
// stale.
|
||||
// annex), and are stale once the new one exists.
|
||||
// `ensure_family_annexes` below only builds what's
|
||||
// *missing*, so the old files must be removed here for
|
||||
// them to be rebuilt rather than silently left stale.
|
||||
// Removed *before* the new annex is built and published
|
||||
// (atomically, at its end): a build killed in between
|
||||
// leaves the previous annex with no derived files — still
|
||||
// consistent, rebuilt on demand — never a new annex next to
|
||||
// derived files computed against the old numbering.
|
||||
for name in ScalarKind::Entropy
|
||||
.file_names()
|
||||
.into_iter()
|
||||
@@ -150,6 +153,7 @@ impl SiblingExt for IndexCache {
|
||||
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
|
||||
}
|
||||
}
|
||||
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
||||
}
|
||||
total_slots += part_slots;
|
||||
pb.inc(1);
|
||||
|
||||
@@ -23,6 +23,7 @@ use std::io;
|
||||
use std::path::Path;
|
||||
|
||||
use memmap2::Mmap;
|
||||
use obicompactvec::{building_path, publish_built};
|
||||
|
||||
use crate::siblings::helpers::{is_carrier, triangle_index};
|
||||
|
||||
@@ -166,7 +167,8 @@ impl FamilyOverlapAccumulator {
|
||||
for &c in &self.counts {
|
||||
buf.extend_from_slice(&c.to_le_bytes());
|
||||
}
|
||||
std::fs::write(path, buf)
|
||||
std::fs::write(building_path(path), buf)?;
|
||||
publish_built(path)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -20,9 +20,10 @@
|
||||
|
||||
use std::fs::{File, OpenOptions};
|
||||
use std::io;
|
||||
use std::path::Path;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use memmap2::{Mmap, MmapMut};
|
||||
use obicompactvec::{building_path, publish_built};
|
||||
|
||||
// Header: magic(4) + flags(4) + n(8) = 16 bytes. Data (4 bytes/entry) follows.
|
||||
// `flags` bit 0 = `no_ambiguity` (the counting convention the values were
|
||||
@@ -125,6 +126,7 @@ impl FamilyScalarAnnex {
|
||||
|
||||
pub struct FamilyScalarAnnexBuilder {
|
||||
mmap: MmapMut,
|
||||
path: PathBuf,
|
||||
}
|
||||
|
||||
impl FamilyScalarAnnexBuilder {
|
||||
@@ -139,7 +141,7 @@ impl FamilyScalarAnnexBuilder {
|
||||
.write(true)
|
||||
.create(true)
|
||||
.truncate(true)
|
||||
.open(path)?;
|
||||
.open(building_path(path))?;
|
||||
file.set_len(file_size as u64)?;
|
||||
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
|
||||
mmap[0..4].copy_from_slice(&kind.magic());
|
||||
@@ -150,7 +152,7 @@ impl FamilyScalarAnnexBuilder {
|
||||
let off = HEADER_SIZE + i * 4;
|
||||
mmap[off..off + 4].copy_from_slice(&SENTINEL.to_le_bytes());
|
||||
}
|
||||
Ok(Self { mmap })
|
||||
Ok(Self { mmap, path: path.to_path_buf() })
|
||||
}
|
||||
|
||||
pub(crate) fn set(&mut self, idx: usize, value: f32) {
|
||||
@@ -159,8 +161,12 @@ impl FamilyScalarAnnexBuilder {
|
||||
self.mmap[off..off + 4].copy_from_slice(&value.to_le_bytes());
|
||||
}
|
||||
|
||||
/// Flushes, then publishes the file under its final name: until then it
|
||||
/// lives under `building_path`, so a killed build never leaves a file
|
||||
/// `path.exists()` would take for a finished annex.
|
||||
pub(crate) fn close(self) -> io::Result<()> {
|
||||
self.mmap.flush()
|
||||
self.mmap.flush()?;
|
||||
publish_built(&self.path)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -53,6 +53,7 @@ use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::AtomicU16;
|
||||
|
||||
use memmap2::{Mmap, MmapMut};
|
||||
use obicompactvec::{building_path, publish_built};
|
||||
|
||||
use obikseq::CanonicalKmer;
|
||||
|
||||
@@ -283,6 +284,7 @@ impl SiblingAnnex {
|
||||
|
||||
pub struct SiblingAnnexBuilder {
|
||||
mmap: MmapMut,
|
||||
path: PathBuf,
|
||||
}
|
||||
|
||||
impl SiblingAnnexBuilder {
|
||||
@@ -292,14 +294,14 @@ impl SiblingAnnexBuilder {
|
||||
let file_size = HEADER_SIZE + n * 2;
|
||||
let file = OpenOptions::new()
|
||||
.read(true).write(true).create(true).truncate(true)
|
||||
.open(path)?;
|
||||
.open(building_path(path))?;
|
||||
file.set_len(file_size as u64)?;
|
||||
let mut mmap = unsafe { MmapMut::map_mut(&file)? };
|
||||
mmap[0..4].copy_from_slice(&MAGIC);
|
||||
mmap[4..8].copy_from_slice(&[0u8; 4]);
|
||||
mmap[8..16].copy_from_slice(&(n as u64).to_le_bytes());
|
||||
// Data region left at 0 by `set_len`/mmap — the sentinel value.
|
||||
Ok(Self { mmap })
|
||||
Ok(Self { mmap, path: path.to_path_buf() })
|
||||
}
|
||||
|
||||
/// `None` means the slot has not (yet) been computed — see module
|
||||
@@ -336,7 +338,13 @@ impl SiblingAnnexBuilder {
|
||||
self.mmap[off..off + 2].copy_from_slice(&mask.encode().to_le_bytes());
|
||||
}
|
||||
|
||||
pub fn close(self) -> io::Result<()> { self.mmap.flush() }
|
||||
/// Flushes, then publishes the annex under its final name — until then
|
||||
/// it lives under `building_path`, so a killed build leaves no
|
||||
/// `siblings.psib` for a reader to trust.
|
||||
pub fn close(self) -> io::Result<()> {
|
||||
self.mmap.flush()?;
|
||||
publish_built(&self.path)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
||||
Reference in new issue
Block a user