Compare commits

...
4 Commits
Author SHA1 Message Date
coissac ee62b131f9 Merge pull request 'Push ltpzonnnkpqs' (#79) from push-ltpzonnnkpqs into main
Reviewed-on: #79
2026-10-16 14:43:50 +00:00
Eric Coissac 85296a2ef8 Implement directory locking with shared and exclusive modes
Release / create-release (push) Successful in 2m41s
Release / build-linux-x86_64 (push) Successful in 9m17s
Release / build-macos-arm64 (push) Successful in 2m2s
ci.yml / build (pull_request) Successful in 3m48s
Introduces a robust, OS-level directory locking mechanism using LockSlot for managing shared and exclusive access, supporting lock transitions and non-blocking probes. Also sets up the index structure with new builder and index modules.
2026-10-16 16:02:00 +02:00
Eric Coissac 9ac3a37529 Implement explicit directory locking for index operations
Introduces explicit directory locking mechanisms (`DirLock`) to ensure exclusive access during index operations across command stages. This includes modifying index opening logic to acquire locks before proceeding, handling lock acquisition failures by exiting, and implementing safe conditional building of sibling annexes using locking to prevent concurrent corruption.
2026-10-16 16:00:49 +02:00
Eric Coissac 7c7d7615cb Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
2026-10-16 15:59:40 +02:00
25 changed files with 821 additions and 239 deletions

No files matched your search

+28 -1
View File
@@ -329,7 +329,34 @@ Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 /
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment has not been given to IQ-TREE.
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment had not been given to IQ-TREE (done 2026-10-16: see the last measured section).
## Tree-level effect of the combined draw: sCF on three trees (measured 2026-10-16)
Closes the "not measured" item of the previous section (the combined alignment given to IQ-TREE). Three trees, all sCF by `--scfl 100 --cf-verbose`, model `GTR+G4+ASC`:
- **A** = `concord` (12/10): entropy-only draw, 180 genomes, 5 003 414 columns; `-t` + `-m` re-ran a full ML search (59 h).
- **B** = `concord_quartet_balance`: combined draw (`--entropy 1 --quartet --pair-balance`), 180 genomes, 5 000 000 columns. Its log stops during the tree search (14/10), so the command that produced its `cf.*` (16/10) is not in the files; presumably `-te` on the 15/10 treefile as for G (unverified).
- **G** = `scf_good`: same draw as B without the 3 data-poor genomes (*A. pyrenaica* PHA000543, *A. saussurei*, *A. triflora*), 177 genomes. 72 sites became invariant and were removed to satisfy ASC; the search (`concord_quartet_balance_good`, 107 iterations, 6 h 47 on 120 threads) crashed in the sCF step on the FASTA (`Assertion all_prob_const < 1.0`; cause untested, ASC with constant sites is a suspect). sCF were recomputed with `-te <treefile> -s <varsites.phy> -m GTR+G4+ASC --scfl 100`: **no search** (0 iterations, lnL −41 053 676.53 vs −41 053 676.41 from the search), 4 999 729 columns. A used `-t`, B/G a fixed tree: the protocols differ.
| | A | B | G |
|---|---|---|---|
| Branches | 177 | 177 | 174 |
| Mean `sN` | 3 219 | 13 163 | 13 396 |
| Branches with `sN` < 100 / < 1 000 | 13 / 50 | 0 / 5 | 0 / 2 |
| Median sCF | 44.2 | 42.9 | 42.8 |
| Branches with sCF < 40 / ≥ 70 | 74 / 28 | 67 / 26 | 69 / 25 |
| *Primula* and *Androsace* monophyletic | no / no | no / no | yes / yes |
**Sampling effect (A → B, same 180 genomes).** `sN` ×4.5 and the scarcity is gone. Sites do not become more concordant: on the 113 shared bipartitions (of 177) median sCF 48.0 → 47.3 (44 up, 69 down). Robinson-Foulds 128: the draw rearranged the tree deeply. Consistent with the earlier reading that the intra-*Androsace* conflict is not a matter of site count; the number of branches with sCF < 40 % stays around 70.
**Interface nodes.** Of the old nodes 209/211/212/216/217 (`sN` 17 / 226 / 184 / 7 / 7) none exists in G. Node 208 survives with `sN` 190 → 5 218 (sCF 54.5 → 37.5). Six mixed clades remain in G (83 to 72 tips, 50 *Primula* + 22-33 *Androsace*), all with `sN` > 3 000.
**Effect of the 3 genomes (B → G, same draw).** RF 16 (166 of 174 shared), median sCF 43.2 → 43.6, median `sN` unchanged. In B, *A. pyrenaica* and *A. triflora* sit inside *Primula* (a clade of 52 tips = 50 *Primula* + these 2; each has a *Primula* sister, sCF 92 % and 52 %, `sN` 556 and 153); *A. saussurei* stays with *Androsace*. Removing them restores both monophylies and changes nothing else measurably: support for "too little data to place them" (data-poor section), not for contamination, though the latter is not excluded.
**Where A and G disagree (2026-10-16).** On the 177 common tips: 124 shared bipartitions, 50 only in A, 50 only in G. Differing bipartitions are weakly supported in both trees (median sCF 35.7 in A-only, 37.7 in G-only, versus 47.5-48.1 on the shared ones), so the disagreement sits where support was already low. Differences are mostly local: the closest bipartition of the other tree is 1 tip away for 25 of 50 A-only and 21 of 50 G-only splits (≤ 2 tips: 37 and 30); only 8 and 9 are 5+ tips apart. Greedy removal of the tip that lowers RF most: the first (*P. auricula* subsp. *serratifolia*) takes RF from 100 to 86, then about 2 per tip (*A. maxima*, *P. stricta* var. *obesior*, then ordinary tips); the top three are data-poor genomes. A bipartition separates a data-poor genome (the 5 left after the 3 removals) in 1 of 124 shared splits but in 23 of 50 A-only and 10 of 50 G-only splits; removing those 5 genomes gives RF 72, against 96.6 (range 90-100, 40 draws) for 5 random tips. So about 28 of the 100 differences trace to data-poor genomes, and the remaining 72 do not. Backbone: A has a clade of 50 *Primula* + 25 *Androsace* (sCF 76, `sN` 453), G one of 50 + 26 (sCF 73, `sN` 6 677): one tip apart. Genuine conflict: a clade of 72 *Androsace* exists in both trees (A: sCF 34, `sN` 17 127; G: sCF 38, `sN` 49 473) with 16 different tips, i.e. a topological disagreement carried by tens of thousands of sites in each, not a scarcity effect; consistent with the intra-*Androsace* conflict above.
**Not established.** Separate contribution of entropy / quartet / pair balance (no ablation); stability of the tree between two independent draws (one draw per setting; A → B shows the topology moves with the draw); whether the quartet-informative restriction biases branch lengths (ASC corrects for variable-only ascertainment, not for requiring two states each in two genomes; tree length 22.5 is not interpretable as substitutions/site).
## Open questions
+1 -1
View File
@@ -48,7 +48,7 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
| Option | Description |
|---|---|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value |
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex, then every annex derived from it (entropy, quartet, pair weight, family overlap, under both `--no-ambiguity` conventions). Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
+1 -1
View File
@@ -1537,7 +1537,7 @@ dependencies = [
[[package]]
name = "obikmer"
version = "1.3.6"
version = "1.4.1"
dependencies = [
"clap",
"csv",
+4 -2
View File
@@ -90,7 +90,7 @@ impl IndexBuilder for KmerIndex {
let idx = KmerIndex {
root_path: output.to_owned(),
meta: Arc::new(meta),
lock: std::sync::Mutex::new(None),
lock: obisys::LockSlot::empty(output),
};
if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other(
@@ -102,7 +102,9 @@ impl IndexBuilder for KmerIndex {
fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> {
let output = output.as_ref();
let idx = KmerIndex::open(output)?;
// `output` is held exclusively by the builder that wrote it: reopen
// without a lock, a locking `open` would wait for that builder for ever.
let idx = KmerIndex::open_unlocked(output)?;
idx.meta.mark_indexed().map_err(OKIError::Io)?;
let t_pack = Stage::start("pack");
idx.pack_matrices(sparse)?;
+78 -56
View File
@@ -1,6 +1,6 @@
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use std::sync::Arc;
use crate::layer::KmerLayer;
use crate::partition::KmerPartition;
@@ -15,7 +15,7 @@ use crate::index::meta::{GenomeInfo, IndexConfig, IndexMeta};
pub struct KmerIndex {
pub(crate) root_path: PathBuf,
pub(crate) meta: Arc<IndexMeta>,
pub(crate) lock: Mutex<Option<obisys::DirLock>>,
pub(crate) lock: obisys::LockSlot,
}
impl KmerIndex {
@@ -37,11 +37,8 @@ impl KmerIndex {
set_m(config.minimizer_size);
let genomes = genome_info.into_iter().collect();
let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?;
let idx = Self {
root_path,
meta: Arc::new(meta),
lock: Mutex::new(None),
};
let lock = obisys::LockSlot::empty(&root_path);
let idx = Self { root_path, meta: Arc::new(meta), lock };
if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock right after create",
@@ -50,67 +47,92 @@ impl KmerIndex {
Ok(idx)
}
/// Open an existing index for reading — doesn't touch the directory
/// lock at all: concurrent readers (`query`, `dump`, a `merge` source,
/// ...) must never block on each other. A caller that needs exclusive
/// access (resuming an interrupted build) uses
/// [`open_lock`](Self::open_lock) instead; either way, `self.lock`
/// starts empty — [`test_lock`](Self::test_lock)/
/// [`acquire_lock`](Self::acquire_lock) are opt-in from here.
/// Open an existing index for reading, holding a **shared** directory
/// lock for as long as the returned index lives: any number of readers
/// coexist, but a writer ([`open_lock`](Self::open_lock),
/// [`with_lock`](Self::with_lock)) waits for them all, and readers wait
/// for a writer at work — a reader never sees files change under it.
/// The lock is taken before the metadata is read.
///
/// Never open, from a process that holds the write lock on a directory
/// through another handle, that same directory again with `open`: the
/// new handle would wait for the first for ever.
pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let root_path = path.as_ref().to_owned();
let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
set_k(meta.config.kmer_size);
set_m(meta.config.minimizer_size);
Ok(Self {
root_path,
meta: Arc::new(meta),
lock: Mutex::new(None),
})
Self::open_with(path.as_ref(), obisys::LockMode::Shared)
}
/// `open`, then hold the directory lock exclusively — for a caller
/// that resumes a write pipeline (e.g. `obikmer index` re-run on an
/// existing, not-yet-`Indexed` directory) rather than reading
/// concurrently with others.
/// `open`, holding the directory lock **exclusively** for as long as the
/// returned index lives — **the** way for a command that modifies an
/// existing index in place (`convert`, `pack`, `annotate`, `utils
/// rename`, resuming an interrupted build, ...) to claim it. The lock is
/// taken *before* the metadata is read, so what the caller reads is what
/// it will modify.
///
/// A command that only reads, or that writes a *new* index elsewhere
/// (`merge`, `filter`, `select`: exclusive on their destination, shared
/// on their sources), uses [`open`](Self::open). Library code that has to
/// write into an index it was handed (annexes built on demand) wraps that
/// write in [`with_lock`](Self::with_lock).
pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
let idx = Self::open(path)?;
if !idx.acquire_lock() {
return Err(OKIError::Io(std::io::Error::other(
"failed to acquire index lock",
)));
Self::open_with(path.as_ref(), obisys::LockMode::Exclusive)
}
fn open_with(path: &Path, mode: obisys::LockMode) -> OKIResult<Self> {
// A directory that is not an index is reported by `open_unlocked`,
// without a lock file being created in it.
if !Self::is_an_index(path) {
return Self::open_unlocked(path);
}
let lock = match mode {
obisys::LockMode::Shared => obisys::LockSlot::shared(path),
obisys::LockMode::Exclusive => obisys::LockSlot::exclusive(path),
}
.map_err(OKIError::Io)?;
let mut idx = Self::open_unlocked(path)?;
idx.lock = lock;
Ok(idx)
}
/// Block until this index's directory lock is acquired, then hold it
/// (released on [`release_lock`](Self::release_lock) or when `self` is
/// dropped). Returns `false` only on a genuine I/O failure acquiring
/// the lock file itself — contention blocks rather than failing.
pub fn acquire_lock(&self) -> bool {
match obisys::DirLock::acquire(&self.root_path) {
Ok(lock) => {
*self.lock.lock().unwrap() = Some(lock);
true
}
Err(_) => false,
}
/// `open` without taking any lock — for a caller that already holds the
/// directory through another handle (`finalize_indexed`, reopening the
/// index its own builder holds exclusively).
pub(crate) fn open_unlocked(path: &Path) -> OKIResult<Self> {
let root_path = path.to_owned();
let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
set_k(meta.config.kmer_size);
set_m(meta.config.minimizer_size);
let lock = obisys::LockSlot::empty(&root_path);
Ok(Self { root_path, meta: Arc::new(meta), lock })
}
/// Take the exclusive directory lock and hold it until `self` is dropped
/// — for an index just created, which is about to be written to.
/// Idempotent. Returns `false` only on a genuine I/O failure.
pub(crate) fn acquire_lock(&self) -> bool {
self.lock.acquire_exclusive().is_ok()
}
/// Whether this index object currently holds its directory lock
/// exclusively (a shared lock is not "write access").
pub fn holds_write_lock(&self) -> bool {
self.lock.holds_write_lock()
}
/// Run `f` with the directory lock held **exclusively** — promoting the
/// shared lock a handle from [`open`](Self::open) holds, and restoring it
/// afterwards; see [`obisys::LockSlot::with_exclusive`], including what
/// `f` must re-check. Library code that writes into an index it was
/// handed (annexes built on demand) wraps that write in this.
pub fn with_lock<T>(&self, f: impl FnOnce() -> OKIResult<T>) -> OKIResult<T> {
self.lock.with_exclusive(f)
}
/// Non-blocking probe of this index's directory lock — doesn't retain
/// it either way: `true` if the lock was free (briefly acquired, then
/// immediately released), `false` if another process currently holds
/// it. For a read-only caller that wants to warn about a build in
/// progress without itself becoming exclusive; use
/// [`acquire_lock`](Self::acquire_lock)/[`open_lock`](Self::open_lock)
/// to actually hold it.
/// anything: `true` if no process currently holds it exclusively, `false`
/// if one does (this process included). For a caller that wants to warn
/// about a write in progress without waiting for it.
pub fn test_lock(&self) -> bool {
matches!(obisys::DirLock::try_acquire(&self.root_path), Ok(Some(_)))
}
/// Release this index's directory lock, if held. A no-op if it isn't.
pub fn release_lock(&self) {
*self.lock.lock().unwrap() = None;
self.lock.is_free_of_writer()
}
/// Return `true` if `path` points to a valid obikmer index directory.
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "obikmer"
version = "1.4.0"
version = "1.4.1"
edition = "2024"
[[bin]]
+5 -1
View File
@@ -93,7 +93,11 @@ fn run_annotate(args: &AnnotateArgs) {
}
};
let idx = open_index(&args.index);
// Rewrites the index metadata in place: exclusive for the whole command.
let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}");
std::process::exit(1);
});
let mut genomes = idx.meta().genomes().unwrap_or_else(|e| {
eprintln!("error reading index metadata: {e}");
+2 -8
View File
@@ -70,14 +70,8 @@ struct ModeArgs {
}
pub fn run(args: ConvertArgs) {
// Modifies the index in place; acquired before opening so a concurrent
// writer can't slip in between the open and the convert below.
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = KmerIndex::open(&args.index).unwrap_or_else(|e| {
// Modifies the index in place: exclusive for the whole command.
let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}");
std::process::exit(1);
});
+2 -8
View File
@@ -26,14 +26,8 @@ pub struct PackArgs {
}
pub fn run(args: PackArgs) {
// Modifies the index in place; acquired before opening so a concurrent
// writer can't slip in between the open and the pack below.
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
let idx = Arc::new(KmerIndex::open(&args.index).unwrap_or_else(|e| {
// Modifies the index in place: exclusive for the whole command.
let idx = Arc::new(KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
eprintln!("error opening index: {e}");
std::process::exit(1);
}));
+3 -7
View File
@@ -78,13 +78,9 @@ pub fn run(args: PhyloArgs) {
// Meant to be (re)built routinely, ahead of any SNP-family distance
// computation that will later consume it.
if args.sibling_annex {
// Writes into the index directory — hold an exclusive lock for the
// duration so a second, concurrent `--sibling-annex` run on the
// same index can't corrupt these writes (see obisys::DirLock).
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
eprintln!("error locking index directory {}: {e}", args.index.display());
std::process::exit(1);
});
// Writes into the index directory; `build_sibling_annex` holds the
// index's exclusive lock for the duration (`KmerIndex::with_lock`),
// so a concurrent run on the same index can't corrupt these writes.
info!("building sibling-count/minorant annex");
let t = Stage::start("sibling_annex");
cache.build_sibling_annex().unwrap_or_else(|e| {
+3 -1
View File
@@ -53,7 +53,9 @@ pub(super) fn run_bits_per_kmer(index_path: &PathBuf) {
pub(super) fn run_rename(index_path: &PathBuf, spec: &str) {
let (old_label, new_label) = parse_rename_spec(spec);
let idx = KmerIndex::open(index_path).unwrap_or_else(|e| {
// Rewrites the index metadata and renames a spectrum file in place:
// exclusive for the whole command.
let idx = KmerIndex::open_lock(index_path).unwrap_or_else(|e| {
eprintln!("error opening index: {e}");
std::process::exit(1);
});
@@ -17,6 +17,7 @@ use super::entropy::{family_entropy, family_entropy_4};
use super::family_scan::{Selection, scan_layer_families};
use super::is_fast_mode;
use super::minorant_selection::non_monomorphic_reindex_layer;
use crate::siblings::extensions::SiblingBuilder;
/// One variable family, as `--shannon` reports it.
pub(crate) struct FamilyEntropyRow {
@@ -45,6 +46,7 @@ pub(crate) fn scan_family_entropies(
cache: &IndexCache,
mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>,
) -> OKIResult<()> {
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache);
let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions");
@@ -22,7 +22,9 @@ use super::masking::is_quartet_informative;
use super::minorant_selection::non_monomorphic_reindex_layer;
use super::pair_weight::pair_balance_weight;
use crate::siblings::quartet_annex::quartet_annex_file_name;
use crate::siblings::{FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind};
use crate::siblings::{
FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind, family_overlap_file_name,
};
/// Which annexes a caller needs, each listed once per `--no-ambiguity`
/// counting convention (`false` = genomes carrying several members count as a
@@ -50,6 +52,24 @@ impl AnnexRequest {
dedup(self.overlap.iter().chain(&self.pair_weight).copied())
}
/// Whether every annex this request asks for already exists — the same
/// per-file existence test the builders apply, so `true` means they would
/// find nothing to build and no lock is needed. Cheap (no file is opened).
pub(crate) fn is_satisfied(&self, cache: &IndexCache) -> bool {
let overlap = self
.overlap_conventions()
.into_iter()
.all(|nb| cache.index().dir().join(family_overlap_file_name(nb)).exists());
let pair_weight = self.pair_weight_conventions();
overlap
&& cache.iter_indexed().all(|(_, _, layer)| {
let has = |name: &str| layer.dir().join(name).exists();
self.entropy.iter().all(|&nb| has(ScalarKind::Entropy.file_name(nb)))
&& self.quartet.iter().all(|&nb| has(quartet_annex_file_name(nb)))
&& pair_weight.iter().all(|&nb| has(ScalarKind::PairWeight.file_name(nb)))
})
}
/// The pair-weight conventions requested, each once.
pub(crate) fn pair_weight_conventions(&self) -> Vec<bool> {
dedup(self.pair_weight.iter().copied())
@@ -54,7 +54,7 @@ use obipipeline::ThrottleGuard;
use crate::siblings::helpers::central_base;
use crate::siblings::iter::{SiblingEntry, SiblingLayerExt};
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, SiblingAnnex};
use crate::siblings::{FamilyMask, SiblingAnnex};
/// Target memory budget for one batch's `genome_mask` allocation
/// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding
@@ -180,7 +180,7 @@ pub(crate) fn scan_layer_families(
})?;
let n_cols = mat.n_cols().min(n_genomes);
let layer_dir: &Path = mat.dir();
let annex = Arc::new(SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?);
let annex = Arc::new(SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?);
let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k });
@@ -11,7 +11,7 @@ use std::path::Path;
use obicompactvec::TempBitVecBuilder;
use obikindex::{OKIError, OKIResult};
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
use crate::siblings::{FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
mask.is_minorant() && mask.siblings() >= 1
@@ -31,7 +31,7 @@ pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
/// the `Selection` a caller needs to bound `scan_layer_families` to just
/// these families.
pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let mut reindex = HashMap::new();
let mut family_idx = 0usize;
for slot in 0..annex.len() {
@@ -62,7 +62,7 @@ pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashM
/// first counts this layer's total minorants (needed up front to size the
/// bitset), the second sets the eligible bits.
pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let n_minorants = (0..annex.len())
.filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant()))
@@ -103,7 +103,7 @@ pub(crate) fn iter_full_scalar(
kind: ScalarKind,
no_ambiguity: bool,
) -> OKIResult<impl Iterator<Item = f32> + use<>> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let scalars =
FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?;
if scalars.no_ambiguity() != no_ambiguity {
@@ -8,7 +8,7 @@ use obikindex::{OKIError, OKIResult};
use super::minorant_selection::is_non_monomorphic_minorant;
use crate::siblings::quartet_annex::quartet_annex_file_name;
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
use crate::siblings::SiblingAnnex;
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
/// included), every family the quartet annex flags as not quartet-informative
@@ -22,7 +22,7 @@ pub(crate) fn restrict_to_quartet(
layer_dir: &Path,
no_ambiguity: bool,
) -> OKIResult<()> {
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity)))
.map_err(OKIError::Io)?;
+9 -16
View File
@@ -9,11 +9,12 @@
//! already stored — no re-derivation, no lookup).
use obikidxcache::index_cache::IndexCache;
use obikindex::{OKIError, OKIResult};
use obikindex::OKIResult;
use super::family_scan::{Selection, scan_layer_families};
use super::is_fast_mode;
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
use crate::siblings::SiblingAnnex;
use crate::siblings::extensions::SiblingBuilder;
/// Layers scanned at once per [`sibling_family_size_histogram`]'s
/// `par_map_reduce` call — same bounding reasoning as
@@ -45,26 +46,17 @@ pub struct SiblingAnnexStats {
/// on top of that would fight it for cache/mmap locality) — this function
/// has no such inner parallelism to protect.
///
/// Existence of every cached layer's annex file is checked up front, before
/// the parallel map runs, so a missing one is reported as an error rather
/// than silently read back as zero.
/// A missing annex is built up front (`ensure_sibling_annex`), before the
/// parallel map runs, so none is silently read back as zero.
pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> {
for (part, l, layer) in cache.iter_indexed() {
let path = layer.dir().join(ANNEX_FILE_NAME);
if !path.exists() {
return Err(OKIError::InvalidInput(format!(
"no sibling annex at {} (partition {part}, layer {l}) — run --sibling-annex first",
path.display()
)));
}
}
cache.ensure_sibling_annex()?;
Ok(cache.par_map_reduce(
MAX_CONCURRENT_LAYERS,
|| [0u64; 4],
|layer| {
let annex = SiblingAnnex::open(&layer.dir().join(ANNEX_FILE_NAME))
.expect("existence already checked before this parallel map ran");
let annex = SiblingAnnex::open_in(layer.dir())
.expect("ensure_sibling_annex already made every layer's annex exist");
let mut counts = [0u64; 4];
for slot in 0..annex.len() {
let Some(mask) = annex.get(slot) else { continue };
@@ -95,6 +87,7 @@ pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u6
/// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/
/// `algorithms::scan::sampled_scan` already follow).
pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> {
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache);
@@ -241,6 +241,9 @@ pub(crate) fn sample_index(
skip_layers: usize,
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
) -> OKIResult<SamplePlan> {
// Every selection below reads the sibling annex (`eligible_bitset_layer`),
// whichever derived annexes the options ask for.
cache.ensure_sibling_annex()?;
let n_genomes = cache.meta().genomes().len();
let fast_mode = is_fast_mode(cache);
@@ -12,10 +12,13 @@ use obikindex::{OKIError, OKIResult};
use obisys::progress_bar;
use crate::siblings::algorithms::{
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes,
inverse_overlap_matrix, is_fast_mode,
AnnexRequest, build_layer_sibling_annex, ensure_layer_family_annexes,
ensure_layer_pair_weight_annexes, inverse_overlap_matrix, is_fast_mode,
};
use crate::siblings::{
ANNEX_FILE_NAME, FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, FamilyOverlapAccumulator,
QUARTET_ANNEX_FILE_NAMES, ScalarKind, family_overlap_file_name,
};
use crate::siblings::{FamilyOverlap, FamilyOverlapAccumulator, family_overlap_file_name};
pub(crate) trait SiblingBuilder {
/// Build, for every cached layer, the annexes `request` asks for that
@@ -26,13 +29,129 @@ pub(crate) trait SiblingBuilder {
/// a missing entropy, quartet or overlap annex triggers one scan that
/// feeds every missing one, per `ensure_layer_family_annexes`'s own docs,
/// then a second scan builds the missing pair weights, which need the
/// finished overlap matrix. A no-op, cheap existence-check pass, once
/// everything requested already exists.
/// finished overlap matrix. Makes sure the sibling annex exists first
/// ([`ensure_sibling_annex`](Self::ensure_sibling_annex)). A no-op, cheap
/// existence-check pass that takes no lock, once everything requested
/// already exists; otherwise the build runs under the index's exclusive
/// lock ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock)).
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
/// [`ensure_family_annexes`](Self::ensure_family_annexes)'s build step,
/// without the existence pre-check and without the lock: the caller
/// holds it.
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
/// Build the sibling annex (`siblings.psib`, every annex above is derived
/// from it) if any cached layer lacks it — the state of an index right
/// after `migrate-mphf`, which sets every annex aside. The build writes
/// into every layer, so it runs under the index's exclusive lock
/// ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock), a no-op
/// when the caller already holds it); a process that waited for it finds
/// the annex built by the one that held it, and does nothing. A no-op,
/// cheap existence-check pass that takes no lock, once every layer has
/// its annex.
fn ensure_sibling_annex(&self) -> OKIResult<()>;
/// (Re)build the sibling annex of every cached layer, unconditionally,
/// removing the annexes derived from it (entropy, quartet, pair weight,
/// family overlap), which its renumbering makes stale. Takes no lock:
/// the caller holds it ([`SiblingExt::build_sibling_annex`], or
/// [`ensure_sibling_annex`](Self::ensure_sibling_annex)).
fn build_base_sibling_annex(&self) -> OKIResult<()>;
}
impl SiblingBuilder for IndexCache {
fn ensure_sibling_annex(&self) -> OKIResult<()> {
let missing = || self.iter_indexed().any(|(_, _, layer)| !layer.dir().join(ANNEX_FILE_NAME).exists());
if !missing() {
return Ok(());
}
self.index().with_lock(|| {
if !missing() {
return Ok(()); // built by the process we waited for
}
tracing::info!("sibling annex missing — building it");
self.build_base_sibling_annex()
})
}
fn build_base_sibling_annex(&self) -> OKIResult<()> {
let n_parts = self.index().n_partitions();
let fast_mode = is_fast_mode(self);
if !fast_mode {
tracing::warn!(
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
disabled; consider compacting this index"
);
}
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
let mut total_slots: u64 = 0;
for part in self.partitions() {
let n_layer = self.n_layer(part).unwrap_or(0);
let mut part_slots: u64 = 0;
for l in 0..n_layer {
let layer = self
.get_layer(part, l)
.expect("cache-consistent: n_layer(part) already bounds l");
// Invalidate this layer's entropy, quartet and pair-weight
// annexes, if any — they were built against the sibling
// annex's *previous* `family_idx` numbering
// (`non_monomorphic_reindex_layer` reads the sibling
// annex), and are stale once the new one exists.
// `ensure_family_annexes` only builds what's *missing*, so
// the old files must be removed here for them to be rebuilt
// rather than silently left stale. Removed *before* the new
// annex is built and published (atomically, at its end): a
// build killed in between leaves the previous annex with no
// derived files — still consistent, rebuilt on demand —
// never a new annex next to derived files computed against
// the old numbering.
for name in ScalarKind::Entropy
.file_names()
.into_iter()
.chain(ScalarKind::PairWeight.file_names())
.chain(QUARTET_ANNEX_FILE_NAMES)
{
let derived_path = layer.dir().join(name);
if derived_path.exists() {
std::fs::remove_file(&derived_path).map_err(OKIError::Io)?;
}
}
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
}
total_slots += part_slots;
pb.inc(1);
pb.set_message(format!(
"partition {part}: {part_slots} kmers ({total_slots} total)"
));
}
pb.finish_and_clear();
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
// The family-overlap annexes are index-level, not per-layer (see
// their own module docs) — any layer's sibling annex changing makes
// them all stale (and, with them, every pair weight computed from
// them, removed per layer above), so they are invalidated once here
// rather than per-layer above.
for name in FAMILY_OVERLAP_FILE_NAMES {
let overlap_path = self.index().dir().join(name);
if overlap_path.exists() {
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
}
}
Ok(())
}
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
self.ensure_sibling_annex()?;
if request.is_satisfied(self) {
return Ok(());
}
self.index().with_lock(|| self.build_family_annexes(request))
}
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
let n_genomes = self.meta().genomes().len();
let fast_mode = is_fast_mode(self);
let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity));
@@ -11,17 +11,12 @@ use std::path::Path;
use obikidxcache::index_cache::IndexCache;
use obikindex::{OKIError, OKIResult};
use obisys::progress_bar;
use crate::siblings::algorithms::{
AnnexRequest, EntropySummary, SiblingAnnexStats, build_layer_sibling_annex, is_fast_mode,
scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
AnnexRequest, EntropySummary, SiblingAnnexStats, scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
};
use crate::siblings::extensions::SiblingBuilder;
use crate::siblings::{
FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, QUARTET_ANNEX_FILE_NAMES, ScalarKind,
family_overlap_file_name,
};
use crate::siblings::{FamilyOverlap, family_overlap_file_name};
pub trait SiblingExt {
/// Build the sibling-count/minorant annex for every layer of every
@@ -111,82 +106,18 @@ pub trait SiblingExt {
impl SiblingExt for IndexCache {
fn build_sibling_annex(&self) -> OKIResult<()> {
let n_parts = self.index().n_partitions();
let fast_mode = is_fast_mode(self);
if !fast_mode {
tracing::warn!(
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
disabled; consider compacting this index"
);
}
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
let mut total_slots: u64 = 0;
for part in self.partitions() {
let n_layer = self.n_layer(part).unwrap_or(0);
let mut part_slots: u64 = 0;
for l in 0..n_layer {
let layer = self
.get_layer(part, l)
.expect("cache-consistent: n_layer(part) already bounds l");
// Invalidate this layer's entropy and quartet annexes, if
// any — they were built against the sibling annex's
// *previous* `family_idx` numbering
// (`non_monomorphic_reindex_layer` reads the sibling
// annex), and are stale once the new one exists.
// `ensure_family_annexes` below only builds what's
// *missing*, so the old files must be removed here for
// them to be rebuilt rather than silently left stale.
// Removed *before* the new annex is built and published
// (atomically, at its end): a build killed in between
// leaves the previous annex with no derived files — still
// consistent, rebuilt on demand — never a new annex next to
// derived files computed against the old numbering.
for name in ScalarKind::Entropy
.file_names()
.into_iter()
.chain(ScalarKind::PairWeight.file_names())
.chain(QUARTET_ANNEX_FILE_NAMES)
{
let entropy_path = layer.dir().join(name);
if entropy_path.exists() {
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
}
}
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
}
total_slots += part_slots;
pb.inc(1);
pb.set_message(format!(
"partition {part}: {part_slots} kmers ({total_slots} total)"
));
}
pb.finish_and_clear();
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
// The family-overlap annexes are index-level, not per-layer (see
// their own module docs) — any layer's sibling annex changing makes
// them all stale (and, with them, every pair weight computed from
// them, removed per layer above), so they are invalidated once here
// rather than per-layer above.
for name in FAMILY_OVERLAP_FILE_NAMES {
let overlap_path = self.index().dir().join(name);
if overlap_path.exists() {
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
}
}
// Precompute the entropy and family-overlap annexes too, now that
// every layer's fresh sibling annex is on disk
// (`non_monomorphic_reindex_layer`, which both depend on, reads
// it) — same routine-rebuild footing as the sibling annex itself,
// rather than leaving them to be built lazily on first
// `--shannon`/`--entropy-bias`/`--min-shared-family`/
// `--family-overlap` use.
tracing::info!("building entropy and quartet (both --no-ambiguity conventions) and family-overlap annexes");
self.ensure_family_annexes(&AnnexRequest::everything())?;
Ok(())
self.index().with_lock(|| {
self.build_base_sibling_annex()?;
// Precompute the entropy, quartet, pair-weight and family-overlap
// annexes too, now that every layer's fresh sibling annex is on
// disk (`non_monomorphic_reindex_layer`, which they depend on,
// reads it) — same routine-rebuild footing as the sibling annex
// itself, rather than leaving them to be built lazily on first
// `--shannon`/`--entropy`/`--quartet`/`--pair-balance`/
// `--min-shared-family`/`--family-overlap` use.
tracing::info!("building entropy, quartet, pair-weight (both --no-ambiguity conventions) and family-overlap annexes");
self.ensure_family_annexes(&AnnexRequest::everything())
})
}
fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> {
@@ -269,6 +269,21 @@ impl SiblingAnnex {
Ok(Self { mmap, n, path: path.to_path_buf() })
}
/// The annex of the layer directory `layer_dir`. Same as
/// [`open`](Self::open), except that an absent file is reported with its
/// path and the way to build it, rather than a bare `NotFound` — the
/// state an index is in right after `migrate-mphf` set its annexes aside.
pub fn open_in(layer_dir: &Path) -> io::Result<Self> {
let path = layer_dir.join(super::ANNEX_FILE_NAME);
Self::open(&path).map_err(|e| match e.kind() {
io::ErrorKind::NotFound => io::Error::new(
io::ErrorKind::NotFound,
format!("no sibling annex at {} — build it with `phylo --sibling-annex`", path.display()),
),
_ => e,
})
}
pub fn path(&self) -> &Path { &self.path }
pub fn len(&self) -> usize { self.n }
pub fn is_empty(&self) -> bool { self.n == 0 }
+5 -1
View File
@@ -3,6 +3,9 @@
mod budget;
mod lock;
mod lock_slot;
#[cfg(test)]
mod lock_tests;
mod numa;
mod profiler;
mod progress;
@@ -10,7 +13,8 @@ mod resources;
mod stage;
pub use budget::MemoryBudget;
pub use lock::DirLock;
pub use lock::{DirLock, LockMode};
pub use lock_slot::LockSlot;
pub use numa::PartitionRunner;
pub use profiler::start as start_profiler;
pub use progress::{Progress, TracedBar, progress_bar, spinner};
+188 -38
View File
@@ -1,56 +1,206 @@
use std::fs::File;
use std::io;
use std::path::{Path, PathBuf};
use std::time::Duration;
use tracing::info;
/// Exclusive advisory lock on an index directory, held for the duration of
/// any command that writes into an already-existing index (building the
/// sibling annex, merging into a destination, filtering/selecting in place,
/// ...). Two such commands racing on the same directory can otherwise
/// corrupt each other's writes with no error from either side.
/// Advisory readers/writer lock on an index directory, shared between
/// processes (and between independent handles of one process).
///
/// Only the directory actually being *written to* needs a lock — a command
/// like `merge` that reads several source indexes to build one destination
/// only needs to lock the destination.
/// - **Shared** ([`acquire_shared`](DirLock::acquire_shared)): held by
/// anything that reads the index (`query`, `dump`, a `merge` source, a
/// `phylo` run, ...). Any number of shared holders coexist.
/// - **Exclusive** ([`acquire`](DirLock::acquire)): held by anything that
/// writes into an already-existing index (building annexes, `convert`,
/// `pack`, `annotate`, ...). It waits for every shared holder to leave and
/// keeps others out while held. Two writers racing on one directory can
/// otherwise corrupt each other's files with no error from either side,
/// and a reader would see files vanish under it.
///
/// Uses the OS's advisory file lock (`flock` on Unix, `LockFileEx` on
/// Windows) via `std::fs::File::lock`/`try_lock`, not a hand-rolled PID
/// file: the OS releases it automatically on process exit, including a
/// crash — no stale-lock cleanup logic needed.
/// Only the directory actually being *written to* needs an exclusive lock —
/// `merge` holds shared locks on its sources and an exclusive one on its
/// destination.
///
/// # How it is built
///
/// Each mode is one OS advisory file lock (`flock` on Unix, `LockFileEx` on
/// Windows) on `.obikmer.lock`, so a crash releases it with no stale-lock
/// cleanup. On top of that, **every transition** (acquire, release into
/// another mode, promote, demote) happens while holding a *gate* — a second
/// lock file, `.obikmer.gate`, taken for the length of one non-blocking
/// attempt and never while waiting. A blocked request therefore releases the
/// gate, sleeps, and tries again (polling, 20 ms growing to 500 ms), instead
/// of sleeping in the kernel while holding it. Because nobody else can touch
/// the lock state while the gate is held, a shared → exclusive
/// [promotion](DirLock::promote) cannot be overtaken by another writer, which
/// the OS's own `flock` conversion does not guarantee.
///
/// There is no writer preference: a steady stream of readers can delay an
/// exclusive request.
#[derive(Debug)]
pub struct DirLock {
_file: std::fs::File,
file: File,
dir: PathBuf,
mode: LockMode,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LockMode {
Shared,
Exclusive,
}
const LOCK_FILE: &str = ".obikmer.lock";
const GATE_FILE: &str = ".obikmer.gate";
const CLAIM_FILE: &str = ".obikmer.promote";
const POLL_MIN: Duration = Duration::from_millis(20);
const POLL_MAX: Duration = Duration::from_millis(500);
fn open_file(dir: &Path, name: &str) -> io::Result<File> {
std::fs::create_dir_all(dir)?;
std::fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(dir.join(name))
}
/// The gate: blocks, but its holders only ever run instantaneous,
/// non-blocking operations, so the wait is negligible. Released on drop.
fn gate(dir: &Path) -> io::Result<File> {
let f = open_file(dir, GATE_FILE)?;
f.lock()?;
Ok(f)
}
/// Non-blocking attempt in `mode`; `Ok(false)` means "held by someone else".
fn try_lock(file: &File, mode: LockMode) -> io::Result<bool> {
let r = match mode {
LockMode::Exclusive => file.try_lock(),
LockMode::Shared => file.try_lock_shared(),
};
match r {
Ok(()) => Ok(true),
Err(std::fs::TryLockError::WouldBlock) => Ok(false),
Err(std::fs::TryLockError::Error(e)) => Err(e),
}
}
/// Sleeps between two attempts and logs once, at the first one.
struct Backoff {
delay: Duration,
logged: bool,
}
impl Backoff {
fn new() -> Self {
Self { delay: POLL_MIN, logged: false }
}
fn wait(&mut self, dir: &Path, why: &str) {
if !self.logged {
info!(dir = %dir.display(), "{why}");
self.logged = true;
}
std::thread::sleep(self.delay);
self.delay = (self.delay * 2).min(POLL_MAX);
}
}
impl DirLock {
/// Block until the exclusive lock on `dir` is acquired (creating `dir`
/// and the lock file within it if needed). Logs once if the wait is
/// and the lock files within it if needed). Logs once if the wait is
/// non-trivial, so a blocked command doesn't look silently hung.
pub fn acquire(dir: &std::path::Path) -> std::io::Result<Self> {
let file = Self::open_lock_file(dir)?;
if file.try_lock().is_err() {
info!(dir = %dir.display(), "waiting for another obikmer process to release this index");
file.lock()?;
}
Ok(Self { _file: file })
pub fn acquire(dir: &Path) -> io::Result<Self> {
Self::acquire_mode(dir, LockMode::Exclusive, "waiting for other obikmer processes to release this index")
}
/// Non-blocking probe: `Ok(Some(lock))` if `dir`'s lock was free and is
/// now held by the returned `DirLock`, `Ok(None)` if another process
/// currently holds it (not an error — contention is the expected,
/// common case this exists to distinguish from a genuine I/O failure).
pub fn try_acquire(dir: &std::path::Path) -> std::io::Result<Option<Self>> {
let file = Self::open_lock_file(dir)?;
match file.try_lock() {
Ok(()) => Ok(Some(Self { _file: file })),
Err(_) => Ok(None),
/// Block until a shared lock on `dir` is acquired — until no exclusive
/// holder remains. Logs once if the wait is non-trivial.
pub fn acquire_shared(dir: &Path) -> io::Result<Self> {
Self::acquire_mode(dir, LockMode::Shared, "waiting for an obikmer process writing this index")
}
fn acquire_mode(dir: &Path, mode: LockMode, why: &str) -> io::Result<Self> {
let file = open_file(dir, LOCK_FILE)?;
let mut backoff = Backoff::new();
loop {
{
let _gate = gate(dir)?;
if try_lock(&file, mode)? {
return Ok(Self { file, dir: dir.to_owned(), mode });
}
}
backoff.wait(dir, why);
}
}
fn open_lock_file(dir: &std::path::Path) -> std::io::Result<std::fs::File> {
std::fs::create_dir_all(dir)?;
let lock_path = dir.join(".obikmer.lock");
std::fs::OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(&lock_path)
pub fn mode(&self) -> LockMode {
self.mode
}
/// Non-blocking probe: `Ok(true)` if no exclusive holder exists right
/// now (a shared lock could be taken), `Ok(false)` if one does. Retains
/// nothing either way.
pub fn is_free_of_writer(dir: &Path) -> io::Result<bool> {
let file = open_file(dir, LOCK_FILE)?;
let _gate = gate(dir)?;
try_lock(&file, LockMode::Shared)
}
/// Shared → exclusive. Returns the exclusive lock and whether the
/// promotion was **gapless**: `true` means the lock was never released,
/// so nothing in the directory can have changed since the shared lock
/// was held and whatever the caller read stays valid; `false` means the
/// shared lock had to be given up while waiting, so a writer may have
/// run in between and the caller must re-check what it relied on.
///
/// Gapless promotion is available to one promoter at a time (it holds
/// `.obikmer.promote` while it waits for the other readers to leave).
/// A second concurrent promoter releases its shared lock instead — two
/// promoters each keeping theirs while waiting for the other's would
/// wait for ever. An exclusive lock is returned unchanged.
pub fn promote(self) -> io::Result<(Self, bool)> {
if self.mode == LockMode::Exclusive {
return Ok((self, true));
}
let claim = open_file(&self.dir, CLAIM_FILE)?;
if !try_lock(&claim, LockMode::Exclusive)? {
let dir = self.dir.clone();
drop(self); // gives the shared lock up
return Ok((Self::acquire(&dir)?, false));
}
let mut backoff = Backoff::new();
loop {
{
let _gate = gate(&self.dir)?;
self.file.unlock()?;
if try_lock(&self.file, LockMode::Exclusive)? {
return Ok((Self { mode: LockMode::Exclusive, ..self }, true));
}
// Other readers remain. Nobody can hold the lock exclusively
// without the gate, so taking our shared lock back succeeds.
if !try_lock(&self.file, LockMode::Shared)? {
return Err(io::Error::other("lost the shared index lock during promotion"));
}
}
backoff.wait(&self.dir, "waiting for other readers of this index before writing to it");
}
}
/// Exclusive → shared, without ever leaving the lock free in between.
/// A shared lock is returned unchanged.
pub fn demote(self) -> io::Result<Self> {
if self.mode == LockMode::Shared {
return Ok(self);
}
let held_gate = gate(&self.dir)?;
self.file.unlock()?;
if !try_lock(&self.file, LockMode::Shared)? {
return Err(io::Error::other("could not take the shared index lock back after writing"));
}
drop(held_gate);
Ok(Self { mode: LockMode::Shared, ..self })
}
}
+96
View File
@@ -0,0 +1,96 @@
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use crate::lock::{DirLock, LockMode};
/// The directory lock of one handle on an index: nothing, shared or
/// exclusive, and the transitions between them that a handle needs. What
/// `obikindex::KmerIndex` delegates its locking to, kept apart so that it can
/// be tested without building an index.
///
/// Interior mutability (`&self` methods): a handle is shared behind `Arc`.
/// Not for concurrent use from several threads of one handle — the first
/// section to finish would restore the lock under the others.
#[derive(Debug)]
pub struct LockSlot {
dir: PathBuf,
held: Mutex<Option<DirLock>>,
}
impl LockSlot {
/// A slot holding nothing (a handle that took no lock, or one that
/// reopens a directory its own builder already holds).
pub fn empty(dir: &Path) -> Self {
Self { dir: dir.to_owned(), held: Mutex::new(None) }
}
/// A slot holding a shared lock on `dir`; blocks while a writer is at work.
pub fn shared(dir: &Path) -> io::Result<Self> {
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire_shared(dir)?)) })
}
/// A slot holding the exclusive lock on `dir`; blocks while anyone else holds it.
pub fn exclusive(dir: &Path) -> io::Result<Self> {
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire(dir)?)) })
}
/// Whether the slot holds the exclusive lock (a shared one is not write access).
pub fn holds_write_lock(&self) -> bool {
matches!(&*self.held.lock().unwrap(), Some(l) if l.mode() == LockMode::Exclusive)
}
/// Hold the exclusive lock from now on: a no-op if already held that way,
/// a promotion of a shared lock, a plain acquisition of an empty slot.
pub fn acquire_exclusive(&self) -> io::Result<()> {
let held = self.held.lock().unwrap().take();
let lock = match held {
Some(l) => l.promote()?.0,
None => DirLock::acquire(&self.dir)?,
};
*self.held.lock().unwrap() = Some(lock);
Ok(())
}
/// Run `f` with the exclusive lock held:
/// - already exclusive: `f` runs as is (so sections nest);
/// - shared: promoted for the duration of `f`, then demoted back, so the
/// slot keeps the shared lock it came with. The promotion is gapless
/// when no other process promotes at the same time; otherwise other
/// writers may run between the shared lock being given up and the
/// exclusive one being taken. Either way **`f` must re-check, before
/// building, what it is about to build**, and the caller must not hold
/// anything read earlier that a writer could have replaced;
/// - empty: the lock is taken for `f` only.
///
/// Restored also if `f` fails; an I/O error taking or restoring the lock
/// is converted to `E`.
pub fn with_exclusive<T, E: From<io::Error>>(&self, f: impl FnOnce() -> Result<T, E>) -> Result<T, E> {
enum Before {
Exclusive,
Shared,
Nothing,
}
let held = self.held.lock().unwrap().take();
let (lock, before) = match held {
Some(l) if l.mode() == LockMode::Exclusive => (l, Before::Exclusive),
Some(l) => (l.promote()?.0, Before::Shared),
None => (DirLock::acquire(&self.dir)?, Before::Nothing),
};
*self.held.lock().unwrap() = Some(lock);
let out = f();
let lock = self.held.lock().unwrap().take();
match (before, lock) {
(Before::Exclusive, lock) => *self.held.lock().unwrap() = lock,
(Before::Shared, Some(l)) => *self.held.lock().unwrap() = Some(l.demote()?),
_ => {} // taken for `f` only: released here
}
out
}
/// Non-blocking probe: `true` if no process holds the directory
/// exclusively right now (this slot included), `false` if one does.
pub fn is_free_of_writer(&self) -> bool {
DirLock::is_free_of_writer(&self.dir).unwrap_or(false)
}
}
+208
View File
@@ -0,0 +1,208 @@
//! The directory lock is a readers/writer lock: `shared` slots coexist, an
//! `exclusive` one waits for them all and keeps new ones out, and a shared
//! slot can be promoted for a section (`with_exclusive`) and demoted back.
//! No index is involved, and therefore none of the process-wide k-mer
//! parameters other crates' tests share.
use std::io;
use std::path::PathBuf;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::mpsc;
use std::sync::{Arc, Barrier};
use std::thread;
use std::time::Duration;
use crate::lock::{DirLock, LockMode};
use crate::lock_slot::LockSlot;
const SETTLE: Duration = Duration::from_millis(400);
const PATIENCE: Duration = Duration::from_secs(10);
/// A scratch directory removed on drop (no `tempfile` dependency here).
struct Scratch(PathBuf);
impl Scratch {
fn new() -> Self {
static N: AtomicUsize = AtomicUsize::new(0);
let dir = std::env::temp_dir().join(format!(
"obisys-lock-{}-{}",
std::process::id(),
N.fetch_add(1, Ordering::Relaxed)
));
std::fs::create_dir_all(&dir).unwrap();
Self(dir)
}
}
impl Drop for Scratch {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
#[test]
fn readers_coexist() {
let d = Scratch::new();
let a = LockSlot::shared(&d.0).unwrap();
let b = LockSlot::shared(&d.0).unwrap();
assert!(!a.holds_write_lock() && !b.holds_write_lock());
assert!(a.is_free_of_writer(), "readers are not writers");
}
#[test]
fn exclusive_is_exclusive_and_released_on_drop() {
let d = Scratch::new();
let w = LockSlot::exclusive(&d.0).unwrap();
assert!(w.holds_write_lock());
assert!(!w.is_free_of_writer());
drop(w);
assert!(LockSlot::empty(&d.0).is_free_of_writer());
}
#[test]
fn a_writer_waits_for_the_readers() {
let d = Scratch::new();
let reader = LockSlot::shared(&d.0).unwrap();
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let writer = thread::spawn(move || {
drop(LockSlot::exclusive(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the writer must wait for the reader");
drop(reader);
rx.recv_timeout(PATIENCE).expect("the writer proceeds once the reader is gone");
writer.join().unwrap();
}
#[test]
fn a_reader_waits_for_the_writer() {
let d = Scratch::new();
let writer = LockSlot::exclusive(&d.0).unwrap();
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let reader = thread::spawn(move || {
drop(LockSlot::shared(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the reader must wait for the writer");
drop(writer);
rx.recv_timeout(PATIENCE).expect("the reader proceeds once the writer is gone");
reader.join().unwrap();
}
#[test]
fn with_exclusive_on_a_reader_promotes_then_restores_the_shared_lock() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
let out: io::Result<i32> = slot.with_exclusive(|| {
assert!(slot.holds_write_lock());
assert!(!slot.is_free_of_writer());
Ok(7)
});
assert_eq!(out.unwrap(), 7);
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
// Still a reader: a writer elsewhere waits for it.
let (tx, rx) = mpsc::channel();
let dir = d.0.clone();
let writer = thread::spawn(move || {
drop(LockSlot::exclusive(&dir).unwrap());
tx.send(()).unwrap();
});
assert!(rx.recv_timeout(SETTLE).is_err(), "the slot must be a reader again");
drop(slot);
rx.recv_timeout(PATIENCE).unwrap();
writer.join().unwrap();
}
#[test]
fn with_exclusive_on_an_empty_slot_holds_the_lock_for_the_section_only() {
let d = Scratch::new();
let slot = LockSlot::empty(&d.0);
slot.with_exclusive::<(), io::Error>(|| {
assert!(slot.holds_write_lock());
Ok(())
})
.unwrap();
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
}
#[test]
fn with_exclusive_restores_on_error() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
let r: io::Result<()> = slot.with_exclusive(|| Err(io::Error::other("boom")));
assert!(r.is_err());
assert!(!slot.holds_write_lock());
assert!(slot.is_free_of_writer());
}
#[test]
fn with_exclusive_nests_and_keeps_a_lock_taken_at_open() {
let d = Scratch::new();
let slot = LockSlot::exclusive(&d.0).unwrap();
slot.with_exclusive::<(), io::Error>(|| {
slot.with_exclusive::<(), io::Error>(|| Ok(()))?;
assert!(slot.holds_write_lock(), "an inner section must not release the outer lock");
Ok(())
})
.unwrap();
assert!(slot.holds_write_lock(), "a slot opened exclusive keeps it");
}
#[test]
fn acquire_exclusive_promotes_a_shared_slot_and_is_idempotent() {
let d = Scratch::new();
let slot = LockSlot::shared(&d.0).unwrap();
slot.acquire_exclusive().unwrap();
slot.acquire_exclusive().unwrap();
assert!(slot.holds_write_lock());
}
/// The case a naive upgrade deadlocks on: two readers, each holding a shared
/// lock, both ask for the exclusive one at the same time.
#[test]
fn two_readers_promoting_at_once_both_finish() {
let d = Scratch::new();
let barrier = Arc::new(Barrier::new(2));
let (tx, rx) = mpsc::channel();
let handles: Vec<_> = (0..2)
.map(|_| {
let (dir, b, tx) = (d.0.clone(), Arc::clone(&barrier), tx.clone());
thread::spawn(move || {
let slot = LockSlot::shared(&dir).unwrap();
b.wait(); // both hold a shared lock
slot.with_exclusive::<(), io::Error>(|| {
assert!(slot.holds_write_lock());
thread::sleep(Duration::from_millis(50));
Ok(())
})
.unwrap();
tx.send(()).unwrap();
})
})
.collect();
for _ in 0..2 {
rx.recv_timeout(PATIENCE).expect("a promotion deadlocked");
}
for h in handles {
h.join().unwrap();
}
}
#[test]
fn a_lone_promotion_is_gapless_and_a_demotion_keeps_the_lock() {
let d = Scratch::new();
let shared = DirLock::acquire_shared(&d.0).unwrap();
let (excl, gapless) = shared.promote().unwrap();
assert_eq!(excl.mode(), LockMode::Exclusive);
assert!(gapless, "nobody else was promoting: the lock was never released");
let shared = excl.demote().unwrap();
assert_eq!(shared.mode(), LockMode::Shared);
// Shared again: other readers get in, a writer does not.
let _other = DirLock::acquire_shared(&d.0).unwrap();
assert!(DirLock::is_free_of_writer(&d.0).unwrap());
}