Push ltpzonnnkpqs #79
No files matched your search
@@ -329,7 +329,34 @@ Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 /
|
|||||||
|
|
||||||
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
|
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
|
||||||
|
|
||||||
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment has not been given to IQ-TREE.
|
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment had not been given to IQ-TREE (done 2026-10-16: see the last measured section).
|
||||||
|
|
||||||
|
## Tree-level effect of the combined draw: sCF on three trees (measured 2026-10-16)
|
||||||
|
|
||||||
|
Closes the "not measured" item of the previous section (the combined alignment given to IQ-TREE). Three trees, all sCF by `--scfl 100 --cf-verbose`, model `GTR+G4+ASC`:
|
||||||
|
|
||||||
|
- **A** = `concord` (12/10): entropy-only draw, 180 genomes, 5 003 414 columns; `-t` + `-m` re-ran a full ML search (59 h).
|
||||||
|
- **B** = `concord_quartet_balance`: combined draw (`--entropy 1 --quartet --pair-balance`), 180 genomes, 5 000 000 columns. Its log stops during the tree search (14/10), so the command that produced its `cf.*` (16/10) is not in the files; presumably `-te` on the 15/10 treefile as for G (unverified).
|
||||||
|
- **G** = `scf_good`: same draw as B without the 3 data-poor genomes (*A. pyrenaica* PHA000543, *A. saussurei*, *A. triflora*), 177 genomes. 72 sites became invariant and were removed to satisfy ASC; the search (`concord_quartet_balance_good`, 107 iterations, 6 h 47 on 120 threads) crashed in the sCF step on the FASTA (`Assertion all_prob_const < 1.0`; cause untested, ASC with constant sites is a suspect). sCF were recomputed with `-te <treefile> -s <varsites.phy> -m GTR+G4+ASC --scfl 100`: **no search** (0 iterations, lnL −41 053 676.53 vs −41 053 676.41 from the search), 4 999 729 columns. A used `-t`, B/G a fixed tree: the protocols differ.
|
||||||
|
|
||||||
|
| | A | B | G |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Branches | 177 | 177 | 174 |
|
||||||
|
| Mean `sN` | 3 219 | 13 163 | 13 396 |
|
||||||
|
| Branches with `sN` < 100 / < 1 000 | 13 / 50 | 0 / 5 | 0 / 2 |
|
||||||
|
| Median sCF | 44.2 | 42.9 | 42.8 |
|
||||||
|
| Branches with sCF < 40 / ≥ 70 | 74 / 28 | 67 / 26 | 69 / 25 |
|
||||||
|
| *Primula* and *Androsace* monophyletic | no / no | no / no | yes / yes |
|
||||||
|
|
||||||
|
**Sampling effect (A → B, same 180 genomes).** `sN` ×4.5 and the scarcity is gone. Sites do not become more concordant: on the 113 shared bipartitions (of 177) median sCF 48.0 → 47.3 (44 up, 69 down). Robinson-Foulds 128: the draw rearranged the tree deeply. Consistent with the earlier reading that the intra-*Androsace* conflict is not a matter of site count; the number of branches with sCF < 40 % stays around 70.
|
||||||
|
|
||||||
|
**Interface nodes.** Of the old nodes 209/211/212/216/217 (`sN` 17 / 226 / 184 / 7 / 7) none exists in G. Node 208 survives with `sN` 190 → 5 218 (sCF 54.5 → 37.5). Six mixed clades remain in G (83 to 72 tips, 50 *Primula* + 22-33 *Androsace*), all with `sN` > 3 000.
|
||||||
|
|
||||||
|
**Effect of the 3 genomes (B → G, same draw).** RF 16 (166 of 174 shared), median sCF 43.2 → 43.6, median `sN` unchanged. In B, *A. pyrenaica* and *A. triflora* sit inside *Primula* (a clade of 52 tips = 50 *Primula* + these 2; each has a *Primula* sister, sCF 92 % and 52 %, `sN` 556 and 153); *A. saussurei* stays with *Androsace*. Removing them restores both monophylies and changes nothing else measurably: support for "too little data to place them" (data-poor section), not for contamination, though the latter is not excluded.
|
||||||
|
|
||||||
|
**Where A and G disagree (2026-10-16).** On the 177 common tips: 124 shared bipartitions, 50 only in A, 50 only in G. Differing bipartitions are weakly supported in both trees (median sCF 35.7 in A-only, 37.7 in G-only, versus 47.5-48.1 on the shared ones), so the disagreement sits where support was already low. Differences are mostly local: the closest bipartition of the other tree is 1 tip away for 25 of 50 A-only and 21 of 50 G-only splits (≤ 2 tips: 37 and 30); only 8 and 9 are 5+ tips apart. Greedy removal of the tip that lowers RF most: the first (*P. auricula* subsp. *serratifolia*) takes RF from 100 to 86, then about 2 per tip (*A. maxima*, *P. stricta* var. *obesior*, then ordinary tips); the top three are data-poor genomes. A bipartition separates a data-poor genome (the 5 left after the 3 removals) in 1 of 124 shared splits but in 23 of 50 A-only and 10 of 50 G-only splits; removing those 5 genomes gives RF 72, against 96.6 (range 90-100, 40 draws) for 5 random tips. So about 28 of the 100 differences trace to data-poor genomes, and the remaining 72 do not. Backbone: A has a clade of 50 *Primula* + 25 *Androsace* (sCF 76, `sN` 453), G one of 50 + 26 (sCF 73, `sN` 6 677): one tip apart. Genuine conflict: a clade of 72 *Androsace* exists in both trees (A: sCF 34, `sN` 17 127; G: sCF 38, `sN` 49 473) with 16 different tips, i.e. a topological disagreement carried by tens of thousands of sites in each, not a scarcity effect; consistent with the intra-*Androsace* conflict above.
|
||||||
|
|
||||||
|
**Not established.** Separate contribution of entropy / quartet / pair balance (no ablation); stability of the tree between two independent draws (one draw per setting; A → B shows the topology moves with the draw); whether the quartet-informative restriction biases branch lengths (ASC corrects for variable-only ascertainment, not for requiring two states each in two genomes; tree length 22.5 is not interpretable as substitutions/site).
|
||||||
|
|
||||||
## Open questions
|
## Open questions
|
||||||
|
|
||||||
|
|||||||
@@ -48,7 +48,7 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
|||||||
|
|
||||||
| Option | Description |
|
| Option | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value |
|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex, then every annex derived from it (entropy, quartet, pair weight, family overlap, under both `--no-ambiguity` conventions). Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
|
||||||
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
|
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
|
||||||
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
||||||
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
||||||
|
|||||||
Generated
+1
-1
@@ -1537,7 +1537,7 @@ dependencies = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "obikmer"
|
name = "obikmer"
|
||||||
version = "1.3.6"
|
version = "1.4.1"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
"clap",
|
"clap",
|
||||||
"csv",
|
"csv",
|
||||||
|
|||||||
@@ -90,7 +90,7 @@ impl IndexBuilder for KmerIndex {
|
|||||||
let idx = KmerIndex {
|
let idx = KmerIndex {
|
||||||
root_path: output.to_owned(),
|
root_path: output.to_owned(),
|
||||||
meta: Arc::new(meta),
|
meta: Arc::new(meta),
|
||||||
lock: std::sync::Mutex::new(None),
|
lock: obisys::LockSlot::empty(output),
|
||||||
};
|
};
|
||||||
if !idx.acquire_lock() {
|
if !idx.acquire_lock() {
|
||||||
return Err(OKIError::Io(std::io::Error::other(
|
return Err(OKIError::Io(std::io::Error::other(
|
||||||
@@ -102,7 +102,9 @@ impl IndexBuilder for KmerIndex {
|
|||||||
|
|
||||||
fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> {
|
fn finalize_indexed<P: AsRef<Path>>(output: P, rep: &mut Reporter, sparse: bool) -> OKIResult<Self> {
|
||||||
let output = output.as_ref();
|
let output = output.as_ref();
|
||||||
let idx = KmerIndex::open(output)?;
|
// `output` is held exclusively by the builder that wrote it: reopen
|
||||||
|
// without a lock, a locking `open` would wait for that builder for ever.
|
||||||
|
let idx = KmerIndex::open_unlocked(output)?;
|
||||||
idx.meta.mark_indexed().map_err(OKIError::Io)?;
|
idx.meta.mark_indexed().map_err(OKIError::Io)?;
|
||||||
let t_pack = Stage::start("pack");
|
let t_pack = Stage::start("pack");
|
||||||
idx.pack_matrices(sparse)?;
|
idx.pack_matrices(sparse)?;
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
use std::fs;
|
use std::fs;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::sync::{Arc, Mutex};
|
use std::sync::Arc;
|
||||||
|
|
||||||
use crate::layer::KmerLayer;
|
use crate::layer::KmerLayer;
|
||||||
use crate::partition::KmerPartition;
|
use crate::partition::KmerPartition;
|
||||||
@@ -15,7 +15,7 @@ use crate::index::meta::{GenomeInfo, IndexConfig, IndexMeta};
|
|||||||
pub struct KmerIndex {
|
pub struct KmerIndex {
|
||||||
pub(crate) root_path: PathBuf,
|
pub(crate) root_path: PathBuf,
|
||||||
pub(crate) meta: Arc<IndexMeta>,
|
pub(crate) meta: Arc<IndexMeta>,
|
||||||
pub(crate) lock: Mutex<Option<obisys::DirLock>>,
|
pub(crate) lock: obisys::LockSlot,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl KmerIndex {
|
impl KmerIndex {
|
||||||
@@ -37,11 +37,8 @@ impl KmerIndex {
|
|||||||
set_m(config.minimizer_size);
|
set_m(config.minimizer_size);
|
||||||
let genomes = genome_info.into_iter().collect();
|
let genomes = genome_info.into_iter().collect();
|
||||||
let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?;
|
let meta = IndexMeta::create_at(&root_path, config, genomes).map_err(OKIError::Io)?;
|
||||||
let idx = Self {
|
let lock = obisys::LockSlot::empty(&root_path);
|
||||||
root_path,
|
let idx = Self { root_path, meta: Arc::new(meta), lock };
|
||||||
meta: Arc::new(meta),
|
|
||||||
lock: Mutex::new(None),
|
|
||||||
};
|
|
||||||
if !idx.acquire_lock() {
|
if !idx.acquire_lock() {
|
||||||
return Err(OKIError::Io(std::io::Error::other(
|
return Err(OKIError::Io(std::io::Error::other(
|
||||||
"failed to acquire index lock right after create",
|
"failed to acquire index lock right after create",
|
||||||
@@ -50,67 +47,92 @@ impl KmerIndex {
|
|||||||
Ok(idx)
|
Ok(idx)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Open an existing index for reading — doesn't touch the directory
|
/// Open an existing index for reading, holding a **shared** directory
|
||||||
/// lock at all: concurrent readers (`query`, `dump`, a `merge` source,
|
/// lock for as long as the returned index lives: any number of readers
|
||||||
/// ...) must never block on each other. A caller that needs exclusive
|
/// coexist, but a writer ([`open_lock`](Self::open_lock),
|
||||||
/// access (resuming an interrupted build) uses
|
/// [`with_lock`](Self::with_lock)) waits for them all, and readers wait
|
||||||
/// [`open_lock`](Self::open_lock) instead; either way, `self.lock`
|
/// for a writer at work — a reader never sees files change under it.
|
||||||
/// starts empty — [`test_lock`](Self::test_lock)/
|
/// The lock is taken before the metadata is read.
|
||||||
/// [`acquire_lock`](Self::acquire_lock) are opt-in from here.
|
///
|
||||||
|
/// Never open, from a process that holds the write lock on a directory
|
||||||
|
/// through another handle, that same directory again with `open`: the
|
||||||
|
/// new handle would wait for the first for ever.
|
||||||
pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
|
pub fn open<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
|
||||||
let root_path = path.as_ref().to_owned();
|
Self::open_with(path.as_ref(), obisys::LockMode::Shared)
|
||||||
let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
|
|
||||||
set_k(meta.config.kmer_size);
|
|
||||||
set_m(meta.config.minimizer_size);
|
|
||||||
Ok(Self {
|
|
||||||
root_path,
|
|
||||||
meta: Arc::new(meta),
|
|
||||||
lock: Mutex::new(None),
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `open`, then hold the directory lock exclusively — for a caller
|
/// `open`, holding the directory lock **exclusively** for as long as the
|
||||||
/// that resumes a write pipeline (e.g. `obikmer index` re-run on an
|
/// returned index lives — **the** way for a command that modifies an
|
||||||
/// existing, not-yet-`Indexed` directory) rather than reading
|
/// existing index in place (`convert`, `pack`, `annotate`, `utils
|
||||||
/// concurrently with others.
|
/// rename`, resuming an interrupted build, ...) to claim it. The lock is
|
||||||
|
/// taken *before* the metadata is read, so what the caller reads is what
|
||||||
|
/// it will modify.
|
||||||
|
///
|
||||||
|
/// A command that only reads, or that writes a *new* index elsewhere
|
||||||
|
/// (`merge`, `filter`, `select`: exclusive on their destination, shared
|
||||||
|
/// on their sources), uses [`open`](Self::open). Library code that has to
|
||||||
|
/// write into an index it was handed (annexes built on demand) wraps that
|
||||||
|
/// write in [`with_lock`](Self::with_lock).
|
||||||
pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
|
pub fn open_lock<P: AsRef<Path>>(path: P) -> OKIResult<Self> {
|
||||||
let idx = Self::open(path)?;
|
Self::open_with(path.as_ref(), obisys::LockMode::Exclusive)
|
||||||
if !idx.acquire_lock() {
|
}
|
||||||
return Err(OKIError::Io(std::io::Error::other(
|
|
||||||
"failed to acquire index lock",
|
fn open_with(path: &Path, mode: obisys::LockMode) -> OKIResult<Self> {
|
||||||
)));
|
// A directory that is not an index is reported by `open_unlocked`,
|
||||||
|
// without a lock file being created in it.
|
||||||
|
if !Self::is_an_index(path) {
|
||||||
|
return Self::open_unlocked(path);
|
||||||
}
|
}
|
||||||
|
let lock = match mode {
|
||||||
|
obisys::LockMode::Shared => obisys::LockSlot::shared(path),
|
||||||
|
obisys::LockMode::Exclusive => obisys::LockSlot::exclusive(path),
|
||||||
|
}
|
||||||
|
.map_err(OKIError::Io)?;
|
||||||
|
let mut idx = Self::open_unlocked(path)?;
|
||||||
|
idx.lock = lock;
|
||||||
Ok(idx)
|
Ok(idx)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Block until this index's directory lock is acquired, then hold it
|
/// `open` without taking any lock — for a caller that already holds the
|
||||||
/// (released on [`release_lock`](Self::release_lock) or when `self` is
|
/// directory through another handle (`finalize_indexed`, reopening the
|
||||||
/// dropped). Returns `false` only on a genuine I/O failure acquiring
|
/// index its own builder holds exclusively).
|
||||||
/// the lock file itself — contention blocks rather than failing.
|
pub(crate) fn open_unlocked(path: &Path) -> OKIResult<Self> {
|
||||||
pub fn acquire_lock(&self) -> bool {
|
let root_path = path.to_owned();
|
||||||
match obisys::DirLock::acquire(&self.root_path) {
|
let meta = IndexMeta::open_at(&root_path).map_err(OKIError::Io)?;
|
||||||
Ok(lock) => {
|
set_k(meta.config.kmer_size);
|
||||||
*self.lock.lock().unwrap() = Some(lock);
|
set_m(meta.config.minimizer_size);
|
||||||
true
|
let lock = obisys::LockSlot::empty(&root_path);
|
||||||
}
|
Ok(Self { root_path, meta: Arc::new(meta), lock })
|
||||||
Err(_) => false,
|
}
|
||||||
}
|
|
||||||
|
/// Take the exclusive directory lock and hold it until `self` is dropped
|
||||||
|
/// — for an index just created, which is about to be written to.
|
||||||
|
/// Idempotent. Returns `false` only on a genuine I/O failure.
|
||||||
|
pub(crate) fn acquire_lock(&self) -> bool {
|
||||||
|
self.lock.acquire_exclusive().is_ok()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether this index object currently holds its directory lock
|
||||||
|
/// exclusively (a shared lock is not "write access").
|
||||||
|
pub fn holds_write_lock(&self) -> bool {
|
||||||
|
self.lock.holds_write_lock()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run `f` with the directory lock held **exclusively** — promoting the
|
||||||
|
/// shared lock a handle from [`open`](Self::open) holds, and restoring it
|
||||||
|
/// afterwards; see [`obisys::LockSlot::with_exclusive`], including what
|
||||||
|
/// `f` must re-check. Library code that writes into an index it was
|
||||||
|
/// handed (annexes built on demand) wraps that write in this.
|
||||||
|
pub fn with_lock<T>(&self, f: impl FnOnce() -> OKIResult<T>) -> OKIResult<T> {
|
||||||
|
self.lock.with_exclusive(f)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Non-blocking probe of this index's directory lock — doesn't retain
|
/// Non-blocking probe of this index's directory lock — doesn't retain
|
||||||
/// it either way: `true` if the lock was free (briefly acquired, then
|
/// anything: `true` if no process currently holds it exclusively, `false`
|
||||||
/// immediately released), `false` if another process currently holds
|
/// if one does (this process included). For a caller that wants to warn
|
||||||
/// it. For a read-only caller that wants to warn about a build in
|
/// about a write in progress without waiting for it.
|
||||||
/// progress without itself becoming exclusive; use
|
|
||||||
/// [`acquire_lock`](Self::acquire_lock)/[`open_lock`](Self::open_lock)
|
|
||||||
/// to actually hold it.
|
|
||||||
pub fn test_lock(&self) -> bool {
|
pub fn test_lock(&self) -> bool {
|
||||||
matches!(obisys::DirLock::try_acquire(&self.root_path), Ok(Some(_)))
|
self.lock.is_free_of_writer()
|
||||||
}
|
|
||||||
|
|
||||||
/// Release this index's directory lock, if held. A no-op if it isn't.
|
|
||||||
pub fn release_lock(&self) {
|
|
||||||
*self.lock.lock().unwrap() = None;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Return `true` if `path` points to a valid obikmer index directory.
|
/// Return `true` if `path` points to a valid obikmer index directory.
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
[package]
|
[package]
|
||||||
name = "obikmer"
|
name = "obikmer"
|
||||||
version = "1.4.0"
|
version = "1.4.1"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
|
|
||||||
[[bin]]
|
[[bin]]
|
||||||
|
|||||||
@@ -93,7 +93,11 @@ fn run_annotate(args: &AnnotateArgs) {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let idx = open_index(&args.index);
|
// Rewrites the index metadata in place: exclusive for the whole command.
|
||||||
|
let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
|
||||||
|
eprintln!("error opening index: {e}");
|
||||||
|
std::process::exit(1);
|
||||||
|
});
|
||||||
|
|
||||||
let mut genomes = idx.meta().genomes().unwrap_or_else(|e| {
|
let mut genomes = idx.meta().genomes().unwrap_or_else(|e| {
|
||||||
eprintln!("error reading index metadata: {e}");
|
eprintln!("error reading index metadata: {e}");
|
||||||
|
|||||||
@@ -70,14 +70,8 @@ struct ModeArgs {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub fn run(args: ConvertArgs) {
|
pub fn run(args: ConvertArgs) {
|
||||||
// Modifies the index in place; acquired before opening so a concurrent
|
// Modifies the index in place: exclusive for the whole command.
|
||||||
// writer can't slip in between the open and the convert below.
|
let idx = KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
|
||||||
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
|
|
||||||
eprintln!("error locking index directory {}: {e}", args.index.display());
|
|
||||||
std::process::exit(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
let idx = KmerIndex::open(&args.index).unwrap_or_else(|e| {
|
|
||||||
eprintln!("error opening index: {e}");
|
eprintln!("error opening index: {e}");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -26,14 +26,8 @@ pub struct PackArgs {
|
|||||||
}
|
}
|
||||||
|
|
||||||
pub fn run(args: PackArgs) {
|
pub fn run(args: PackArgs) {
|
||||||
// Modifies the index in place; acquired before opening so a concurrent
|
// Modifies the index in place: exclusive for the whole command.
|
||||||
// writer can't slip in between the open and the pack below.
|
let idx = Arc::new(KmerIndex::open_lock(&args.index).unwrap_or_else(|e| {
|
||||||
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
|
|
||||||
eprintln!("error locking index directory {}: {e}", args.index.display());
|
|
||||||
std::process::exit(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
let idx = Arc::new(KmerIndex::open(&args.index).unwrap_or_else(|e| {
|
|
||||||
eprintln!("error opening index: {e}");
|
eprintln!("error opening index: {e}");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
}));
|
}));
|
||||||
|
|||||||
@@ -78,13 +78,9 @@ pub fn run(args: PhyloArgs) {
|
|||||||
// Meant to be (re)built routinely, ahead of any SNP-family distance
|
// Meant to be (re)built routinely, ahead of any SNP-family distance
|
||||||
// computation that will later consume it.
|
// computation that will later consume it.
|
||||||
if args.sibling_annex {
|
if args.sibling_annex {
|
||||||
// Writes into the index directory — hold an exclusive lock for the
|
// Writes into the index directory; `build_sibling_annex` holds the
|
||||||
// duration so a second, concurrent `--sibling-annex` run on the
|
// index's exclusive lock for the duration (`KmerIndex::with_lock`),
|
||||||
// same index can't corrupt these writes (see obisys::DirLock).
|
// so a concurrent run on the same index can't corrupt these writes.
|
||||||
let _lock = obisys::DirLock::acquire(&args.index).unwrap_or_else(|e| {
|
|
||||||
eprintln!("error locking index directory {}: {e}", args.index.display());
|
|
||||||
std::process::exit(1);
|
|
||||||
});
|
|
||||||
info!("building sibling-count/minorant annex");
|
info!("building sibling-count/minorant annex");
|
||||||
let t = Stage::start("sibling_annex");
|
let t = Stage::start("sibling_annex");
|
||||||
cache.build_sibling_annex().unwrap_or_else(|e| {
|
cache.build_sibling_annex().unwrap_or_else(|e| {
|
||||||
|
|||||||
@@ -53,7 +53,9 @@ pub(super) fn run_bits_per_kmer(index_path: &PathBuf) {
|
|||||||
pub(super) fn run_rename(index_path: &PathBuf, spec: &str) {
|
pub(super) fn run_rename(index_path: &PathBuf, spec: &str) {
|
||||||
let (old_label, new_label) = parse_rename_spec(spec);
|
let (old_label, new_label) = parse_rename_spec(spec);
|
||||||
|
|
||||||
let idx = KmerIndex::open(index_path).unwrap_or_else(|e| {
|
// Rewrites the index metadata and renames a spectrum file in place:
|
||||||
|
// exclusive for the whole command.
|
||||||
|
let idx = KmerIndex::open_lock(index_path).unwrap_or_else(|e| {
|
||||||
eprintln!("error opening index: {e}");
|
eprintln!("error opening index: {e}");
|
||||||
std::process::exit(1);
|
std::process::exit(1);
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -17,6 +17,7 @@ use super::entropy::{family_entropy, family_entropy_4};
|
|||||||
use super::family_scan::{Selection, scan_layer_families};
|
use super::family_scan::{Selection, scan_layer_families};
|
||||||
use super::is_fast_mode;
|
use super::is_fast_mode;
|
||||||
use super::minorant_selection::non_monomorphic_reindex_layer;
|
use super::minorant_selection::non_monomorphic_reindex_layer;
|
||||||
|
use crate::siblings::extensions::SiblingBuilder;
|
||||||
|
|
||||||
/// One variable family, as `--shannon` reports it.
|
/// One variable family, as `--shannon` reports it.
|
||||||
pub(crate) struct FamilyEntropyRow {
|
pub(crate) struct FamilyEntropyRow {
|
||||||
@@ -45,6 +46,7 @@ pub(crate) fn scan_family_entropies(
|
|||||||
cache: &IndexCache,
|
cache: &IndexCache,
|
||||||
mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>,
|
mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>,
|
||||||
) -> OKIResult<()> {
|
) -> OKIResult<()> {
|
||||||
|
cache.ensure_sibling_annex()?;
|
||||||
let n_genomes = cache.meta().genomes().len();
|
let n_genomes = cache.meta().genomes().len();
|
||||||
let fast_mode = is_fast_mode(cache);
|
let fast_mode = is_fast_mode(cache);
|
||||||
let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions");
|
let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions");
|
||||||
|
|||||||
@@ -22,7 +22,9 @@ use super::masking::is_quartet_informative;
|
|||||||
use super::minorant_selection::non_monomorphic_reindex_layer;
|
use super::minorant_selection::non_monomorphic_reindex_layer;
|
||||||
use super::pair_weight::pair_balance_weight;
|
use super::pair_weight::pair_balance_weight;
|
||||||
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
||||||
use crate::siblings::{FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind};
|
use crate::siblings::{
|
||||||
|
FamilyOverlapAccumulator, FamilyScalarAnnexBuilder, ScalarKind, family_overlap_file_name,
|
||||||
|
};
|
||||||
|
|
||||||
/// Which annexes a caller needs, each listed once per `--no-ambiguity`
|
/// Which annexes a caller needs, each listed once per `--no-ambiguity`
|
||||||
/// counting convention (`false` = genomes carrying several members count as a
|
/// counting convention (`false` = genomes carrying several members count as a
|
||||||
@@ -50,6 +52,24 @@ impl AnnexRequest {
|
|||||||
dedup(self.overlap.iter().chain(&self.pair_weight).copied())
|
dedup(self.overlap.iter().chain(&self.pair_weight).copied())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Whether every annex this request asks for already exists — the same
|
||||||
|
/// per-file existence test the builders apply, so `true` means they would
|
||||||
|
/// find nothing to build and no lock is needed. Cheap (no file is opened).
|
||||||
|
pub(crate) fn is_satisfied(&self, cache: &IndexCache) -> bool {
|
||||||
|
let overlap = self
|
||||||
|
.overlap_conventions()
|
||||||
|
.into_iter()
|
||||||
|
.all(|nb| cache.index().dir().join(family_overlap_file_name(nb)).exists());
|
||||||
|
let pair_weight = self.pair_weight_conventions();
|
||||||
|
overlap
|
||||||
|
&& cache.iter_indexed().all(|(_, _, layer)| {
|
||||||
|
let has = |name: &str| layer.dir().join(name).exists();
|
||||||
|
self.entropy.iter().all(|&nb| has(ScalarKind::Entropy.file_name(nb)))
|
||||||
|
&& self.quartet.iter().all(|&nb| has(quartet_annex_file_name(nb)))
|
||||||
|
&& pair_weight.iter().all(|&nb| has(ScalarKind::PairWeight.file_name(nb)))
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// The pair-weight conventions requested, each once.
|
/// The pair-weight conventions requested, each once.
|
||||||
pub(crate) fn pair_weight_conventions(&self) -> Vec<bool> {
|
pub(crate) fn pair_weight_conventions(&self) -> Vec<bool> {
|
||||||
dedup(self.pair_weight.iter().copied())
|
dedup(self.pair_weight.iter().copied())
|
||||||
|
|||||||
@@ -54,7 +54,7 @@ use obipipeline::ThrottleGuard;
|
|||||||
|
|
||||||
use crate::siblings::helpers::central_base;
|
use crate::siblings::helpers::central_base;
|
||||||
use crate::siblings::iter::{SiblingEntry, SiblingLayerExt};
|
use crate::siblings::iter::{SiblingEntry, SiblingLayerExt};
|
||||||
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, SiblingAnnex};
|
use crate::siblings::{FamilyMask, SiblingAnnex};
|
||||||
|
|
||||||
/// Target memory budget for one batch's `genome_mask` allocation
|
/// Target memory budget for one batch's `genome_mask` allocation
|
||||||
/// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding
|
/// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding
|
||||||
@@ -180,7 +180,7 @@ pub(crate) fn scan_layer_families(
|
|||||||
})?;
|
})?;
|
||||||
let n_cols = mat.n_cols().min(n_genomes);
|
let n_cols = mat.n_cols().min(n_genomes);
|
||||||
let layer_dir: &Path = mat.dir();
|
let layer_dir: &Path = mat.dir();
|
||||||
let annex = Arc::new(SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?);
|
let annex = Arc::new(SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?);
|
||||||
|
|
||||||
let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k });
|
let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k });
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ use std::path::Path;
|
|||||||
use obicompactvec::TempBitVecBuilder;
|
use obicompactvec::TempBitVecBuilder;
|
||||||
use obikindex::{OKIError, OKIResult};
|
use obikindex::{OKIError, OKIResult};
|
||||||
|
|
||||||
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
|
use crate::siblings::{FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
|
||||||
|
|
||||||
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
|
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
|
||||||
mask.is_minorant() && mask.siblings() >= 1
|
mask.is_minorant() && mask.siblings() >= 1
|
||||||
@@ -31,7 +31,7 @@ pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
|
|||||||
/// the `Selection` a caller needs to bound `scan_layer_families` to just
|
/// the `Selection` a caller needs to bound `scan_layer_families` to just
|
||||||
/// these families.
|
/// these families.
|
||||||
pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> {
|
pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> {
|
||||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||||
let mut reindex = HashMap::new();
|
let mut reindex = HashMap::new();
|
||||||
let mut family_idx = 0usize;
|
let mut family_idx = 0usize;
|
||||||
for slot in 0..annex.len() {
|
for slot in 0..annex.len() {
|
||||||
@@ -62,7 +62,7 @@ pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashM
|
|||||||
/// first counts this layer's total minorants (needed up front to size the
|
/// first counts this layer's total minorants (needed up front to size the
|
||||||
/// bitset), the second sets the eligible bits.
|
/// bitset), the second sets the eligible bits.
|
||||||
pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> {
|
pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> {
|
||||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||||
|
|
||||||
let n_minorants = (0..annex.len())
|
let n_minorants = (0..annex.len())
|
||||||
.filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant()))
|
.filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant()))
|
||||||
@@ -103,7 +103,7 @@ pub(crate) fn iter_full_scalar(
|
|||||||
kind: ScalarKind,
|
kind: ScalarKind,
|
||||||
no_ambiguity: bool,
|
no_ambiguity: bool,
|
||||||
) -> OKIResult<impl Iterator<Item = f32> + use<>> {
|
) -> OKIResult<impl Iterator<Item = f32> + use<>> {
|
||||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||||
let scalars =
|
let scalars =
|
||||||
FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?;
|
FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?;
|
||||||
if scalars.no_ambiguity() != no_ambiguity {
|
if scalars.no_ambiguity() != no_ambiguity {
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ use obikindex::{OKIError, OKIResult};
|
|||||||
|
|
||||||
use super::minorant_selection::is_non_monomorphic_minorant;
|
use super::minorant_selection::is_non_monomorphic_minorant;
|
||||||
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
||||||
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
|
use crate::siblings::SiblingAnnex;
|
||||||
|
|
||||||
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
|
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
|
||||||
/// included), every family the quartet annex flags as not quartet-informative
|
/// included), every family the quartet annex flags as not quartet-informative
|
||||||
@@ -22,7 +22,7 @@ pub(crate) fn restrict_to_quartet(
|
|||||||
layer_dir: &Path,
|
layer_dir: &Path,
|
||||||
no_ambiguity: bool,
|
no_ambiguity: bool,
|
||||||
) -> OKIResult<()> {
|
) -> OKIResult<()> {
|
||||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||||
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity)))
|
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity)))
|
||||||
.map_err(OKIError::Io)?;
|
.map_err(OKIError::Io)?;
|
||||||
|
|
||||||
|
|||||||
@@ -9,11 +9,12 @@
|
|||||||
//! already stored — no re-derivation, no lookup).
|
//! already stored — no re-derivation, no lookup).
|
||||||
|
|
||||||
use obikidxcache::index_cache::IndexCache;
|
use obikidxcache::index_cache::IndexCache;
|
||||||
use obikindex::{OKIError, OKIResult};
|
use obikindex::OKIResult;
|
||||||
|
|
||||||
use super::family_scan::{Selection, scan_layer_families};
|
use super::family_scan::{Selection, scan_layer_families};
|
||||||
use super::is_fast_mode;
|
use super::is_fast_mode;
|
||||||
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
|
use crate::siblings::SiblingAnnex;
|
||||||
|
use crate::siblings::extensions::SiblingBuilder;
|
||||||
|
|
||||||
/// Layers scanned at once per [`sibling_family_size_histogram`]'s
|
/// Layers scanned at once per [`sibling_family_size_histogram`]'s
|
||||||
/// `par_map_reduce` call — same bounding reasoning as
|
/// `par_map_reduce` call — same bounding reasoning as
|
||||||
@@ -45,26 +46,17 @@ pub struct SiblingAnnexStats {
|
|||||||
/// on top of that would fight it for cache/mmap locality) — this function
|
/// on top of that would fight it for cache/mmap locality) — this function
|
||||||
/// has no such inner parallelism to protect.
|
/// has no such inner parallelism to protect.
|
||||||
///
|
///
|
||||||
/// Existence of every cached layer's annex file is checked up front, before
|
/// A missing annex is built up front (`ensure_sibling_annex`), before the
|
||||||
/// the parallel map runs, so a missing one is reported as an error rather
|
/// parallel map runs, so none is silently read back as zero.
|
||||||
/// than silently read back as zero.
|
|
||||||
pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> {
|
pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> {
|
||||||
for (part, l, layer) in cache.iter_indexed() {
|
cache.ensure_sibling_annex()?;
|
||||||
let path = layer.dir().join(ANNEX_FILE_NAME);
|
|
||||||
if !path.exists() {
|
|
||||||
return Err(OKIError::InvalidInput(format!(
|
|
||||||
"no sibling annex at {} (partition {part}, layer {l}) — run --sibling-annex first",
|
|
||||||
path.display()
|
|
||||||
)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
Ok(cache.par_map_reduce(
|
Ok(cache.par_map_reduce(
|
||||||
MAX_CONCURRENT_LAYERS,
|
MAX_CONCURRENT_LAYERS,
|
||||||
|| [0u64; 4],
|
|| [0u64; 4],
|
||||||
|layer| {
|
|layer| {
|
||||||
let annex = SiblingAnnex::open(&layer.dir().join(ANNEX_FILE_NAME))
|
let annex = SiblingAnnex::open_in(layer.dir())
|
||||||
.expect("existence already checked before this parallel map ran");
|
.expect("ensure_sibling_annex already made every layer's annex exist");
|
||||||
let mut counts = [0u64; 4];
|
let mut counts = [0u64; 4];
|
||||||
for slot in 0..annex.len() {
|
for slot in 0..annex.len() {
|
||||||
let Some(mask) = annex.get(slot) else { continue };
|
let Some(mask) = annex.get(slot) else { continue };
|
||||||
@@ -95,6 +87,7 @@ pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u6
|
|||||||
/// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/
|
/// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/
|
||||||
/// `algorithms::scan::sampled_scan` already follow).
|
/// `algorithms::scan::sampled_scan` already follow).
|
||||||
pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> {
|
pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> {
|
||||||
|
cache.ensure_sibling_annex()?;
|
||||||
let n_genomes = cache.meta().genomes().len();
|
let n_genomes = cache.meta().genomes().len();
|
||||||
let fast_mode = is_fast_mode(cache);
|
let fast_mode = is_fast_mode(cache);
|
||||||
|
|
||||||
|
|||||||
@@ -241,6 +241,9 @@ pub(crate) fn sample_index(
|
|||||||
skip_layers: usize,
|
skip_layers: usize,
|
||||||
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
|
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
|
||||||
) -> OKIResult<SamplePlan> {
|
) -> OKIResult<SamplePlan> {
|
||||||
|
// Every selection below reads the sibling annex (`eligible_bitset_layer`),
|
||||||
|
// whichever derived annexes the options ask for.
|
||||||
|
cache.ensure_sibling_annex()?;
|
||||||
let n_genomes = cache.meta().genomes().len();
|
let n_genomes = cache.meta().genomes().len();
|
||||||
let fast_mode = is_fast_mode(cache);
|
let fast_mode = is_fast_mode(cache);
|
||||||
|
|
||||||
|
|||||||
@@ -12,10 +12,13 @@ use obikindex::{OKIError, OKIResult};
|
|||||||
use obisys::progress_bar;
|
use obisys::progress_bar;
|
||||||
|
|
||||||
use crate::siblings::algorithms::{
|
use crate::siblings::algorithms::{
|
||||||
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes,
|
AnnexRequest, build_layer_sibling_annex, ensure_layer_family_annexes,
|
||||||
inverse_overlap_matrix, is_fast_mode,
|
ensure_layer_pair_weight_annexes, inverse_overlap_matrix, is_fast_mode,
|
||||||
|
};
|
||||||
|
use crate::siblings::{
|
||||||
|
ANNEX_FILE_NAME, FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, FamilyOverlapAccumulator,
|
||||||
|
QUARTET_ANNEX_FILE_NAMES, ScalarKind, family_overlap_file_name,
|
||||||
};
|
};
|
||||||
use crate::siblings::{FamilyOverlap, FamilyOverlapAccumulator, family_overlap_file_name};
|
|
||||||
|
|
||||||
pub(crate) trait SiblingBuilder {
|
pub(crate) trait SiblingBuilder {
|
||||||
/// Build, for every cached layer, the annexes `request` asks for that
|
/// Build, for every cached layer, the annexes `request` asks for that
|
||||||
@@ -26,13 +29,129 @@ pub(crate) trait SiblingBuilder {
|
|||||||
/// a missing entropy, quartet or overlap annex triggers one scan that
|
/// a missing entropy, quartet or overlap annex triggers one scan that
|
||||||
/// feeds every missing one, per `ensure_layer_family_annexes`'s own docs,
|
/// feeds every missing one, per `ensure_layer_family_annexes`'s own docs,
|
||||||
/// then a second scan builds the missing pair weights, which need the
|
/// then a second scan builds the missing pair weights, which need the
|
||||||
/// finished overlap matrix. A no-op, cheap existence-check pass, once
|
/// finished overlap matrix. Makes sure the sibling annex exists first
|
||||||
/// everything requested already exists.
|
/// ([`ensure_sibling_annex`](Self::ensure_sibling_annex)). A no-op, cheap
|
||||||
|
/// existence-check pass that takes no lock, once everything requested
|
||||||
|
/// already exists; otherwise the build runs under the index's exclusive
|
||||||
|
/// lock ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock)).
|
||||||
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
|
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
|
||||||
|
|
||||||
|
/// [`ensure_family_annexes`](Self::ensure_family_annexes)'s build step,
|
||||||
|
/// without the existence pre-check and without the lock: the caller
|
||||||
|
/// holds it.
|
||||||
|
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
|
||||||
|
|
||||||
|
/// Build the sibling annex (`siblings.psib`, every annex above is derived
|
||||||
|
/// from it) if any cached layer lacks it — the state of an index right
|
||||||
|
/// after `migrate-mphf`, which sets every annex aside. The build writes
|
||||||
|
/// into every layer, so it runs under the index's exclusive lock
|
||||||
|
/// ([`KmerIndex::with_lock`](obikindex::KmerIndex::with_lock), a no-op
|
||||||
|
/// when the caller already holds it); a process that waited for it finds
|
||||||
|
/// the annex built by the one that held it, and does nothing. A no-op,
|
||||||
|
/// cheap existence-check pass that takes no lock, once every layer has
|
||||||
|
/// its annex.
|
||||||
|
fn ensure_sibling_annex(&self) -> OKIResult<()>;
|
||||||
|
|
||||||
|
/// (Re)build the sibling annex of every cached layer, unconditionally,
|
||||||
|
/// removing the annexes derived from it (entropy, quartet, pair weight,
|
||||||
|
/// family overlap), which its renumbering makes stale. Takes no lock:
|
||||||
|
/// the caller holds it ([`SiblingExt::build_sibling_annex`], or
|
||||||
|
/// [`ensure_sibling_annex`](Self::ensure_sibling_annex)).
|
||||||
|
fn build_base_sibling_annex(&self) -> OKIResult<()>;
|
||||||
}
|
}
|
||||||
|
|
||||||
impl SiblingBuilder for IndexCache {
|
impl SiblingBuilder for IndexCache {
|
||||||
|
fn ensure_sibling_annex(&self) -> OKIResult<()> {
|
||||||
|
let missing = || self.iter_indexed().any(|(_, _, layer)| !layer.dir().join(ANNEX_FILE_NAME).exists());
|
||||||
|
if !missing() {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
self.index().with_lock(|| {
|
||||||
|
if !missing() {
|
||||||
|
return Ok(()); // built by the process we waited for
|
||||||
|
}
|
||||||
|
tracing::info!("sibling annex missing — building it");
|
||||||
|
self.build_base_sibling_annex()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_base_sibling_annex(&self) -> OKIResult<()> {
|
||||||
|
let n_parts = self.index().n_partitions();
|
||||||
|
let fast_mode = is_fast_mode(self);
|
||||||
|
if !fast_mode {
|
||||||
|
tracing::warn!(
|
||||||
|
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
|
||||||
|
disabled; consider compacting this index"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
|
||||||
|
let mut total_slots: u64 = 0;
|
||||||
|
for part in self.partitions() {
|
||||||
|
let n_layer = self.n_layer(part).unwrap_or(0);
|
||||||
|
let mut part_slots: u64 = 0;
|
||||||
|
for l in 0..n_layer {
|
||||||
|
let layer = self
|
||||||
|
.get_layer(part, l)
|
||||||
|
.expect("cache-consistent: n_layer(part) already bounds l");
|
||||||
|
// Invalidate this layer's entropy, quartet and pair-weight
|
||||||
|
// annexes, if any — they were built against the sibling
|
||||||
|
// annex's *previous* `family_idx` numbering
|
||||||
|
// (`non_monomorphic_reindex_layer` reads the sibling
|
||||||
|
// annex), and are stale once the new one exists.
|
||||||
|
// `ensure_family_annexes` only builds what's *missing*, so
|
||||||
|
// the old files must be removed here for them to be rebuilt
|
||||||
|
// rather than silently left stale. Removed *before* the new
|
||||||
|
// annex is built and published (atomically, at its end): a
|
||||||
|
// build killed in between leaves the previous annex with no
|
||||||
|
// derived files — still consistent, rebuilt on demand —
|
||||||
|
// never a new annex next to derived files computed against
|
||||||
|
// the old numbering.
|
||||||
|
for name in ScalarKind::Entropy
|
||||||
|
.file_names()
|
||||||
|
.into_iter()
|
||||||
|
.chain(ScalarKind::PairWeight.file_names())
|
||||||
|
.chain(QUARTET_ANNEX_FILE_NAMES)
|
||||||
|
{
|
||||||
|
let derived_path = layer.dir().join(name);
|
||||||
|
if derived_path.exists() {
|
||||||
|
std::fs::remove_file(&derived_path).map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
||||||
|
}
|
||||||
|
total_slots += part_slots;
|
||||||
|
pb.inc(1);
|
||||||
|
pb.set_message(format!(
|
||||||
|
"partition {part}: {part_slots} kmers ({total_slots} total)"
|
||||||
|
));
|
||||||
|
}
|
||||||
|
pb.finish_and_clear();
|
||||||
|
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
|
||||||
|
|
||||||
|
// The family-overlap annexes are index-level, not per-layer (see
|
||||||
|
// their own module docs) — any layer's sibling annex changing makes
|
||||||
|
// them all stale (and, with them, every pair weight computed from
|
||||||
|
// them, removed per layer above), so they are invalidated once here
|
||||||
|
// rather than per-layer above.
|
||||||
|
for name in FAMILY_OVERLAP_FILE_NAMES {
|
||||||
|
let overlap_path = self.index().dir().join(name);
|
||||||
|
if overlap_path.exists() {
|
||||||
|
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
|
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
|
||||||
|
self.ensure_sibling_annex()?;
|
||||||
|
if request.is_satisfied(self) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
self.index().with_lock(|| self.build_family_annexes(request))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
|
||||||
let n_genomes = self.meta().genomes().len();
|
let n_genomes = self.meta().genomes().len();
|
||||||
let fast_mode = is_fast_mode(self);
|
let fast_mode = is_fast_mode(self);
|
||||||
let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity));
|
let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity));
|
||||||
|
|||||||
@@ -11,17 +11,12 @@ use std::path::Path;
|
|||||||
|
|
||||||
use obikidxcache::index_cache::IndexCache;
|
use obikidxcache::index_cache::IndexCache;
|
||||||
use obikindex::{OKIError, OKIResult};
|
use obikindex::{OKIError, OKIResult};
|
||||||
use obisys::progress_bar;
|
|
||||||
|
|
||||||
use crate::siblings::algorithms::{
|
use crate::siblings::algorithms::{
|
||||||
AnnexRequest, EntropySummary, SiblingAnnexStats, build_layer_sibling_annex, is_fast_mode,
|
AnnexRequest, EntropySummary, SiblingAnnexStats, scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
|
||||||
scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
|
|
||||||
};
|
};
|
||||||
use crate::siblings::extensions::SiblingBuilder;
|
use crate::siblings::extensions::SiblingBuilder;
|
||||||
use crate::siblings::{
|
use crate::siblings::{FamilyOverlap, family_overlap_file_name};
|
||||||
FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, QUARTET_ANNEX_FILE_NAMES, ScalarKind,
|
|
||||||
family_overlap_file_name,
|
|
||||||
};
|
|
||||||
|
|
||||||
pub trait SiblingExt {
|
pub trait SiblingExt {
|
||||||
/// Build the sibling-count/minorant annex for every layer of every
|
/// Build the sibling-count/minorant annex for every layer of every
|
||||||
@@ -111,82 +106,18 @@ pub trait SiblingExt {
|
|||||||
|
|
||||||
impl SiblingExt for IndexCache {
|
impl SiblingExt for IndexCache {
|
||||||
fn build_sibling_annex(&self) -> OKIResult<()> {
|
fn build_sibling_annex(&self) -> OKIResult<()> {
|
||||||
let n_parts = self.index().n_partitions();
|
self.index().with_lock(|| {
|
||||||
let fast_mode = is_fast_mode(self);
|
self.build_base_sibling_annex()?;
|
||||||
if !fast_mode {
|
// Precompute the entropy, quartet, pair-weight and family-overlap
|
||||||
tracing::warn!(
|
// annexes too, now that every layer's fresh sibling annex is on
|
||||||
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
|
// disk (`non_monomorphic_reindex_layer`, which they depend on,
|
||||||
disabled; consider compacting this index"
|
// reads it) — same routine-rebuild footing as the sibling annex
|
||||||
);
|
// itself, rather than leaving them to be built lazily on first
|
||||||
}
|
// `--shannon`/`--entropy`/`--quartet`/`--pair-balance`/
|
||||||
|
// `--min-shared-family`/`--family-overlap` use.
|
||||||
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
|
tracing::info!("building entropy, quartet, pair-weight (both --no-ambiguity conventions) and family-overlap annexes");
|
||||||
let mut total_slots: u64 = 0;
|
self.ensure_family_annexes(&AnnexRequest::everything())
|
||||||
for part in self.partitions() {
|
})
|
||||||
let n_layer = self.n_layer(part).unwrap_or(0);
|
|
||||||
let mut part_slots: u64 = 0;
|
|
||||||
for l in 0..n_layer {
|
|
||||||
let layer = self
|
|
||||||
.get_layer(part, l)
|
|
||||||
.expect("cache-consistent: n_layer(part) already bounds l");
|
|
||||||
// Invalidate this layer's entropy and quartet annexes, if
|
|
||||||
// any — they were built against the sibling annex's
|
|
||||||
// *previous* `family_idx` numbering
|
|
||||||
// (`non_monomorphic_reindex_layer` reads the sibling
|
|
||||||
// annex), and are stale once the new one exists.
|
|
||||||
// `ensure_family_annexes` below only builds what's
|
|
||||||
// *missing*, so the old files must be removed here for
|
|
||||||
// them to be rebuilt rather than silently left stale.
|
|
||||||
// Removed *before* the new annex is built and published
|
|
||||||
// (atomically, at its end): a build killed in between
|
|
||||||
// leaves the previous annex with no derived files — still
|
|
||||||
// consistent, rebuilt on demand — never a new annex next to
|
|
||||||
// derived files computed against the old numbering.
|
|
||||||
for name in ScalarKind::Entropy
|
|
||||||
.file_names()
|
|
||||||
.into_iter()
|
|
||||||
.chain(ScalarKind::PairWeight.file_names())
|
|
||||||
.chain(QUARTET_ANNEX_FILE_NAMES)
|
|
||||||
{
|
|
||||||
let entropy_path = layer.dir().join(name);
|
|
||||||
if entropy_path.exists() {
|
|
||||||
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
|
||||||
}
|
|
||||||
total_slots += part_slots;
|
|
||||||
pb.inc(1);
|
|
||||||
pb.set_message(format!(
|
|
||||||
"partition {part}: {part_slots} kmers ({total_slots} total)"
|
|
||||||
));
|
|
||||||
}
|
|
||||||
pb.finish_and_clear();
|
|
||||||
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
|
|
||||||
|
|
||||||
// The family-overlap annexes are index-level, not per-layer (see
|
|
||||||
// their own module docs) — any layer's sibling annex changing makes
|
|
||||||
// them all stale (and, with them, every pair weight computed from
|
|
||||||
// them, removed per layer above), so they are invalidated once here
|
|
||||||
// rather than per-layer above.
|
|
||||||
for name in FAMILY_OVERLAP_FILE_NAMES {
|
|
||||||
let overlap_path = self.index().dir().join(name);
|
|
||||||
if overlap_path.exists() {
|
|
||||||
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Precompute the entropy and family-overlap annexes too, now that
|
|
||||||
// every layer's fresh sibling annex is on disk
|
|
||||||
// (`non_monomorphic_reindex_layer`, which both depend on, reads
|
|
||||||
// it) — same routine-rebuild footing as the sibling annex itself,
|
|
||||||
// rather than leaving them to be built lazily on first
|
|
||||||
// `--shannon`/`--entropy-bias`/`--min-shared-family`/
|
|
||||||
// `--family-overlap` use.
|
|
||||||
tracing::info!("building entropy and quartet (both --no-ambiguity conventions) and family-overlap annexes");
|
|
||||||
self.ensure_family_annexes(&AnnexRequest::everything())?;
|
|
||||||
|
|
||||||
Ok(())
|
|
||||||
}
|
}
|
||||||
|
|
||||||
fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> {
|
fn shannon_entropy_csv(&self, path: &Path) -> OKIResult<()> {
|
||||||
|
|||||||
@@ -269,6 +269,21 @@ impl SiblingAnnex {
|
|||||||
Ok(Self { mmap, n, path: path.to_path_buf() })
|
Ok(Self { mmap, n, path: path.to_path_buf() })
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The annex of the layer directory `layer_dir`. Same as
|
||||||
|
/// [`open`](Self::open), except that an absent file is reported with its
|
||||||
|
/// path and the way to build it, rather than a bare `NotFound` — the
|
||||||
|
/// state an index is in right after `migrate-mphf` set its annexes aside.
|
||||||
|
pub fn open_in(layer_dir: &Path) -> io::Result<Self> {
|
||||||
|
let path = layer_dir.join(super::ANNEX_FILE_NAME);
|
||||||
|
Self::open(&path).map_err(|e| match e.kind() {
|
||||||
|
io::ErrorKind::NotFound => io::Error::new(
|
||||||
|
io::ErrorKind::NotFound,
|
||||||
|
format!("no sibling annex at {} — build it with `phylo --sibling-annex`", path.display()),
|
||||||
|
),
|
||||||
|
_ => e,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
pub fn path(&self) -> &Path { &self.path }
|
pub fn path(&self) -> &Path { &self.path }
|
||||||
pub fn len(&self) -> usize { self.n }
|
pub fn len(&self) -> usize { self.n }
|
||||||
pub fn is_empty(&self) -> bool { self.n == 0 }
|
pub fn is_empty(&self) -> bool { self.n == 0 }
|
||||||
|
|||||||
@@ -3,6 +3,9 @@
|
|||||||
|
|
||||||
mod budget;
|
mod budget;
|
||||||
mod lock;
|
mod lock;
|
||||||
|
mod lock_slot;
|
||||||
|
#[cfg(test)]
|
||||||
|
mod lock_tests;
|
||||||
mod numa;
|
mod numa;
|
||||||
mod profiler;
|
mod profiler;
|
||||||
mod progress;
|
mod progress;
|
||||||
@@ -10,7 +13,8 @@ mod resources;
|
|||||||
mod stage;
|
mod stage;
|
||||||
|
|
||||||
pub use budget::MemoryBudget;
|
pub use budget::MemoryBudget;
|
||||||
pub use lock::DirLock;
|
pub use lock::{DirLock, LockMode};
|
||||||
|
pub use lock_slot::LockSlot;
|
||||||
pub use numa::PartitionRunner;
|
pub use numa::PartitionRunner;
|
||||||
pub use profiler::start as start_profiler;
|
pub use profiler::start as start_profiler;
|
||||||
pub use progress::{Progress, TracedBar, progress_bar, spinner};
|
pub use progress::{Progress, TracedBar, progress_bar, spinner};
|
||||||
|
|||||||
+188
-38
@@ -1,56 +1,206 @@
|
|||||||
|
use std::fs::File;
|
||||||
|
use std::io;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
use tracing::info;
|
use tracing::info;
|
||||||
|
|
||||||
/// Exclusive advisory lock on an index directory, held for the duration of
|
/// Advisory readers/writer lock on an index directory, shared between
|
||||||
/// any command that writes into an already-existing index (building the
|
/// processes (and between independent handles of one process).
|
||||||
/// sibling annex, merging into a destination, filtering/selecting in place,
|
|
||||||
/// ...). Two such commands racing on the same directory can otherwise
|
|
||||||
/// corrupt each other's writes with no error from either side.
|
|
||||||
///
|
///
|
||||||
/// Only the directory actually being *written to* needs a lock — a command
|
/// - **Shared** ([`acquire_shared`](DirLock::acquire_shared)): held by
|
||||||
/// like `merge` that reads several source indexes to build one destination
|
/// anything that reads the index (`query`, `dump`, a `merge` source, a
|
||||||
/// only needs to lock the destination.
|
/// `phylo` run, ...). Any number of shared holders coexist.
|
||||||
|
/// - **Exclusive** ([`acquire`](DirLock::acquire)): held by anything that
|
||||||
|
/// writes into an already-existing index (building annexes, `convert`,
|
||||||
|
/// `pack`, `annotate`, ...). It waits for every shared holder to leave and
|
||||||
|
/// keeps others out while held. Two writers racing on one directory can
|
||||||
|
/// otherwise corrupt each other's files with no error from either side,
|
||||||
|
/// and a reader would see files vanish under it.
|
||||||
///
|
///
|
||||||
/// Uses the OS's advisory file lock (`flock` on Unix, `LockFileEx` on
|
/// Only the directory actually being *written to* needs an exclusive lock —
|
||||||
/// Windows) via `std::fs::File::lock`/`try_lock`, not a hand-rolled PID
|
/// `merge` holds shared locks on its sources and an exclusive one on its
|
||||||
/// file: the OS releases it automatically on process exit, including a
|
/// destination.
|
||||||
/// crash — no stale-lock cleanup logic needed.
|
///
|
||||||
|
/// # How it is built
|
||||||
|
///
|
||||||
|
/// Each mode is one OS advisory file lock (`flock` on Unix, `LockFileEx` on
|
||||||
|
/// Windows) on `.obikmer.lock`, so a crash releases it with no stale-lock
|
||||||
|
/// cleanup. On top of that, **every transition** (acquire, release into
|
||||||
|
/// another mode, promote, demote) happens while holding a *gate* — a second
|
||||||
|
/// lock file, `.obikmer.gate`, taken for the length of one non-blocking
|
||||||
|
/// attempt and never while waiting. A blocked request therefore releases the
|
||||||
|
/// gate, sleeps, and tries again (polling, 20 ms growing to 500 ms), instead
|
||||||
|
/// of sleeping in the kernel while holding it. Because nobody else can touch
|
||||||
|
/// the lock state while the gate is held, a shared → exclusive
|
||||||
|
/// [promotion](DirLock::promote) cannot be overtaken by another writer, which
|
||||||
|
/// the OS's own `flock` conversion does not guarantee.
|
||||||
|
///
|
||||||
|
/// There is no writer preference: a steady stream of readers can delay an
|
||||||
|
/// exclusive request.
|
||||||
#[derive(Debug)]
|
#[derive(Debug)]
|
||||||
pub struct DirLock {
|
pub struct DirLock {
|
||||||
_file: std::fs::File,
|
file: File,
|
||||||
|
dir: PathBuf,
|
||||||
|
mode: LockMode,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum LockMode {
|
||||||
|
Shared,
|
||||||
|
Exclusive,
|
||||||
|
}
|
||||||
|
|
||||||
|
const LOCK_FILE: &str = ".obikmer.lock";
|
||||||
|
const GATE_FILE: &str = ".obikmer.gate";
|
||||||
|
const CLAIM_FILE: &str = ".obikmer.promote";
|
||||||
|
const POLL_MIN: Duration = Duration::from_millis(20);
|
||||||
|
const POLL_MAX: Duration = Duration::from_millis(500);
|
||||||
|
|
||||||
|
fn open_file(dir: &Path, name: &str) -> io::Result<File> {
|
||||||
|
std::fs::create_dir_all(dir)?;
|
||||||
|
std::fs::OpenOptions::new()
|
||||||
|
.create(true)
|
||||||
|
.truncate(false)
|
||||||
|
.write(true)
|
||||||
|
.open(dir.join(name))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The gate: blocks, but its holders only ever run instantaneous,
|
||||||
|
/// non-blocking operations, so the wait is negligible. Released on drop.
|
||||||
|
fn gate(dir: &Path) -> io::Result<File> {
|
||||||
|
let f = open_file(dir, GATE_FILE)?;
|
||||||
|
f.lock()?;
|
||||||
|
Ok(f)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Non-blocking attempt in `mode`; `Ok(false)` means "held by someone else".
|
||||||
|
fn try_lock(file: &File, mode: LockMode) -> io::Result<bool> {
|
||||||
|
let r = match mode {
|
||||||
|
LockMode::Exclusive => file.try_lock(),
|
||||||
|
LockMode::Shared => file.try_lock_shared(),
|
||||||
|
};
|
||||||
|
match r {
|
||||||
|
Ok(()) => Ok(true),
|
||||||
|
Err(std::fs::TryLockError::WouldBlock) => Ok(false),
|
||||||
|
Err(std::fs::TryLockError::Error(e)) => Err(e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Sleeps between two attempts and logs once, at the first one.
|
||||||
|
struct Backoff {
|
||||||
|
delay: Duration,
|
||||||
|
logged: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Backoff {
|
||||||
|
fn new() -> Self {
|
||||||
|
Self { delay: POLL_MIN, logged: false }
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wait(&mut self, dir: &Path, why: &str) {
|
||||||
|
if !self.logged {
|
||||||
|
info!(dir = %dir.display(), "{why}");
|
||||||
|
self.logged = true;
|
||||||
|
}
|
||||||
|
std::thread::sleep(self.delay);
|
||||||
|
self.delay = (self.delay * 2).min(POLL_MAX);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
impl DirLock {
|
impl DirLock {
|
||||||
/// Block until the exclusive lock on `dir` is acquired (creating `dir`
|
/// Block until the exclusive lock on `dir` is acquired (creating `dir`
|
||||||
/// and the lock file within it if needed). Logs once if the wait is
|
/// and the lock files within it if needed). Logs once if the wait is
|
||||||
/// non-trivial, so a blocked command doesn't look silently hung.
|
/// non-trivial, so a blocked command doesn't look silently hung.
|
||||||
pub fn acquire(dir: &std::path::Path) -> std::io::Result<Self> {
|
pub fn acquire(dir: &Path) -> io::Result<Self> {
|
||||||
let file = Self::open_lock_file(dir)?;
|
Self::acquire_mode(dir, LockMode::Exclusive, "waiting for other obikmer processes to release this index")
|
||||||
if file.try_lock().is_err() {
|
|
||||||
info!(dir = %dir.display(), "waiting for another obikmer process to release this index");
|
|
||||||
file.lock()?;
|
|
||||||
}
|
|
||||||
Ok(Self { _file: file })
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Non-blocking probe: `Ok(Some(lock))` if `dir`'s lock was free and is
|
/// Block until a shared lock on `dir` is acquired — until no exclusive
|
||||||
/// now held by the returned `DirLock`, `Ok(None)` if another process
|
/// holder remains. Logs once if the wait is non-trivial.
|
||||||
/// currently holds it (not an error — contention is the expected,
|
pub fn acquire_shared(dir: &Path) -> io::Result<Self> {
|
||||||
/// common case this exists to distinguish from a genuine I/O failure).
|
Self::acquire_mode(dir, LockMode::Shared, "waiting for an obikmer process writing this index")
|
||||||
pub fn try_acquire(dir: &std::path::Path) -> std::io::Result<Option<Self>> {
|
}
|
||||||
let file = Self::open_lock_file(dir)?;
|
|
||||||
match file.try_lock() {
|
fn acquire_mode(dir: &Path, mode: LockMode, why: &str) -> io::Result<Self> {
|
||||||
Ok(()) => Ok(Some(Self { _file: file })),
|
let file = open_file(dir, LOCK_FILE)?;
|
||||||
Err(_) => Ok(None),
|
let mut backoff = Backoff::new();
|
||||||
|
loop {
|
||||||
|
{
|
||||||
|
let _gate = gate(dir)?;
|
||||||
|
if try_lock(&file, mode)? {
|
||||||
|
return Ok(Self { file, dir: dir.to_owned(), mode });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
backoff.wait(dir, why);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn open_lock_file(dir: &std::path::Path) -> std::io::Result<std::fs::File> {
|
pub fn mode(&self) -> LockMode {
|
||||||
std::fs::create_dir_all(dir)?;
|
self.mode
|
||||||
let lock_path = dir.join(".obikmer.lock");
|
}
|
||||||
std::fs::OpenOptions::new()
|
|
||||||
.create(true)
|
/// Non-blocking probe: `Ok(true)` if no exclusive holder exists right
|
||||||
.truncate(false)
|
/// now (a shared lock could be taken), `Ok(false)` if one does. Retains
|
||||||
.write(true)
|
/// nothing either way.
|
||||||
.open(&lock_path)
|
pub fn is_free_of_writer(dir: &Path) -> io::Result<bool> {
|
||||||
|
let file = open_file(dir, LOCK_FILE)?;
|
||||||
|
let _gate = gate(dir)?;
|
||||||
|
try_lock(&file, LockMode::Shared)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Shared → exclusive. Returns the exclusive lock and whether the
|
||||||
|
/// promotion was **gapless**: `true` means the lock was never released,
|
||||||
|
/// so nothing in the directory can have changed since the shared lock
|
||||||
|
/// was held and whatever the caller read stays valid; `false` means the
|
||||||
|
/// shared lock had to be given up while waiting, so a writer may have
|
||||||
|
/// run in between and the caller must re-check what it relied on.
|
||||||
|
///
|
||||||
|
/// Gapless promotion is available to one promoter at a time (it holds
|
||||||
|
/// `.obikmer.promote` while it waits for the other readers to leave).
|
||||||
|
/// A second concurrent promoter releases its shared lock instead — two
|
||||||
|
/// promoters each keeping theirs while waiting for the other's would
|
||||||
|
/// wait for ever. An exclusive lock is returned unchanged.
|
||||||
|
pub fn promote(self) -> io::Result<(Self, bool)> {
|
||||||
|
if self.mode == LockMode::Exclusive {
|
||||||
|
return Ok((self, true));
|
||||||
|
}
|
||||||
|
let claim = open_file(&self.dir, CLAIM_FILE)?;
|
||||||
|
if !try_lock(&claim, LockMode::Exclusive)? {
|
||||||
|
let dir = self.dir.clone();
|
||||||
|
drop(self); // gives the shared lock up
|
||||||
|
return Ok((Self::acquire(&dir)?, false));
|
||||||
|
}
|
||||||
|
let mut backoff = Backoff::new();
|
||||||
|
loop {
|
||||||
|
{
|
||||||
|
let _gate = gate(&self.dir)?;
|
||||||
|
self.file.unlock()?;
|
||||||
|
if try_lock(&self.file, LockMode::Exclusive)? {
|
||||||
|
return Ok((Self { mode: LockMode::Exclusive, ..self }, true));
|
||||||
|
}
|
||||||
|
// Other readers remain. Nobody can hold the lock exclusively
|
||||||
|
// without the gate, so taking our shared lock back succeeds.
|
||||||
|
if !try_lock(&self.file, LockMode::Shared)? {
|
||||||
|
return Err(io::Error::other("lost the shared index lock during promotion"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
backoff.wait(&self.dir, "waiting for other readers of this index before writing to it");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Exclusive → shared, without ever leaving the lock free in between.
|
||||||
|
/// A shared lock is returned unchanged.
|
||||||
|
pub fn demote(self) -> io::Result<Self> {
|
||||||
|
if self.mode == LockMode::Shared {
|
||||||
|
return Ok(self);
|
||||||
|
}
|
||||||
|
let held_gate = gate(&self.dir)?;
|
||||||
|
self.file.unlock()?;
|
||||||
|
if !try_lock(&self.file, LockMode::Shared)? {
|
||||||
|
return Err(io::Error::other("could not take the shared index lock back after writing"));
|
||||||
|
}
|
||||||
|
drop(held_gate);
|
||||||
|
Ok(Self { mode: LockMode::Shared, ..self })
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
use std::io;
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
use std::sync::Mutex;
|
||||||
|
|
||||||
|
use crate::lock::{DirLock, LockMode};
|
||||||
|
|
||||||
|
/// The directory lock of one handle on an index: nothing, shared or
|
||||||
|
/// exclusive, and the transitions between them that a handle needs. What
|
||||||
|
/// `obikindex::KmerIndex` delegates its locking to, kept apart so that it can
|
||||||
|
/// be tested without building an index.
|
||||||
|
///
|
||||||
|
/// Interior mutability (`&self` methods): a handle is shared behind `Arc`.
|
||||||
|
/// Not for concurrent use from several threads of one handle — the first
|
||||||
|
/// section to finish would restore the lock under the others.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub struct LockSlot {
|
||||||
|
dir: PathBuf,
|
||||||
|
held: Mutex<Option<DirLock>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LockSlot {
|
||||||
|
/// A slot holding nothing (a handle that took no lock, or one that
|
||||||
|
/// reopens a directory its own builder already holds).
|
||||||
|
pub fn empty(dir: &Path) -> Self {
|
||||||
|
Self { dir: dir.to_owned(), held: Mutex::new(None) }
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A slot holding a shared lock on `dir`; blocks while a writer is at work.
|
||||||
|
pub fn shared(dir: &Path) -> io::Result<Self> {
|
||||||
|
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire_shared(dir)?)) })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A slot holding the exclusive lock on `dir`; blocks while anyone else holds it.
|
||||||
|
pub fn exclusive(dir: &Path) -> io::Result<Self> {
|
||||||
|
Ok(Self { dir: dir.to_owned(), held: Mutex::new(Some(DirLock::acquire(dir)?)) })
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Whether the slot holds the exclusive lock (a shared one is not write access).
|
||||||
|
pub fn holds_write_lock(&self) -> bool {
|
||||||
|
matches!(&*self.held.lock().unwrap(), Some(l) if l.mode() == LockMode::Exclusive)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hold the exclusive lock from now on: a no-op if already held that way,
|
||||||
|
/// a promotion of a shared lock, a plain acquisition of an empty slot.
|
||||||
|
pub fn acquire_exclusive(&self) -> io::Result<()> {
|
||||||
|
let held = self.held.lock().unwrap().take();
|
||||||
|
let lock = match held {
|
||||||
|
Some(l) => l.promote()?.0,
|
||||||
|
None => DirLock::acquire(&self.dir)?,
|
||||||
|
};
|
||||||
|
*self.held.lock().unwrap() = Some(lock);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Run `f` with the exclusive lock held:
|
||||||
|
/// - already exclusive: `f` runs as is (so sections nest);
|
||||||
|
/// - shared: promoted for the duration of `f`, then demoted back, so the
|
||||||
|
/// slot keeps the shared lock it came with. The promotion is gapless
|
||||||
|
/// when no other process promotes at the same time; otherwise other
|
||||||
|
/// writers may run between the shared lock being given up and the
|
||||||
|
/// exclusive one being taken. Either way **`f` must re-check, before
|
||||||
|
/// building, what it is about to build**, and the caller must not hold
|
||||||
|
/// anything read earlier that a writer could have replaced;
|
||||||
|
/// - empty: the lock is taken for `f` only.
|
||||||
|
///
|
||||||
|
/// Restored also if `f` fails; an I/O error taking or restoring the lock
|
||||||
|
/// is converted to `E`.
|
||||||
|
pub fn with_exclusive<T, E: From<io::Error>>(&self, f: impl FnOnce() -> Result<T, E>) -> Result<T, E> {
|
||||||
|
enum Before {
|
||||||
|
Exclusive,
|
||||||
|
Shared,
|
||||||
|
Nothing,
|
||||||
|
}
|
||||||
|
let held = self.held.lock().unwrap().take();
|
||||||
|
let (lock, before) = match held {
|
||||||
|
Some(l) if l.mode() == LockMode::Exclusive => (l, Before::Exclusive),
|
||||||
|
Some(l) => (l.promote()?.0, Before::Shared),
|
||||||
|
None => (DirLock::acquire(&self.dir)?, Before::Nothing),
|
||||||
|
};
|
||||||
|
*self.held.lock().unwrap() = Some(lock);
|
||||||
|
let out = f();
|
||||||
|
let lock = self.held.lock().unwrap().take();
|
||||||
|
match (before, lock) {
|
||||||
|
(Before::Exclusive, lock) => *self.held.lock().unwrap() = lock,
|
||||||
|
(Before::Shared, Some(l)) => *self.held.lock().unwrap() = Some(l.demote()?),
|
||||||
|
_ => {} // taken for `f` only: released here
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Non-blocking probe: `true` if no process holds the directory
|
||||||
|
/// exclusively right now (this slot included), `false` if one does.
|
||||||
|
pub fn is_free_of_writer(&self) -> bool {
|
||||||
|
DirLock::is_free_of_writer(&self.dir).unwrap_or(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,208 @@
|
|||||||
|
//! The directory lock is a readers/writer lock: `shared` slots coexist, an
|
||||||
|
//! `exclusive` one waits for them all and keeps new ones out, and a shared
|
||||||
|
//! slot can be promoted for a section (`with_exclusive`) and demoted back.
|
||||||
|
//! No index is involved, and therefore none of the process-wide k-mer
|
||||||
|
//! parameters other crates' tests share.
|
||||||
|
|
||||||
|
use std::io;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||||
|
use std::sync::mpsc;
|
||||||
|
use std::sync::{Arc, Barrier};
|
||||||
|
use std::thread;
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use crate::lock::{DirLock, LockMode};
|
||||||
|
use crate::lock_slot::LockSlot;
|
||||||
|
|
||||||
|
const SETTLE: Duration = Duration::from_millis(400);
|
||||||
|
const PATIENCE: Duration = Duration::from_secs(10);
|
||||||
|
|
||||||
|
/// A scratch directory removed on drop (no `tempfile` dependency here).
|
||||||
|
struct Scratch(PathBuf);
|
||||||
|
|
||||||
|
impl Scratch {
|
||||||
|
fn new() -> Self {
|
||||||
|
static N: AtomicUsize = AtomicUsize::new(0);
|
||||||
|
let dir = std::env::temp_dir().join(format!(
|
||||||
|
"obisys-lock-{}-{}",
|
||||||
|
std::process::id(),
|
||||||
|
N.fetch_add(1, Ordering::Relaxed)
|
||||||
|
));
|
||||||
|
std::fs::create_dir_all(&dir).unwrap();
|
||||||
|
Self(dir)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for Scratch {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
let _ = std::fs::remove_dir_all(&self.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn readers_coexist() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let a = LockSlot::shared(&d.0).unwrap();
|
||||||
|
let b = LockSlot::shared(&d.0).unwrap();
|
||||||
|
assert!(!a.holds_write_lock() && !b.holds_write_lock());
|
||||||
|
assert!(a.is_free_of_writer(), "readers are not writers");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn exclusive_is_exclusive_and_released_on_drop() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let w = LockSlot::exclusive(&d.0).unwrap();
|
||||||
|
assert!(w.holds_write_lock());
|
||||||
|
assert!(!w.is_free_of_writer());
|
||||||
|
drop(w);
|
||||||
|
assert!(LockSlot::empty(&d.0).is_free_of_writer());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_writer_waits_for_the_readers() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let reader = LockSlot::shared(&d.0).unwrap();
|
||||||
|
let (tx, rx) = mpsc::channel();
|
||||||
|
let dir = d.0.clone();
|
||||||
|
let writer = thread::spawn(move || {
|
||||||
|
drop(LockSlot::exclusive(&dir).unwrap());
|
||||||
|
tx.send(()).unwrap();
|
||||||
|
});
|
||||||
|
assert!(rx.recv_timeout(SETTLE).is_err(), "the writer must wait for the reader");
|
||||||
|
drop(reader);
|
||||||
|
rx.recv_timeout(PATIENCE).expect("the writer proceeds once the reader is gone");
|
||||||
|
writer.join().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_reader_waits_for_the_writer() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let writer = LockSlot::exclusive(&d.0).unwrap();
|
||||||
|
let (tx, rx) = mpsc::channel();
|
||||||
|
let dir = d.0.clone();
|
||||||
|
let reader = thread::spawn(move || {
|
||||||
|
drop(LockSlot::shared(&dir).unwrap());
|
||||||
|
tx.send(()).unwrap();
|
||||||
|
});
|
||||||
|
assert!(rx.recv_timeout(SETTLE).is_err(), "the reader must wait for the writer");
|
||||||
|
drop(writer);
|
||||||
|
rx.recv_timeout(PATIENCE).expect("the reader proceeds once the writer is gone");
|
||||||
|
reader.join().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn with_exclusive_on_a_reader_promotes_then_restores_the_shared_lock() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let slot = LockSlot::shared(&d.0).unwrap();
|
||||||
|
let out: io::Result<i32> = slot.with_exclusive(|| {
|
||||||
|
assert!(slot.holds_write_lock());
|
||||||
|
assert!(!slot.is_free_of_writer());
|
||||||
|
Ok(7)
|
||||||
|
});
|
||||||
|
assert_eq!(out.unwrap(), 7);
|
||||||
|
assert!(!slot.holds_write_lock());
|
||||||
|
assert!(slot.is_free_of_writer());
|
||||||
|
|
||||||
|
// Still a reader: a writer elsewhere waits for it.
|
||||||
|
let (tx, rx) = mpsc::channel();
|
||||||
|
let dir = d.0.clone();
|
||||||
|
let writer = thread::spawn(move || {
|
||||||
|
drop(LockSlot::exclusive(&dir).unwrap());
|
||||||
|
tx.send(()).unwrap();
|
||||||
|
});
|
||||||
|
assert!(rx.recv_timeout(SETTLE).is_err(), "the slot must be a reader again");
|
||||||
|
drop(slot);
|
||||||
|
rx.recv_timeout(PATIENCE).unwrap();
|
||||||
|
writer.join().unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn with_exclusive_on_an_empty_slot_holds_the_lock_for_the_section_only() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let slot = LockSlot::empty(&d.0);
|
||||||
|
slot.with_exclusive::<(), io::Error>(|| {
|
||||||
|
assert!(slot.holds_write_lock());
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
assert!(!slot.holds_write_lock());
|
||||||
|
assert!(slot.is_free_of_writer());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn with_exclusive_restores_on_error() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let slot = LockSlot::shared(&d.0).unwrap();
|
||||||
|
let r: io::Result<()> = slot.with_exclusive(|| Err(io::Error::other("boom")));
|
||||||
|
assert!(r.is_err());
|
||||||
|
assert!(!slot.holds_write_lock());
|
||||||
|
assert!(slot.is_free_of_writer());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn with_exclusive_nests_and_keeps_a_lock_taken_at_open() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let slot = LockSlot::exclusive(&d.0).unwrap();
|
||||||
|
slot.with_exclusive::<(), io::Error>(|| {
|
||||||
|
slot.with_exclusive::<(), io::Error>(|| Ok(()))?;
|
||||||
|
assert!(slot.holds_write_lock(), "an inner section must not release the outer lock");
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
assert!(slot.holds_write_lock(), "a slot opened exclusive keeps it");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn acquire_exclusive_promotes_a_shared_slot_and_is_idempotent() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let slot = LockSlot::shared(&d.0).unwrap();
|
||||||
|
slot.acquire_exclusive().unwrap();
|
||||||
|
slot.acquire_exclusive().unwrap();
|
||||||
|
assert!(slot.holds_write_lock());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The case a naive upgrade deadlocks on: two readers, each holding a shared
|
||||||
|
/// lock, both ask for the exclusive one at the same time.
|
||||||
|
#[test]
|
||||||
|
fn two_readers_promoting_at_once_both_finish() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let barrier = Arc::new(Barrier::new(2));
|
||||||
|
let (tx, rx) = mpsc::channel();
|
||||||
|
let handles: Vec<_> = (0..2)
|
||||||
|
.map(|_| {
|
||||||
|
let (dir, b, tx) = (d.0.clone(), Arc::clone(&barrier), tx.clone());
|
||||||
|
thread::spawn(move || {
|
||||||
|
let slot = LockSlot::shared(&dir).unwrap();
|
||||||
|
b.wait(); // both hold a shared lock
|
||||||
|
slot.with_exclusive::<(), io::Error>(|| {
|
||||||
|
assert!(slot.holds_write_lock());
|
||||||
|
thread::sleep(Duration::from_millis(50));
|
||||||
|
Ok(())
|
||||||
|
})
|
||||||
|
.unwrap();
|
||||||
|
tx.send(()).unwrap();
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
for _ in 0..2 {
|
||||||
|
rx.recv_timeout(PATIENCE).expect("a promotion deadlocked");
|
||||||
|
}
|
||||||
|
for h in handles {
|
||||||
|
h.join().unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_lone_promotion_is_gapless_and_a_demotion_keeps_the_lock() {
|
||||||
|
let d = Scratch::new();
|
||||||
|
let shared = DirLock::acquire_shared(&d.0).unwrap();
|
||||||
|
let (excl, gapless) = shared.promote().unwrap();
|
||||||
|
assert_eq!(excl.mode(), LockMode::Exclusive);
|
||||||
|
assert!(gapless, "nobody else was promoting: the lock was never released");
|
||||||
|
let shared = excl.demote().unwrap();
|
||||||
|
assert_eq!(shared.mode(), LockMode::Shared);
|
||||||
|
// Shared again: other readers get in, a writer does not.
|
||||||
|
let _other = DirLock::acquire_shared(&d.0).unwrap();
|
||||||
|
assert!(DirLock::is_free_of_writer(&d.0).unwrap());
|
||||||
|
}
|
||||||
Reference in new issue
Block a user