Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
This commit is contained in:
1 parent
8296e3544c
commit
7c7d7615cb
12 files changed
+177
-103
No files matched your search
@@ -329,7 +329,34 @@ Global family-size histogram (`global` row): 34 047 616 087 families; sizes 1 /
|
||||
|
||||
**`_dist.phy` (not asked for).** 180 × 180, symmetric, values 0.456-0.99999, median 0.999; Spearman with the Mash matrix 0.9999998 and with the index overlap −0.95, so it carries Mash-like information, consistent with the `jaccard` default (not verified). `args::distance_to_compute` at HEAD computes no matrix for this command line; why the file exists is unknown (binary predating the change, or a leftover; not checked). Not used above.
|
||||
|
||||
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment has not been given to IQ-TREE.
|
||||
**Not measured:** any tree-level effect (sCF, species monophyly, support at the interface nodes 209/216 where `sN` was 17 and 7); the separate contribution of entropy, quartet and pair balance (needs ablation runs: without `--entropy`, without `--pair-balance`, `--quartet` alone); variance between draws (one draw per setting); clade-stratified 4-way coverage; the new alignment had not been given to IQ-TREE (done 2026-10-16: see the last measured section).
|
||||
|
||||
## Tree-level effect of the combined draw: sCF on three trees (measured 2026-10-16)
|
||||
|
||||
Closes the "not measured" item of the previous section (the combined alignment given to IQ-TREE). Three trees, all sCF by `--scfl 100 --cf-verbose`, model `GTR+G4+ASC`:
|
||||
|
||||
- **A** = `concord` (12/10): entropy-only draw, 180 genomes, 5 003 414 columns; `-t` + `-m` re-ran a full ML search (59 h).
|
||||
- **B** = `concord_quartet_balance`: combined draw (`--entropy 1 --quartet --pair-balance`), 180 genomes, 5 000 000 columns. Its log stops during the tree search (14/10), so the command that produced its `cf.*` (16/10) is not in the files; presumably `-te` on the 15/10 treefile as for G (unverified).
|
||||
- **G** = `scf_good`: same draw as B without the 3 data-poor genomes (*A. pyrenaica* PHA000543, *A. saussurei*, *A. triflora*), 177 genomes. 72 sites became invariant and were removed to satisfy ASC; the search (`concord_quartet_balance_good`, 107 iterations, 6 h 47 on 120 threads) crashed in the sCF step on the FASTA (`Assertion all_prob_const < 1.0`; cause untested, ASC with constant sites is a suspect). sCF were recomputed with `-te <treefile> -s <varsites.phy> -m GTR+G4+ASC --scfl 100`: **no search** (0 iterations, lnL −41 053 676.53 vs −41 053 676.41 from the search), 4 999 729 columns. A used `-t`, B/G a fixed tree: the protocols differ.
|
||||
|
||||
| | A | B | G |
|
||||
|---|---|---|---|
|
||||
| Branches | 177 | 177 | 174 |
|
||||
| Mean `sN` | 3 219 | 13 163 | 13 396 |
|
||||
| Branches with `sN` < 100 / < 1 000 | 13 / 50 | 0 / 5 | 0 / 2 |
|
||||
| Median sCF | 44.2 | 42.9 | 42.8 |
|
||||
| Branches with sCF < 40 / ≥ 70 | 74 / 28 | 67 / 26 | 69 / 25 |
|
||||
| *Primula* and *Androsace* monophyletic | no / no | no / no | yes / yes |
|
||||
|
||||
**Sampling effect (A → B, same 180 genomes).** `sN` ×4.5 and the scarcity is gone. Sites do not become more concordant: on the 113 shared bipartitions (of 177) median sCF 48.0 → 47.3 (44 up, 69 down). Robinson-Foulds 128: the draw rearranged the tree deeply. Consistent with the earlier reading that the intra-*Androsace* conflict is not a matter of site count; the number of branches with sCF < 40 % stays around 70.
|
||||
|
||||
**Interface nodes.** Of the old nodes 209/211/212/216/217 (`sN` 17 / 226 / 184 / 7 / 7) none exists in G. Node 208 survives with `sN` 190 → 5 218 (sCF 54.5 → 37.5). Six mixed clades remain in G (83 to 72 tips, 50 *Primula* + 22-33 *Androsace*), all with `sN` > 3 000.
|
||||
|
||||
**Effect of the 3 genomes (B → G, same draw).** RF 16 (166 of 174 shared), median sCF 43.2 → 43.6, median `sN` unchanged. In B, *A. pyrenaica* and *A. triflora* sit inside *Primula* (a clade of 52 tips = 50 *Primula* + these 2; each has a *Primula* sister, sCF 92 % and 52 %, `sN` 556 and 153); *A. saussurei* stays with *Androsace*. Removing them restores both monophylies and changes nothing else measurably: support for "too little data to place them" (data-poor section), not for contamination, though the latter is not excluded.
|
||||
|
||||
**Where A and G disagree (2026-10-16).** On the 177 common tips: 124 shared bipartitions, 50 only in A, 50 only in G. Differing bipartitions are weakly supported in both trees (median sCF 35.7 in A-only, 37.7 in G-only, versus 47.5-48.1 on the shared ones), so the disagreement sits where support was already low. Differences are mostly local: the closest bipartition of the other tree is 1 tip away for 25 of 50 A-only and 21 of 50 G-only splits (≤ 2 tips: 37 and 30); only 8 and 9 are 5+ tips apart. Greedy removal of the tip that lowers RF most: the first (*P. auricula* subsp. *serratifolia*) takes RF from 100 to 86, then about 2 per tip (*A. maxima*, *P. stricta* var. *obesior*, then ordinary tips); the top three are data-poor genomes. A bipartition separates a data-poor genome (the 5 left after the 3 removals) in 1 of 124 shared splits but in 23 of 50 A-only and 10 of 50 G-only splits; removing those 5 genomes gives RF 72, against 96.6 (range 90-100, 40 draws) for 5 random tips. So about 28 of the 100 differences trace to data-poor genomes, and the remaining 72 do not. Backbone: A has a clade of 50 *Primula* + 25 *Androsace* (sCF 76, `sN` 453), G one of 50 + 26 (sCF 73, `sN` 6 677): one tip apart. Genuine conflict: a clade of 72 *Androsace* exists in both trees (A: sCF 34, `sN` 17 127; G: sCF 38, `sN` 49 473) with 16 different tips, i.e. a topological disagreement carried by tens of thousands of sites in each, not a scarcity effect; consistent with the intra-*Androsace* conflict above.
|
||||
|
||||
**Not established.** Separate contribution of entropy / quartet / pair balance (no ablation); stability of the tree between two independent draws (one draw per setting; A → B shows the topology moves with the draw); whether the quartet-informative restriction biases branch lengths (ASC corrects for variable-only ascertainment, not for requiring two states each in two genomes; tree length 22.5 is not interpretable as substitutions/site).
|
||||
|
||||
## Open questions
|
||||
|
||||
|
||||
@@ -48,7 +48,7 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex — prerequisite for every option in this section, and for a `snp-*` `--distance` value |
|
||||
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex, then every annex derived from it (entropy, quartet, pair weight, family overlap, under both `--no-ambiguity` conventions). Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
|
||||
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
|
||||
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
||||
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
||||
|
||||
Generated
+1
-1
@@ -1537,7 +1537,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "obikmer"
|
||||
version = "1.3.6"
|
||||
version = "1.4.0"
|
||||
dependencies = [
|
||||
"clap",
|
||||
"csv",
|
||||
|
||||
@@ -17,6 +17,7 @@ use super::entropy::{family_entropy, family_entropy_4};
|
||||
use super::family_scan::{Selection, scan_layer_families};
|
||||
use super::is_fast_mode;
|
||||
use super::minorant_selection::non_monomorphic_reindex_layer;
|
||||
use crate::siblings::extensions::SiblingBuilder;
|
||||
|
||||
/// One variable family, as `--shannon` reports it.
|
||||
pub(crate) struct FamilyEntropyRow {
|
||||
@@ -45,6 +46,7 @@ pub(crate) fn scan_family_entropies(
|
||||
cache: &IndexCache,
|
||||
mut visit: impl FnMut(&FamilyEntropyRow) -> io::Result<()>,
|
||||
) -> OKIResult<()> {
|
||||
cache.ensure_sibling_annex()?;
|
||||
let n_genomes = cache.meta().genomes().len();
|
||||
let fast_mode = is_fast_mode(cache);
|
||||
let pb = progress_bar("shannon_entropy", cache.n_partition() as u64, "partitions");
|
||||
|
||||
@@ -54,7 +54,7 @@ use obipipeline::ThrottleGuard;
|
||||
|
||||
use crate::siblings::helpers::central_base;
|
||||
use crate::siblings::iter::{SiblingEntry, SiblingLayerExt};
|
||||
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, SiblingAnnex};
|
||||
use crate::siblings::{FamilyMask, SiblingAnnex};
|
||||
|
||||
/// Target memory budget for one batch's `genome_mask` allocation
|
||||
/// (`family_batch_size(n_genomes) * n_genomes` bytes, see below) — bounding
|
||||
@@ -180,7 +180,7 @@ pub(crate) fn scan_layer_families(
|
||||
})?;
|
||||
let n_cols = mat.n_cols().min(n_genomes);
|
||||
let layer_dir: &Path = mat.dir();
|
||||
let annex = Arc::new(SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?);
|
||||
let annex = Arc::new(SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?);
|
||||
|
||||
let ctx = Arc::new(LayerCtx { mat, n_parts, n_genomes, n_cols, k });
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ use std::path::Path;
|
||||
use obicompactvec::TempBitVecBuilder;
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
|
||||
use crate::siblings::{ANNEX_FILE_NAME, FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
|
||||
use crate::siblings::{FamilyMask, FamilyScalarAnnex, ScalarKind, SiblingAnnex};
|
||||
|
||||
pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
|
||||
mask.is_minorant() && mask.siblings() >= 1
|
||||
@@ -31,7 +31,7 @@ pub(crate) fn is_non_monomorphic_minorant(mask: FamilyMask) -> bool {
|
||||
/// the `Selection` a caller needs to bound `scan_layer_families` to just
|
||||
/// these families.
|
||||
pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashMap<usize, usize>> {
|
||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
||||
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||
let mut reindex = HashMap::new();
|
||||
let mut family_idx = 0usize;
|
||||
for slot in 0..annex.len() {
|
||||
@@ -62,7 +62,7 @@ pub(crate) fn non_monomorphic_reindex_layer(layer_dir: &Path) -> OKIResult<HashM
|
||||
/// first counts this layer's total minorants (needed up front to size the
|
||||
/// bitset), the second sets the eligible bits.
|
||||
pub(crate) fn eligible_bitset_layer(layer_dir: &Path) -> OKIResult<TempBitVecBuilder> {
|
||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
||||
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||
|
||||
let n_minorants = (0..annex.len())
|
||||
.filter(|&slot| annex.get(slot).is_some_and(|m| m.is_minorant()))
|
||||
@@ -103,7 +103,7 @@ pub(crate) fn iter_full_scalar(
|
||||
kind: ScalarKind,
|
||||
no_ambiguity: bool,
|
||||
) -> OKIResult<impl Iterator<Item = f32> + use<>> {
|
||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
||||
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||
let scalars =
|
||||
FamilyScalarAnnex::open(&layer_dir.join(kind.file_name(no_ambiguity)), kind).map_err(OKIError::Io)?;
|
||||
if scalars.no_ambiguity() != no_ambiguity {
|
||||
|
||||
@@ -8,7 +8,7 @@ use obikindex::{OKIError, OKIResult};
|
||||
|
||||
use super::minorant_selection::is_non_monomorphic_minorant;
|
||||
use crate::siblings::quartet_annex::quartet_annex_file_name;
|
||||
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
|
||||
use crate::siblings::SiblingAnnex;
|
||||
|
||||
/// Clears, in `eligible` (indexed by `family_idx`, monomorphic minorants
|
||||
/// included), every family the quartet annex flags as not quartet-informative
|
||||
@@ -22,7 +22,7 @@ pub(crate) fn restrict_to_quartet(
|
||||
layer_dir: &Path,
|
||||
no_ambiguity: bool,
|
||||
) -> OKIResult<()> {
|
||||
let annex = SiblingAnnex::open(&layer_dir.join(ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
|
||||
let annex = SiblingAnnex::open_in(layer_dir).map_err(OKIError::Io)?;
|
||||
let flags = PersistentBitVec::open(&layer_dir.join(quartet_annex_file_name(no_ambiguity)))
|
||||
.map_err(OKIError::Io)?;
|
||||
|
||||
|
||||
@@ -9,11 +9,12 @@
|
||||
//! already stored — no re-derivation, no lookup).
|
||||
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
use obikindex::OKIResult;
|
||||
|
||||
use super::family_scan::{Selection, scan_layer_families};
|
||||
use super::is_fast_mode;
|
||||
use crate::siblings::{ANNEX_FILE_NAME, SiblingAnnex};
|
||||
use crate::siblings::SiblingAnnex;
|
||||
use crate::siblings::extensions::SiblingBuilder;
|
||||
|
||||
/// Layers scanned at once per [`sibling_family_size_histogram`]'s
|
||||
/// `par_map_reduce` call — same bounding reasoning as
|
||||
@@ -45,26 +46,17 @@ pub struct SiblingAnnexStats {
|
||||
/// on top of that would fight it for cache/mmap locality) — this function
|
||||
/// has no such inner parallelism to protect.
|
||||
///
|
||||
/// Existence of every cached layer's annex file is checked up front, before
|
||||
/// the parallel map runs, so a missing one is reported as an error rather
|
||||
/// than silently read back as zero.
|
||||
/// A missing annex is built up front (`ensure_sibling_annex`), before the
|
||||
/// parallel map runs, so none is silently read back as zero.
|
||||
pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u64; 4]> {
|
||||
for (part, l, layer) in cache.iter_indexed() {
|
||||
let path = layer.dir().join(ANNEX_FILE_NAME);
|
||||
if !path.exists() {
|
||||
return Err(OKIError::InvalidInput(format!(
|
||||
"no sibling annex at {} (partition {part}, layer {l}) — run --sibling-annex first",
|
||||
path.display()
|
||||
)));
|
||||
}
|
||||
}
|
||||
cache.ensure_sibling_annex()?;
|
||||
|
||||
Ok(cache.par_map_reduce(
|
||||
MAX_CONCURRENT_LAYERS,
|
||||
|| [0u64; 4],
|
||||
|layer| {
|
||||
let annex = SiblingAnnex::open(&layer.dir().join(ANNEX_FILE_NAME))
|
||||
.expect("existence already checked before this parallel map ran");
|
||||
let annex = SiblingAnnex::open_in(layer.dir())
|
||||
.expect("ensure_sibling_annex already made every layer's annex exist");
|
||||
let mut counts = [0u64; 4];
|
||||
for slot in 0..annex.len() {
|
||||
let Some(mask) = annex.get(slot) else { continue };
|
||||
@@ -95,6 +87,7 @@ pub(crate) fn sibling_family_size_histogram(cache: &IndexCache) -> OKIResult<[u6
|
||||
/// same reasoning `SiblingExt::build_sibling_annex`/`shannon_entropy_csv`/
|
||||
/// `algorithms::scan::sampled_scan` already follow).
|
||||
pub(crate) fn sibling_annex_stats(cache: &IndexCache) -> OKIResult<SiblingAnnexStats> {
|
||||
cache.ensure_sibling_annex()?;
|
||||
let n_genomes = cache.meta().genomes().len();
|
||||
let fast_mode = is_fast_mode(cache);
|
||||
|
||||
|
||||
@@ -241,6 +241,9 @@ pub(crate) fn sample_index(
|
||||
skip_layers: usize,
|
||||
mut on_layer: impl FnMut(usize, usize, usize, Arc<Vec<SurvivingFamily>>),
|
||||
) -> OKIResult<SamplePlan> {
|
||||
// Every selection below reads the sibling annex (`eligible_bitset_layer`),
|
||||
// whichever derived annexes the options ask for.
|
||||
cache.ensure_sibling_annex()?;
|
||||
let n_genomes = cache.meta().genomes().len();
|
||||
let fast_mode = is_fast_mode(cache);
|
||||
|
||||
|
||||
@@ -9,13 +9,16 @@
|
||||
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
use obisys::progress_bar;
|
||||
use obisys::{DirLock, progress_bar};
|
||||
|
||||
use crate::siblings::algorithms::{
|
||||
AnnexRequest, ensure_layer_family_annexes, ensure_layer_pair_weight_annexes,
|
||||
inverse_overlap_matrix, is_fast_mode,
|
||||
AnnexRequest, build_layer_sibling_annex, ensure_layer_family_annexes,
|
||||
ensure_layer_pair_weight_annexes, inverse_overlap_matrix, is_fast_mode,
|
||||
};
|
||||
use crate::siblings::{
|
||||
ANNEX_FILE_NAME, FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, FamilyOverlapAccumulator,
|
||||
QUARTET_ANNEX_FILE_NAMES, ScalarKind, family_overlap_file_name,
|
||||
};
|
||||
use crate::siblings::{FamilyOverlap, FamilyOverlapAccumulator, family_overlap_file_name};
|
||||
|
||||
pub(crate) trait SiblingBuilder {
|
||||
/// Build, for every cached layer, the annexes `request` asks for that
|
||||
@@ -29,10 +32,110 @@ pub(crate) trait SiblingBuilder {
|
||||
/// finished overlap matrix. A no-op, cheap existence-check pass, once
|
||||
/// everything requested already exists.
|
||||
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()>;
|
||||
|
||||
/// Build the sibling annex (`siblings.psib`, every annex above is derived
|
||||
/// from it) if any cached layer lacks it — the state of an index right
|
||||
/// after `migrate-mphf`, which sets every annex aside. The build writes
|
||||
/// into every layer, so it runs under the index's exclusive
|
||||
/// [`DirLock`]; a process that waited for it finds the annex built by the
|
||||
/// one that held it, and does nothing. A no-op, cheap existence-check
|
||||
/// pass, once every layer has its annex. Must not be called while this
|
||||
/// process already holds the lock *and* an annex is missing (the lock is
|
||||
/// not reentrant); `--sibling-annex` holds it but builds the annex first.
|
||||
fn ensure_sibling_annex(&self) -> OKIResult<()>;
|
||||
|
||||
/// (Re)build the sibling annex of every cached layer, unconditionally,
|
||||
/// removing the annexes derived from it (entropy, quartet, pair weight,
|
||||
/// family overlap), which its renumbering makes stale. Takes no lock:
|
||||
/// the caller holds it (`--sibling-annex`) or goes through
|
||||
/// [`ensure_sibling_annex`](Self::ensure_sibling_annex).
|
||||
fn build_base_sibling_annex(&self) -> OKIResult<()>;
|
||||
}
|
||||
|
||||
impl SiblingBuilder for IndexCache {
|
||||
fn ensure_sibling_annex(&self) -> OKIResult<()> {
|
||||
let missing = || self.iter_indexed().any(|(_, _, layer)| !layer.dir().join(ANNEX_FILE_NAME).exists());
|
||||
if !missing() {
|
||||
return Ok(());
|
||||
}
|
||||
let _lock = DirLock::acquire(self.index().dir()).map_err(OKIError::Io)?;
|
||||
if !missing() {
|
||||
return Ok(()); // built by the process we waited for
|
||||
}
|
||||
tracing::info!("sibling annex missing — building it");
|
||||
self.build_base_sibling_annex()
|
||||
}
|
||||
|
||||
fn build_base_sibling_annex(&self) -> OKIResult<()> {
|
||||
let n_parts = self.index().n_partitions();
|
||||
let fast_mode = is_fast_mode(self);
|
||||
if !fast_mode {
|
||||
tracing::warn!(
|
||||
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
|
||||
disabled; consider compacting this index"
|
||||
);
|
||||
}
|
||||
|
||||
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
|
||||
let mut total_slots: u64 = 0;
|
||||
for part in self.partitions() {
|
||||
let n_layer = self.n_layer(part).unwrap_or(0);
|
||||
let mut part_slots: u64 = 0;
|
||||
for l in 0..n_layer {
|
||||
let layer = self
|
||||
.get_layer(part, l)
|
||||
.expect("cache-consistent: n_layer(part) already bounds l");
|
||||
// Invalidate this layer's entropy, quartet and pair-weight
|
||||
// annexes, if any — they were built against the sibling
|
||||
// annex's *previous* `family_idx` numbering
|
||||
// (`non_monomorphic_reindex_layer` reads the sibling
|
||||
// annex), and are stale once the new one exists.
|
||||
// `ensure_family_annexes` only builds what's *missing*, so
|
||||
// the old files must be removed here for them to be rebuilt
|
||||
// rather than silently left stale. Removed *before* the new
|
||||
// annex is built and published (atomically, at its end): a
|
||||
// build killed in between leaves the previous annex with no
|
||||
// derived files — still consistent, rebuilt on demand —
|
||||
// never a new annex next to derived files computed against
|
||||
// the old numbering.
|
||||
for name in ScalarKind::Entropy
|
||||
.file_names()
|
||||
.into_iter()
|
||||
.chain(ScalarKind::PairWeight.file_names())
|
||||
.chain(QUARTET_ANNEX_FILE_NAMES)
|
||||
{
|
||||
let derived_path = layer.dir().join(name);
|
||||
if derived_path.exists() {
|
||||
std::fs::remove_file(&derived_path).map_err(OKIError::Io)?;
|
||||
}
|
||||
}
|
||||
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
||||
}
|
||||
total_slots += part_slots;
|
||||
pb.inc(1);
|
||||
pb.set_message(format!(
|
||||
"partition {part}: {part_slots} kmers ({total_slots} total)"
|
||||
));
|
||||
}
|
||||
pb.finish_and_clear();
|
||||
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
|
||||
|
||||
// The family-overlap annexes are index-level, not per-layer (see
|
||||
// their own module docs) — any layer's sibling annex changing makes
|
||||
// them all stale (and, with them, every pair weight computed from
|
||||
// them, removed per layer above), so they are invalidated once here
|
||||
// rather than per-layer above.
|
||||
for name in FAMILY_OVERLAP_FILE_NAMES {
|
||||
let overlap_path = self.index().dir().join(name);
|
||||
if overlap_path.exists() {
|
||||
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn ensure_family_annexes(&self, request: &AnnexRequest) -> OKIResult<()> {
|
||||
self.ensure_sibling_annex()?;
|
||||
let n_genomes = self.meta().genomes().len();
|
||||
let fast_mode = is_fast_mode(self);
|
||||
let overlap_path = |no_ambiguity: bool| self.index().dir().join(family_overlap_file_name(no_ambiguity));
|
||||
|
||||
@@ -11,17 +11,12 @@ use std::path::Path;
|
||||
|
||||
use obikidxcache::index_cache::IndexCache;
|
||||
use obikindex::{OKIError, OKIResult};
|
||||
use obisys::progress_bar;
|
||||
|
||||
use crate::siblings::algorithms::{
|
||||
AnnexRequest, EntropySummary, SiblingAnnexStats, build_layer_sibling_annex, is_fast_mode,
|
||||
scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
|
||||
AnnexRequest, EntropySummary, SiblingAnnexStats, scan_family_entropies, sibling_annex_stats, sibling_family_size_histogram,
|
||||
};
|
||||
use crate::siblings::extensions::SiblingBuilder;
|
||||
use crate::siblings::{
|
||||
FAMILY_OVERLAP_FILE_NAMES, FamilyOverlap, QUARTET_ANNEX_FILE_NAMES, ScalarKind,
|
||||
family_overlap_file_name,
|
||||
};
|
||||
use crate::siblings::{FamilyOverlap, family_overlap_file_name};
|
||||
|
||||
pub trait SiblingExt {
|
||||
/// Build the sibling-count/minorant annex for every layer of every
|
||||
@@ -111,71 +106,7 @@ pub trait SiblingExt {
|
||||
|
||||
impl SiblingExt for IndexCache {
|
||||
fn build_sibling_annex(&self) -> OKIResult<()> {
|
||||
let n_parts = self.index().n_partitions();
|
||||
let fast_mode = is_fast_mode(self);
|
||||
if !fast_mode {
|
||||
tracing::warn!(
|
||||
"index has more than 7 layers per partition — sibling-annex fast layer lookup \
|
||||
disabled; consider compacting this index"
|
||||
);
|
||||
}
|
||||
|
||||
let pb = progress_bar("sibling_annex", self.n_partition() as u64, "partitions");
|
||||
let mut total_slots: u64 = 0;
|
||||
for part in self.partitions() {
|
||||
let n_layer = self.n_layer(part).unwrap_or(0);
|
||||
let mut part_slots: u64 = 0;
|
||||
for l in 0..n_layer {
|
||||
let layer = self
|
||||
.get_layer(part, l)
|
||||
.expect("cache-consistent: n_layer(part) already bounds l");
|
||||
// Invalidate this layer's entropy and quartet annexes, if
|
||||
// any — they were built against the sibling annex's
|
||||
// *previous* `family_idx` numbering
|
||||
// (`non_monomorphic_reindex_layer` reads the sibling
|
||||
// annex), and are stale once the new one exists.
|
||||
// `ensure_family_annexes` below only builds what's
|
||||
// *missing*, so the old files must be removed here for
|
||||
// them to be rebuilt rather than silently left stale.
|
||||
// Removed *before* the new annex is built and published
|
||||
// (atomically, at its end): a build killed in between
|
||||
// leaves the previous annex with no derived files — still
|
||||
// consistent, rebuilt on demand — never a new annex next to
|
||||
// derived files computed against the old numbering.
|
||||
for name in ScalarKind::Entropy
|
||||
.file_names()
|
||||
.into_iter()
|
||||
.chain(ScalarKind::PairWeight.file_names())
|
||||
.chain(QUARTET_ANNEX_FILE_NAMES)
|
||||
{
|
||||
let entropy_path = layer.dir().join(name);
|
||||
if entropy_path.exists() {
|
||||
std::fs::remove_file(&entropy_path).map_err(OKIError::Io)?;
|
||||
}
|
||||
}
|
||||
part_slots += build_layer_sibling_annex(self, &layer, l, n_parts, fast_mode)?;
|
||||
}
|
||||
total_slots += part_slots;
|
||||
pb.inc(1);
|
||||
pb.set_message(format!(
|
||||
"partition {part}: {part_slots} kmers ({total_slots} total)"
|
||||
));
|
||||
}
|
||||
pb.finish_and_clear();
|
||||
tracing::info!("sibling annex built — {total_slots} kmers across {n_parts} partitions");
|
||||
|
||||
// The family-overlap annexes are index-level, not per-layer (see
|
||||
// their own module docs) — any layer's sibling annex changing makes
|
||||
// them all stale (and, with them, every pair weight computed from
|
||||
// them, removed per layer above), so they are invalidated once here
|
||||
// rather than per-layer above.
|
||||
for name in FAMILY_OVERLAP_FILE_NAMES {
|
||||
let overlap_path = self.index().dir().join(name);
|
||||
if overlap_path.exists() {
|
||||
std::fs::remove_file(&overlap_path).map_err(OKIError::Io)?;
|
||||
}
|
||||
}
|
||||
|
||||
self.build_base_sibling_annex()?;
|
||||
// Precompute the entropy and family-overlap annexes too, now that
|
||||
// every layer's fresh sibling annex is on disk
|
||||
// (`non_monomorphic_reindex_layer`, which both depend on, reads
|
||||
|
||||
@@ -269,6 +269,21 @@ impl SiblingAnnex {
|
||||
Ok(Self { mmap, n, path: path.to_path_buf() })
|
||||
}
|
||||
|
||||
/// The annex of the layer directory `layer_dir`. Same as
|
||||
/// [`open`](Self::open), except that an absent file is reported with its
|
||||
/// path and the way to build it, rather than a bare `NotFound` — the
|
||||
/// state an index is in right after `migrate-mphf` set its annexes aside.
|
||||
pub fn open_in(layer_dir: &Path) -> io::Result<Self> {
|
||||
let path = layer_dir.join(super::ANNEX_FILE_NAME);
|
||||
Self::open(&path).map_err(|e| match e.kind() {
|
||||
io::ErrorKind::NotFound => io::Error::new(
|
||||
io::ErrorKind::NotFound,
|
||||
format!("no sibling annex at {} — build it with `phylo --sibling-annex`", path.display()),
|
||||
),
|
||||
_ => e,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn path(&self) -> &Path { &self.path }
|
||||
pub fn len(&self) -> usize { self.n }
|
||||
pub fn is_empty(&self) -> bool { self.n == 0 }
|
||||
|
||||
Reference in new issue
Block a user