From f5e4bbfc6bd8329f1bc1adbd6c6e610a364b1a36 Mon Sep 17 00:00:00 2001 From: Eric Coissac Date: Fri, 14 Aug 2026 22:45:08 +0200 Subject: [PATCH] Document architecture redesign and add partition layer accessor Documents a proposed redesign for cross-partition batch resolution, shifting trigger logic to per-destination accumulator thresholds and introducing entropy-based pruning criteria. Adds an `n_layers_per_partition` method to the index, exposing partition metadata with consistent error handling and clarified documentation regarding build-time structural properties. --- docmd/architecture/siblings.md | 102 +++++++++++++++++++++++++++++++++ src/obikindex/src/index.rs | 13 +++++ 2 files changed, 115 insertions(+) diff --git a/docmd/architecture/siblings.md b/docmd/architecture/siblings.md index 3df7c6aa..785f18a7 100644 --- a/docmd/architecture/siblings.md +++ b/docmd/architecture/siblings.md @@ -148,6 +148,108 @@ Baseline: mostly one active core, with short multi-core bursts — average Measured ~10% *slower*, wider dips, not narrower. Reverted to the original chunk sizing. +## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14) + +`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`, +`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already +implements most of a dispatch/accumulate/resolve pipeline: generation +(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and +buckets cross-partition queries) runs on an `obipipeline::throttle` + +`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter` +*across partitions*, one generated batch resolved at a time, never several +concurrently — this ordering is deliberate, see the module's own docs on a +reverted concurrent-batch-resolution attempt that scattered mmap access). +The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`) +already exists: `n_layers <= 7` (checked once from the first non-empty +partition's `PartitionMeta::n_layers`, documented as identical across every +partition of an index — a structural, build-time property, never a +per-partition state) decides whether `FamilyMask`'s recorded `layer_value` +can be trusted to skip straight to the right layer +(`find_presence_batch_fast`) or must fall back to scanning every layer of +the destination partition (`find_presence_batch`). + +**Real gap, confirmed not implemented**: resolution is triggered by the +*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the +scanned layer), not by an *output* accumulator filling up. Since most +central-base variants of a family route back to the same partition being +scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s +`outgoing[dest]` is large for the local/self partition and thin for the +other ~255 (or however many) destination partitions — each of those gets +resolved at low query density every batch instead of being accumulated +across several source batches until resolving it is worthwhile. This is +distinct from, and not fixed by, the fast/slow layer gate above. + +Redesign sketched (not built): per-destination accumulators decoupled from +`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch +completion — a "hot" accumulator for the partition being scanned (sharded +one-per-generation-worker, no lock, since all `n_workers` pipeline workers +write to it concurrently — this differs from an earlier, simpler mental +model of "one thread owns one layer's local collector," which doesn't hold +here since `n_workers` threads cooperate on scanning *one* layer at a time, +not one thread per layer) and "cold" mutex-per-partition accumulators for +the rest, low contention expected since traffic to any single cold +destination is a small fraction of total. + +This breaks the current strict-iteration-order delivery of `on_family` +(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH` +resolves atomically). With cross-batch accumulation, a family only becomes +complete once *every* accumulator holding one of its outgoing queries has +flushed, at unpredictable, independent times — no longer streamable +strictly in order without a large, unbounded pending buffer. Resolution +sketched: replace order-dependent consumers with coordinate-addressed +writes instead of order-dependent appends (see `PseudoAlignment` idea +below) wherever possible, since `sibling_annex_stats`'s reduction (plain +counts) is already order-independent and needs nothing here. + +## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14) + +The reference run (`phyloskims_sal_vac`-scale bacterial test set, +`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites +(4.8 MB) — trivially small. The in-progress plant index is expected to +carry on the order of 9 billion minorant families; a dense byte-per-cell +alignment at that column count is unbuildable regardless of genome count +(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000– +8,000 genomes on a large machine, which makes the per-cell cost dominant in +the other dimension too. Pruning the retained family set before +materializing anything is mandatory, not an optimization. + +**Already free**: `family_size() < 2` (no sibling variant registered at +all) is a zero-cost structural filter, read directly off `FamilyMask` bits, +already applied in `snp_pseudo_alignment`. Insufficient alone — per the +~80-85% mono-family estimate from earlier discussion, this only brings 9 +billion down to roughly 1.3-1.8 billion, still unusable. + +**Criterion under discussion**: fix a global cell budget +(`n_genomes × n_columns_retained ≤ threshold`) and retain the +highest-entropy families first until the budget is spent — column count +self-adjusts to genome count automatically. Entropy of a family: Shannon +entropy over the observed base distribution across genomes carrying that +family, **decided 2026-08-14: genomes where the family is absent are +excluded from the calculation** (denominator = genomes where present, not +all genomes) — measures signal purity where it exists, independent of +coverage rate. + +Key consequence for storage: entropy needs per-variant genome counts, which +cannot be derived from `FamilyMask`'s bits alone — it requires the same +cross-partition resolution work the alignment/stats pipeline already pays +for. This favors computing it once at annex-build time and persisting it in +an **auxiliary vector alongside the annex** (one value per retained +minorant) over encoding a fixed keep/discard decision as one of +`FamilyMask`'s 3 remaining free bits (bits 13-15, `siblingannex.rs:77`): the +bit approach bakes a single threshold in permanently (any different budget +needs a full annex rebuild), the vector approach pays the expensive +resolution once and lets budget/threshold be chosen freely per analysis +afterward. + +**Open, explicitly deferred**: scope of the top-K selection — global across +the whole index (needs a first pass computing/storing every family's +entropy, then a global threshold from the full distribution, e.g. a +quantile, before building the final alignment: two full passes, but the +size budget is honored precisely) vs. local per layer/partition (stays +within the current single-pass streaming model, but the global budget is +no longer exactly guaranteed — depends on how unevenly entropy is +distributed across layers). + **Known remaining limitation, not yet worth fixing:** within one layer, the four stages (sequential `unitigs.bin` read → parallel generation → parallel resolution → sequential annex write) never overlap — confirmed by diff --git a/src/obikindex/src/index.rs b/src/obikindex/src/index.rs index 3aedca98..6b7e117d 100644 --- a/src/obikindex/src/index.rs +++ b/src/obikindex/src/index.rs @@ -142,6 +142,19 @@ impl KmerIndex { pub fn minimizer_size(&self) -> usize { self.meta.config.minimizer_size } pub fn n_partitions(&self) -> usize { self.partition.n_partitions() } + /// Number of layers per partition. + /// + /// Structural property of the index, fixed at build time and + /// homogeneous across all partitions — reading it off partition 0 + /// is enough, no need to scan every partition. + pub fn n_layers_per_partition(&self) -> OKIResult { + use obilayeredmap::meta::PartitionMeta; + let index_dir = self.partition.part_dir(0).join("index"); + let meta = PartitionMeta::load(&index_dir) + .map_err(|e| OKIError::Io(std::io::Error::new(std::io::ErrorKind::Other, e.to_string())))?; + Ok(meta.n_layers) + } + /// Expose the inner partition so the caller can run scatter into it. /// Call `mark_scattered` once scatter is complete. pub fn partition_mut(&mut self) -> &mut KmerPartition {