docmd/architecture/siblings.md

@@ -148,6 +148,108 @@ Baseline: mostly one active core, with short multi-core bursts — average
   Measured ~10% *slower*, wider dips, not narrower. Reverted to the
   original chunk sizing.
 
+## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)
+
+`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`,
+`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already
+implements most of a dispatch/accumulate/resolve pipeline: generation
+(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and
+buckets cross-partition queries) runs on an `obipipeline::throttle` +
+`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter`
+*across partitions*, one generated batch resolved at a time, never several
+concurrently — this ordering is deliberate, see the module's own docs on a
+reverted concurrent-batch-resolution attempt that scattered mmap access).
+The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`)
+already exists: `n_layers <= 7` (checked once from the first non-empty
+partition's `PartitionMeta::n_layers`, documented as identical across every
+partition of an index — a structural, build-time property, never a
+per-partition state) decides whether `FamilyMask`'s recorded `layer_value`
+can be trusted to skip straight to the right layer
+(`find_presence_batch_fast`) or must fall back to scanning every layer of
+the destination partition (`find_presence_batch`).
+
+**Real gap, confirmed not implemented**: resolution is triggered by the
+*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the
+scanned layer), not by an *output* accumulator filling up. Since most
+central-base variants of a family route back to the same partition being
+scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s
+`outgoing[dest]` is large for the local/self partition and thin for the
+other ~255 (or however many) destination partitions — each of those gets
+resolved at low query density every batch instead of being accumulated
+across several source batches until resolving it is worthwhile. This is
+distinct from, and not fixed by, the fast/slow layer gate above.
+
+Redesign sketched (not built): per-destination accumulators decoupled from
+`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch
+completion — a "hot" accumulator for the partition being scanned (sharded
+one-per-generation-worker, no lock, since all `n_workers` pipeline workers
+write to it concurrently — this differs from an earlier, simpler mental
+model of "one thread owns one layer's local collector," which doesn't hold
+here since `n_workers` threads cooperate on scanning *one* layer at a time,
+not one thread per layer) and "cold" mutex-per-partition accumulators for
+the rest, low contention expected since traffic to any single cold
+destination is a small fraction of total.
+
+This breaks the current strict-iteration-order delivery of `on_family`
+(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH`
+resolves atomically). With cross-batch accumulation, a family only becomes
+complete once *every* accumulator holding one of its outgoing queries has
+flushed, at unpredictable, independent times — no longer streamable
+strictly in order without a large, unbounded pending buffer. Resolution
+sketched: replace order-dependent consumers with coordinate-addressed
+writes instead of order-dependent appends (see `PseudoAlignment` idea
+below) wherever possible, since `sibling_annex_stats`'s reduction (plain
+counts) is already order-independent and needs nothing here.
+
+## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14)
+
+The reference run (`phyloskims_sal_vac`-scale bacterial test set,
+`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites
+(4.8 MB) — trivially small. The in-progress plant index is expected to
+carry on the order of 9 billion minorant families; a dense byte-per-cell
+alignment at that column count is unbuildable regardless of genome count
+(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000–
+8,000 genomes on a large machine, which makes the per-cell cost dominant in
+the other dimension too. Pruning the retained family set before
+materializing anything is mandatory, not an optimization.
+
+**Already free**: `family_size() < 2` (no sibling variant registered at
+all) is a zero-cost structural filter, read directly off `FamilyMask` bits,
+already applied in `snp_pseudo_alignment`. Insufficient alone — per the
+~80-85% mono-family estimate from earlier discussion, this only brings 9
+billion down to roughly 1.3-1.8 billion, still unusable.
+
+**Criterion under discussion**: fix a global cell budget
+(`n_genomes × n_columns_retained ≤ threshold`) and retain the
+highest-entropy families first until the budget is spent — column count
+self-adjusts to genome count automatically. Entropy of a family: Shannon
+entropy over the observed base distribution across genomes carrying that
+family, **decided 2026-08-14: genomes where the family is absent are
+excluded from the calculation** (denominator = genomes where present, not
+all genomes) — measures signal purity where it exists, independent of
+coverage rate.
+
+Key consequence for storage: entropy needs per-variant genome counts, which
+cannot be derived from `FamilyMask`'s bits alone — it requires the same
+cross-partition resolution work the alignment/stats pipeline already pays
+for. This favors computing it once at annex-build time and persisting it in
+an **auxiliary vector alongside the annex** (one value per retained
+minorant) over encoding a fixed keep/discard decision as one of
+`FamilyMask`'s 3 remaining free bits (bits 13-15, `siblingannex.rs:77`): the
+bit approach bakes a single threshold in permanently (any different budget
+needs a full annex rebuild), the vector approach pays the expensive
+resolution once and lets budget/threshold be chosen freely per analysis
+afterward.
+
+**Open, explicitly deferred**: scope of the top-K selection — global across
+the whole index (needs a first pass computing/storing every family's
+entropy, then a global threshold from the full distribution, e.g. a
+quantile, before building the final alignment: two full passes, but the
+size budget is honored precisely) vs. local per layer/partition (stays
+within the current single-pass streaming model, but the global budget is
+no longer exactly guaranteed — depends on how unevenly entropy is
+distributed across layers).
+
 **Known remaining limitation, not yet worth fixing:** within one layer, the
 four stages (sequential `unitigs.bin` read → parallel generation →
 parallel resolution → sequential annex write) never overlap — confirmed by

src/obikindex/src/index.rs

@@ -142,6 +142,19 @@ impl KmerIndex {
     pub fn minimizer_size(&self) -> usize { self.meta.config.minimizer_size }
     pub fn n_partitions(&self) -> usize { self.partition.n_partitions() }
 
+    /// Number of layers per partition.
+    ///
+    /// Structural property of the index, fixed at build time and
+    /// homogeneous across all partitions — reading it off partition 0
+    /// is enough, no need to scan every partition.
+    pub fn n_layers_per_partition(&self) -> OKIResult<usize> {
+        use obilayeredmap::meta::PartitionMeta;
+        let index_dir = self.partition.part_dir(0).join("index");
+        let meta = PartitionMeta::load(&index_dir)
+            .map_err(|e| OKIError::Io(std::io::Error::new(std::io::ErrorKind::Other, e.to_string())))?;
+        Ok(meta.n_layers)
+    }
+
     /// Expose the inner partition so the caller can run scatter into it.
     /// Call `mark_scattered` once scatter is complete.
     pub fn partition_mut(&mut self) -> &mut KmerPartition {