OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

perf: optimize entropy computation by pre-filtering monomorphic minorants

Browse Source 
Shift monomorphism filtering from the entropy scan layer to a lightweight, annex-only pre-pass. By replacing `Selection::All` with a pre-filtered subset, expensive per-genome resolution is strictly limited to non-monomorphic families. This avoids processing ~98% of minorants that are known to be monomorphic, while preserving positional read speedups for subsequent runs. The change is an internal performance refinement with no public API modifications.
This commit is contained in:
Eric Coissac
2026-08-17 09:32:45 +02:00
parent 7bac0f3850
commit c6cfdac043
3 changed files with 62 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DevDocMD/architecture/siblings.md
+19 -3
View File
@@ -504,14 +504,30 @@ but indexed by `family_idx` (every minorant of the layer, monomorphic
included — the same numbering `Selection`/`scan_layer_families` already
use), one `f32` entropy15 value per entry, `-1.0` sentinel for monomorphic/
not-yet-computed. First use of `--entropy`/`--entropy-sd` on an index
pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`: a full,
unsampled `Selection::All` scan, resolving every non-monomorphic
minorant's `genome_mask` once to compute and persist its entropy) — every
pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`) — every
later run (any `μ`/`σ`, any command) reads the file positionally, no
re-scan, restoring the usual `Selection::Some` "skip resolving excluded
families" speedup that a naive "weigh during the resolving scan" design
would have permanently forfeited.
**Bug found and fixed (2026-08-15): `ensure_entropy_annexes` scanned with
`Selection::All` instead of bounding to non-monomorphic minorants.**
Monomorphism (`family_size() < 2`) is knowable directly from the annex
bits alone, no per-genome resolution needed — but the original
implementation called `scan_layer_families` with `Selection::All`
anyway, so `fill_sub_matrix_carries` (the expensive per-genome
resolution) ran for *every* minorant, ~98% of which are monomorphic
(measured elsewhere in this doc) and had their `genome_mask` immediately
discarded once the callback checked `family_size() < 2`. Fixed by adding
[`subsample::non_monomorphic_selection_layer`] — a cheap, annex-only,
non-sampling pass (same shape as `reservoir_sample_layer`, but keeping
every non-monomorphic minorant's `family_idx` instead of a bounded
reservoir) — and passing `Selection::Some(&eligible)` instead of
`Selection::All`, so the expensive resolution now runs only for the ~2%
of minorants that can actually produce a real entropy value. A
`debug_assert!(mask.family_size() >= 2, ...)` inside the
`scan_layer_families` callback guards the invariant.
**Resolved**: the existing hard "non-monomorphic minorant" eligibility
filter stays a hard gate upstream of the Gaussian weighting — only
qualifying families ever get a stored entropy value or a weighted draw.