large refactoring
This commit is contained in:
Eric Coissac
@@ -0,0 +1,742 @@
|
||||
# Sibling annex — architecture (discussion)
|
||||
|
||||
Status: architecture decided (2026-08-14). Implementation not yet mandated.
|
||||
|
||||
## Two index spaces, uncorrelated
|
||||
|
||||
Every kmer stored in a `Layer` lives in two independent index spaces:
|
||||
|
||||
- **Iteration order**: its position when enumerating `unitigs.bin` (the
|
||||
superkmer file), deterministic but arbitrary with respect to slot.
|
||||
- **MPHF slot**: `MphfLayer::index(kmer)`, the number the MPHF assigns.
|
||||
|
||||
The two are not correlated by any formula. Converting from one to the other
|
||||
requires either recomputing the MPHF (kmer → slot) or scanning the iteration
|
||||
stream (kmer → order). There is no `slot → kmer` operation: the MPHF is a
|
||||
one-way function, not an invertible bijection with a stored inverse. Any
|
||||
method that reconstructs a kmer from a bare slot number is wrong by
|
||||
construction, regardless of the mechanism used (MPHF re-hash, or evidence
|
||||
decode + direct unitig read). See `MphfLayer::kmer_at`
|
||||
(`obilayeredmap/src/mphf_layer.rs`) — flagged for removal, currently called
|
||||
from `obikphylo/siblings/build.rs` and `family_scan.rs` (since removed — see
|
||||
"Pending work" status below).
|
||||
|
||||
## Two pipelines, never mixed
|
||||
|
||||
| | origin of the kmer | membership known? | correct mapping |
|
||||
|---|---|---|---|
|
||||
| **query pipeline** | external (caller-supplied) | no | `query`/`find`/`find_strict` — MPHF + evidence check |
|
||||
| **iteration pipeline** | enumerated from this layer's own `unitigs.bin` | yes, by construction | `index`/`index_batch` — MPHF only, no evidence |
|
||||
|
||||
Evidence exists solely to answer "is this external kmer a member of the
|
||||
layer" for the query pipeline. Using it (or the MPHF) to go the other way —
|
||||
recover a kmer from a slot, or re-verify a kmer that was just produced by
|
||||
iterating the layer — is a conceptual error: evidence can be probabilistic
|
||||
(`Approx` mode), so any slot→kmer attempt is unsound in general, and
|
||||
pointless even in `Exact`/`Hybrid` mode since the kmer was already known.
|
||||
|
||||
## Sibling annex: an iteration-pipeline artifact only
|
||||
|
||||
The sibling annex (`FamilyMask`/`SiblingAnnex`, `.psib`,
|
||||
`obicompactvec/src/siblingannex.rs`) records, per kmer, whether it is a
|
||||
family minorant and which family members are present in the index. Its only
|
||||
consumers (`obikphylo/siblings/stats.rs`, `family_scan.rs`) enumerate it
|
||||
exhaustively (`0..annex.len()`); no query-pipeline code path touches it.
|
||||
|
||||
**Decision**: the annex must be persisted in iteration order, not slot
|
||||
order. This lets readers zip-iterate `Layer::iter_kmers()` and the annex
|
||||
file directly — one linear, cache-friendly pass, no MPHF/slot indirection,
|
||||
no `kmer_at`. It also enables specialized iterators building on this zip:
|
||||
minorants-only iteration, batch-of-kmers → batch-of-family-members, etc.
|
||||
|
||||
Today the annex is built and stored in **slot** order
|
||||
(`build_layer_sibling_annex`, `siblings/build.rs`): `slot_kmer` is populated
|
||||
via `(0..n_slots).map(|slot| mphf.kmer_at(slot))`, and the origin `slot` is
|
||||
threaded through the whole cross-partition reconciliation pipeline (variant
|
||||
generation, `query_partition_with`, final `mask[slot].fetch_or(...)`). This
|
||||
must change to iterating `iter_kmers()`/`enumerate_kmers()` and threading
|
||||
the **iteration index** instead of the slot end to end — eliminating
|
||||
`kmer_at` from the build path entirely, not just the read path. No
|
||||
slot-indexed intermediate is needed even during construction; the
|
||||
iteration-order id is sufficient throughout.
|
||||
|
||||
The cross-partition side of the same pipeline is unaffected: checking
|
||||
whether a generated family-variant kmer exists in another partition is a
|
||||
genuine query-pipeline operation (the variant's membership in the *target*
|
||||
partition is unknown) and must keep going through
|
||||
`KmerPartition::query_partition_with` (MPHF + evidence), never a raw
|
||||
`index()`.
|
||||
|
||||
## Pending work — done
|
||||
|
||||
The plan above shipped: `obikphylo` (a new crate — phylo-domain extension
|
||||
traits over `obikindex::KmerIndex`/`obilayeredmap::Layer<D>`, replacing the
|
||||
old `obikindex::siblings` module) builds and reads the annex purely in
|
||||
iteration order (`SiblingLayerExt::iter_siblings`/`iter_minorants`, both with
|
||||
batch variants, mirroring `Layer<D>`'s own `KmerIter`/`KmerBatchIter`
|
||||
shape). `MphfLayer::kmer_at` has no remaining callers.
|
||||
|
||||
A separate, unrelated bug surfaced during this work and was fixed
|
||||
(2026-08-14): `MphfLayer::enumerate_kmers_batch` computed its
|
||||
`batch_start_index` via the stdlib `.enumerate()` adapter, which counts
|
||||
*batches* (0, 1, 2…), not the cumulative k-mer offset the annex is actually
|
||||
keyed on — every batch past the first wrote its mask/annex entries at the
|
||||
wrong iteration-order position. Fixed by tracking a running offset instead;
|
||||
regression tests added (`sibling_annex_no_empty_masks_after_build`,
|
||||
`sibling_histogram_does_not_panic_on_partial_last_batch`).
|
||||
|
||||
## Performance: `build_sibling_annex` parallelism (2026-08-14)
|
||||
|
||||
Investigated on a real multi-genome run (`phyloskims_sal_vac`, k=31/m=11).
|
||||
Baseline: mostly one active core, with short multi-core bursts — average
|
||||
~3 cores.
|
||||
|
||||
**Fixes that helped, kept:**
|
||||
|
||||
- `CanonicalKmerOf::minimizer()` (`obikseq/src/kmer.rs`) — a direct O(k)
|
||||
bit-arithmetic minimiser for a single isolated k-mer, replacing a
|
||||
`RollingStat` instance fed byte-by-byte through an ASCII round-trip (used
|
||||
by `helpers::partition_of`, called for every generated family variant).
|
||||
~3x wall-clock improvement on its own, confirmed by sampling
|
||||
(`obiskbuilder::rolling_stat`/`obikentropy` frames disappeared from the
|
||||
hot path). `CanonicalKmerOf::partition()` added alongside it (wraps
|
||||
`minimizer().seq_hash() & mask`, the same routing rule
|
||||
`KmerPartition`/`RoutableSuperKmer` use).
|
||||
- Cross-partition resolution (`outgoing.par_iter()` in
|
||||
`build_layer_sibling_annex`) parallelised at the *partition* level — one
|
||||
Rayon task per non-empty `outgoing[dest]` bucket. For k=31/m=11, a
|
||||
central-base substitution changes the winning minimiser (and thus the
|
||||
destination partition) only when that window overlaps the central base:
|
||||
~11 of the 21 possible windows do, so ~10/21 (≈48%) of generated variants
|
||||
route right back to the partition already being built. That self bucket
|
||||
ends up far larger than any other, so the per-partition split pinned one
|
||||
thread to it alone while the rest of the pool finished instantly —
|
||||
confirmed by sampling: one thread solid in `MphfLayer::find`, everyone
|
||||
else idle. Fixed by splitting each non-empty bucket into
|
||||
`total_queries / n_workers` (capped 4096) chunks *before* `par_iter()`,
|
||||
preserving per-partition mmap locality (each chunk stays contiguous
|
||||
within one partition) while letting Rayon spread an oversized bucket
|
||||
across several threads. Net effect of both fixes together: ~3 cores
|
||||
average → ~10-13 cores average on the same run, and a projected total
|
||||
build time of ~1h15 down to ~30min on the real `phyloskims_sal_vac` run
|
||||
this was measured against.
|
||||
- `TracedBar`'s ETA (`obisys/src/progress.rs`) was silently starved: the
|
||||
custom progress message and the self-computed ETA text used to share one
|
||||
`pb.set_message()` slot, with the ETA holding off for 2s after any custom
|
||||
message — fine when custom messages are rare, broken once
|
||||
`build_sibling_annex`'s per-partition callback fires more often than
|
||||
that. Fixed by keeping the two texts in separate fields, composed
|
||||
together on every render instead of one overwriting the other.
|
||||
|
||||
**Tried and reverted — do not repeat blindly:**
|
||||
|
||||
- Parallelising the *outer* partition loop in `build_sibling_annex` with
|
||||
`obikindex::PartitionRunner` (already used by `merge`/`build_layers`),
|
||||
splitting a fixed core budget between outer (partition) and inner
|
||||
(pipeline + resolution) concurrency so their product wouldn't exceed the
|
||||
budget. Measured *worse*: throughput dropped over time (26
|
||||
partitions/5min → 38/11-12min) and peak resolution concurrency fell from
|
||||
~11-12 cores to ~7-8. Cause: this capped the resolution burst — which
|
||||
scales very well on its own — to make room for outer concurrency, and
|
||||
running several partitions' resolution at once scatters access across
|
||||
multiple partitions' mmap regions at once, working against the
|
||||
locality `outgoing`'s per-partition grouping exists for. `PartitionRunner`
|
||||
stayed exported from `obikindex` (`new_capped` too) since it's
|
||||
general-purpose, but nothing in `obikphylo` calls it.
|
||||
- Splitting resolution chunks even finer (`/(n_workers*8)`, cap 1024,
|
||||
instead of `/n_workers`, cap 4096) to smooth the residual sawtooth.
|
||||
Measured ~10% *slower*, wider dips, not narrower. Reverted to the
|
||||
original chunk sizing.
|
||||
|
||||
**Known remaining limitation, not yet worth fixing:** within one layer, the
|
||||
four stages (sequential `unitigs.bin` read → parallel generation →
|
||||
parallel resolution → sequential annex write) never overlap — confirmed by
|
||||
1s-interval sampling: generation alone occupies ~17 threads evenly, but the
|
||||
next layer's read/generation never starts until the current layer's
|
||||
resolution and write are both done. This produces a real, periodic (~layer
|
||||
duration) alternation between "many cores" and "few cores" that neither of
|
||||
the fixes above touches, since both operate *within* one layer's resolution
|
||||
step. The only remaining lever is overlapping consecutive layers (e.g. a
|
||||
depth-2 pipeline: start layer N+1's read/generation while layer N's
|
||||
resolution/write is still running) — a real restructuring, not a parameter
|
||||
tweak, and explicitly *not* to be combined with the reverted
|
||||
budget-capping idea above (let each phase use however many cores it
|
||||
naturally wants; only the *scheduling* needs to overlap). Deferred, not
|
||||
started.
|
||||
|
||||
## Cross-partition batch resolution — current state vs. the batched-accumulator design (discussion, 2026-08-14)
|
||||
|
||||
`family_scan.rs::scan_layer_families` (shared by `snp_pseudo_alignment`,
|
||||
`sibling_annex_stats`, `cardinality_tally`, `scan_family_pairs`) already
|
||||
implements most of a dispatch/accumulate/resolve pipeline: generation
|
||||
(cheap, CPU-only — builds `outgoing[dest_partition]` from `FamilyMask` and
|
||||
buckets cross-partition queries) runs on an `obipipeline::throttle` +
|
||||
`make_pipe!` stage, decoupled from resolution (I/O-bound, `rayon::par_iter`
|
||||
*across partitions*, one generated batch resolved at a time, never several
|
||||
concurrently — this ordering is deliberate, see the module's own docs on a
|
||||
reverted concurrent-batch-resolution attempt that scattered mmap access).
|
||||
The fast/slow mode gate (`PartitionCache::fast_mode`, `cache.rs:162-163`)
|
||||
already exists: `n_layers <= 7` (checked once from the first non-empty
|
||||
partition's `PartitionMeta::n_layers`, documented as identical across every
|
||||
partition of an index — a structural, build-time property, never a
|
||||
per-partition state) decides whether `FamilyMask`'s recorded `layer_value`
|
||||
can be trusted to skip straight to the right layer
|
||||
(`find_presence_batch_fast`) or must fall back to scanning every layer of
|
||||
the destination partition (`find_presence_batch`).
|
||||
|
||||
**Real gap, confirmed not implemented**: resolution is triggered by the
|
||||
*source* batch finishing (`FAMILY_BATCH = 65536` minorants read from the
|
||||
scanned layer), not by an *output* accumulator filling up. Since most
|
||||
central-base variants of a family route back to the same partition being
|
||||
scanned (~48% per the k=31/m=11 measurement above), a `FAMILY_BATCH`'s
|
||||
`outgoing[dest]` is large for the local/self partition and thin for the
|
||||
other ~255 (or however many) destination partitions — each of those gets
|
||||
resolved at low query density every batch instead of being accumulated
|
||||
across several source batches until resolving it is worthwhile. This is
|
||||
distinct from, and not fixed by, the fast/slow layer gate above.
|
||||
|
||||
Redesign sketched (not built): per-destination accumulators decoupled from
|
||||
`FAMILY_BATCH`, flushed on reaching a size threshold instead of on source-batch
|
||||
completion — a "hot" accumulator for the partition being scanned (sharded
|
||||
one-per-generation-worker, no lock, since all `n_workers` pipeline workers
|
||||
write to it concurrently — this differs from an earlier, simpler mental
|
||||
model of "one thread owns one layer's local collector," which doesn't hold
|
||||
here since `n_workers` threads cooperate on scanning *one* layer at a time,
|
||||
not one thread per layer) and "cold" mutex-per-partition accumulators for
|
||||
the rest, low contention expected since traffic to any single cold
|
||||
destination is a small fraction of total.
|
||||
|
||||
This breaks the current strict-iteration-order delivery of `on_family`
|
||||
(today: a reorder buffer keyed by batch, since a whole `FAMILY_BATCH`
|
||||
resolves atomically). With cross-batch accumulation, a family only becomes
|
||||
complete once *every* accumulator holding one of its outgoing queries has
|
||||
flushed, at unpredictable, independent times — no longer streamable
|
||||
strictly in order without a large, unbounded pending buffer. Resolution
|
||||
sketched: replace order-dependent consumers with coordinate-addressed
|
||||
writes instead of order-dependent appends (see `PseudoAlignment` idea
|
||||
below) wherever possible, since `sibling_annex_stats`'s reduction (plain
|
||||
counts) is already order-independent and needs nothing here.
|
||||
|
||||
**Superseded 2026-08-15** by the `--subsample`/`--shannon` design below,
|
||||
which sidesteps the accumulator redesign for now: bounding the number of
|
||||
families actually resolved per layer (via sampling) keeps per-layer
|
||||
resolution volume small enough that the batch-density problem above stops
|
||||
mattering in practice for these two consumers. The accumulator redesign
|
||||
remains relevant for a future *unsampled, full-index* run, but is not
|
||||
required to ship `--subsample`/`--shannon`.
|
||||
|
||||
## Pseudo-alignment at scale — pruning is unavoidable (discussion, 2026-08-14/15)
|
||||
|
||||
The reference run (`phyloskims_sal_vac`-scale bacterial test set,
|
||||
`iqtree.fasta`) produced a dense alignment for 13 genomes × 383,965 sites
|
||||
(4.8 MB) — trivially small. The in-progress plant index is expected to
|
||||
carry on the order of 9 billion minorant families; a dense byte-per-cell
|
||||
alignment at that column count is unbuildable regardless of genome count
|
||||
(hundreds of GB even at a handful of genomes). Long-term ambition is 6,000–
|
||||
8,000 genomes on a large machine, which makes the per-cell cost dominant in
|
||||
the other dimension too. Pruning the retained family set before
|
||||
materializing anything is mandatory, not an optimization.
|
||||
|
||||
**Already free**: `family_size() < 2` (no sibling variant registered at
|
||||
all) is a zero-cost structural filter, read directly off `FamilyMask` bits,
|
||||
already applied in `snp_pseudo_alignment`. Insufficient alone — per the
|
||||
~80-85% mono-family estimate from earlier discussion, this only brings 9
|
||||
billion down to roughly 1.3-1.8 billion, still unusable.
|
||||
|
||||
**Entropy definition — settled 2026-08-15, correcting an earlier wrong
|
||||
turn.** The project does **not** encode families as IUPAC ambiguity codes
|
||||
interpreted the classical way (Fitch-parsimony subset-compatibility, or
|
||||
ML's "one true state, uncertain which"); see
|
||||
`docmd/theory/evolutionary_distances.md` ("Why the IUPAC/DNA encoding used
|
||||
for the first `--snp` test was wrong") and the Sankoff resolution that
|
||||
followed it. The real model is a genuine 16-state alphabet (the powerset of
|
||||
`{A,C,G,T}`, `∅` included as a real state) scored with a *calibrated
|
||||
pairwise cost matrix* (`obikphylo::cardcomp::pairwise_cost_matrix`,
|
||||
`cmd/phylo/sankoff.rs`), not a compatibility/subset relation between
|
||||
states. Under that model, each of the 16 states — including multi-bit ones
|
||||
like `AC` — is a first-class, independently-costed state, not an
|
||||
uncertainty encoding of a single true base. So: **entropy over the 15
|
||||
non-empty states (`∅` excluded, matching the earlier decision to exclude
|
||||
genomes where the family is absent) is the correct informativeness
|
||||
measure** for this project — not a 4-symbol reduction, which would discard
|
||||
exactly the cardinality/composition information the calibrated cost matrix
|
||||
is built to exploit.
|
||||
|
||||
## `--subsample` / `--shannon` — sampling strategy (decided 2026-08-15)
|
||||
|
||||
Goal: make both the pseudo-alignment (`--snp`) and a Shannon-entropy
|
||||
diagnostic usable at any index scale, from the 13-genome bacterial
|
||||
reference run up to the 9-billion-family plant index, without requiring the
|
||||
batched-accumulator redesign above.
|
||||
|
||||
**`--subsample N`** (integer, families to retain): bounds the pseudo-alignment
|
||||
to `N` minorant families, sampled **proportionally per layer** among
|
||||
non-monomorphic minorants (`family_size >= 2`) — this sidesteps the need
|
||||
for a true global reservoir merge across layers while still approximating a
|
||||
uniform sample over the whole index, and directly answers the earlier open
|
||||
question of global-vs-per-layer selection scope.
|
||||
|
||||
Three passes, in order:
|
||||
|
||||
1. **Global count** (cheap, structural, parallel across layers — same shape
|
||||
as the existing `sibling_family_size_histogram`, extended to report a
|
||||
**per-layer** breakdown rather than one index-wide aggregate): for each
|
||||
layer, `count_layer` = number of non-monomorphic minorants. Gives
|
||||
`total_count = Σ count_layer`.
|
||||
2. **Per-layer proportional reservoir sampling** (cheap, structural, one
|
||||
pass per layer, no cross-partition resolution): `N_layer = round(N ×
|
||||
count_layer / total_count)`. Since `N_layer` is a proportion of
|
||||
`count_layer`, it can never exceed it as long as `N <= total_count` — the
|
||||
one edge case is `total_count <= N`, in which case sampling is skipped
|
||||
entirely and *every* non-monomorphic minorant of every layer is kept
|
||||
(no reservoir needed, `N` was never a real constraint). Otherwise:
|
||||
Algorithm-R reservoir sampling over the layer's non-monomorphic minorant
|
||||
indices, producing `N_layer` iteration-order indices directly, no
|
||||
intermediate full list ever materialized.
|
||||
3. **Filtered resolution** (the expensive step, the existing
|
||||
`scan_layer_families` engine, unchanged): re-scan the layer, generating
|
||||
and resolving cross-partition queries **only** for the indices selected
|
||||
in step 2 (cheap membership test against a small per-layer index set) —
|
||||
this is what keeps `--subsample` cheap even on an unsampled-scale index,
|
||||
since the cross-partition resolution volume is bounded by `N`, not by
|
||||
the layer's true size.
|
||||
|
||||
Steps 2 and 3 cannot be merged into one pass: true single-pass reservoir
|
||||
sampling would waste step-3's expensive resolution work on candidates later
|
||||
evicted by the reservoir. Step 1 must fully complete (every layer) before
|
||||
step 2 can start for any layer, since `total_count` is a global quantity.
|
||||
|
||||
**`--shannon`** (no argument): emits a CSV of per-family Shannon entropy
|
||||
(15 non-empty states, `∅`/absent genomes excluded from the denominator, per
|
||||
the settled definition above). Independent of `--subsample` — entropy is
|
||||
computed and written per family as soon as its `genome_mask` resolves,
|
||||
O(1) memory per family, so it streams fine even unsampled at full index
|
||||
scale (a time cost, not a memory one). Combined with `--subsample N`, it
|
||||
delivers the original exploratory diagnostic (e.g. `--subsample 1000000
|
||||
--shannon`) directly from this general machinery, rather than a
|
||||
purpose-built one-off script.
|
||||
|
||||
Validated end-to-end (2026-08-15) against real data: `--sibling-hist` on
|
||||
`phyloskims_sal_vac` (91 real genomes, k=31/m=11, 256 partitions × 2
|
||||
layers) confirms the ~9-billion-family estimate almost exactly (8,925,068,238
|
||||
total, 97.9% monomorphic — a sharper mono fraction than the ~80-85% earlier
|
||||
guess, corrected here). `--subsample`/`--shannon` on the smaller 20-genome
|
||||
bacterial reference (`benchmark/global_index_presence`) produced a sample
|
||||
size within rounding of the request (99,742/100,000) and a
|
||||
[0.8,1.2)-bucket share (40.7%) matching the full unsampled population
|
||||
(41.6%) — the two histograms only diverged wildly (5‰ vs 41.6%) under a
|
||||
real bug in `reservoir_sample_layer` (see next section), now fixed.
|
||||
|
||||
**Bug found and fixed (2026-08-15): `family_idx` numbering mismatch.**
|
||||
`scan_layer_families`'s `family_idx` counts *every minorant* of a layer
|
||||
(monomorphic ones included, since `iter_minorants_batch` filters only on
|
||||
`is_minorant()`), not the raw annex slot (`SiblingAnnex::get(slot)` spans
|
||||
every k-mer, minorant or not) and not a counter over non-monomorphic
|
||||
minorants alone. `subsample.rs`'s `reservoir_sample_layer` originally
|
||||
stored raw slot numbers in its `HashSet<usize>` selection, which drifts
|
||||
away from `family_idx` as soon as *any* monomorphic minorant is seen —
|
||||
i.e. almost immediately, since ~98% of minorants are monomorphic. Fixed by
|
||||
tracking two separate counters: `family_idx` (every minorant, matching
|
||||
`scan_layer_families`) and `seen` (non-monomorphic minorants only, what
|
||||
Algorithm R actually samples over) — only `family_idx` values are ever
|
||||
stored in the selection set. The existing unit test never exercised this
|
||||
(its fixture has exactly one non-monomorphic family, always hitting the
|
||||
"keep everything" shortcut) — a stronger fixture with several interleaved
|
||||
monomorphic/non-monomorphic families would be needed to catch a regression
|
||||
here automatically; not yet written.
|
||||
|
||||
## Cheap entropy pre-filtering — row-marginal sums (idea, not implemented, 2026-08-15)
|
||||
|
||||
Motivation: on real data (bacterial reference, full unsampled run), only
|
||||
~5‰ of non-monomorphic minorants fall in the `[0.5, 1.5]` bit band judged
|
||||
phylogenetically interesting (entropy too low = uninformative near-invariant
|
||||
site; too high = saturated/noisy, see `family_entropy`'s 15-state
|
||||
discussion) — roughly 1 in 10,000 minorants overall. Computing exact
|
||||
entropy for every candidate just to discard 99.99% of them is wasteful at
|
||||
the full 9-billion scale.
|
||||
|
||||
**The idea**: `PersistentBitMatrix::col_view(c)` gives a genome's whole
|
||||
presence column as a `BitSliceView` (sequential, no MPHF, no cross-partition
|
||||
routing — purely local to one layer's own matrix). Accumulating
|
||||
`TempCompactIntVecBuilder::inc_present(col)` (already exists in
|
||||
`obicompactvec/src/builder.rs:121`, along with `add`/`min`/`max`/`diff` on
|
||||
`IntSliceView` — no new low-level API needed) over every column of a layer
|
||||
produces `coverage[slot]`: how many genomes carry each exact k-mer, in one
|
||||
sequential per-layer pass, entirely decoupled from family/sibling
|
||||
structure. Persisted once per layer, a family's members' coverage could
|
||||
then be looked up via a plain local MPHF `index()` (cheap) instead of a
|
||||
full cross-partition presence resolution (`find_presence_batch`) — the
|
||||
expensive part today is specifically the cross-partition/cross-layer
|
||||
routing to a sibling's own matrix, not the bit-reading itself, and
|
||||
`coverage[slot]` sidesteps that routing entirely by moving the cost into a
|
||||
one-time, purely local, embarrassingly-parallel build step.
|
||||
|
||||
**Why not implemented**: `coverage[slot]` is a per-member marginal —
|
||||
summing members' coverages to approximate a family's entropy silently
|
||||
assumes no genome carries more than one member at once. It cannot
|
||||
represent or detect joint co-occurrence (a genome carrying both `A` and
|
||||
`C` at once, i.e. a combined 15-symbol state) at all, which is exactly the
|
||||
phenomenon `family_entropy`'s 15-state definition exists to capture (see
|
||||
`CardinalityTally`/`cardinality_transition_probs`, the project's own
|
||||
existing machinery for this same co-occurrence structure, built for the
|
||||
Sankoff matrix calibration). A family that is in truth uniformly `AC`
|
||||
across every carrying genome would look like a well-balanced 2-state split
|
||||
under the marginal approximation (entropy ≈ 1) while its true 15-state
|
||||
entropy is 0 — i.e. the marginal proxy's failure mode lands on exactly the
|
||||
"saturated, uninformative" tail this pre-filter would need to catch,
|
||||
undermining the point. It stays plausible as a coarse filter for the
|
||||
*low* tail only (a dominant single member's marginal share reliably
|
||||
predicts low true entropy too), but not as a stand-in for the high tail —
|
||||
not pursued further for now.
|
||||
|
||||
## `--free-loss`/`--tnt` pipeline: four independent scans, three of them unsampled (found 2026-08-15, fixed 2026-08-15 — see "Implemented" below)
|
||||
|
||||
Measured on `phyloskims_sal_vac` (91 genomes): `obikmer phylo --subsample 500000
|
||||
--free-loss --tnt` logs four sequential stages —
|
||||
`raw_snp_distance` (1413s), `base_pair_tally` (1456s),
|
||||
`cardinality_tally` (2078s), `snp_pseudo_alignment` (143s). Reading the
|
||||
code (`obikmer/src/cmd/phylo/mod.rs:210-242`,
|
||||
`obikphylo/src/siblings/distance.rs`, `cardinality.rs`, `alignment.rs`)
|
||||
surfaced two compounding problems, not one:
|
||||
|
||||
1. **Four separate full scans of the annex**, each opening its own
|
||||
`KmerPartition`/`PartitionCache` and calling `scan_family_pairs`/
|
||||
`scan_layer_families` independently — nothing computed in one stage is
|
||||
reused by another. `base_pair_tally` is explicitly documented
|
||||
(`distance.rs:138-143`) as a second full pass over the same
|
||||
traversal `raw_snp_distance` already did, needed only because
|
||||
`raw_snp_distance` doesn't keep the resolved bases, only aggregate
|
||||
counts. `cardinality_tally` and `snp_pseudo_alignment` are each a
|
||||
third and fourth independent full pass. Per the module's own earlier
|
||||
profiling note (`family_scan.rs:26-28`, cited already above), this
|
||||
traversal is page-fault/mmap-bound, not compute-bound — the ~10-12%
|
||||
CPU efficiency ("contention" status) observed on these three slow
|
||||
stages is consistent with I/O stalls scaled by repeated full scans,
|
||||
not lock contention (there are no `Mutex`/`RwLock` anywhere in
|
||||
`siblings/*.rs`; shared writes use per-slot `AtomicU8::fetch_or`).
|
||||
2. **Worse: `raw_snp_distance` and `cardinality_tally` don't honor
|
||||
`--subsample` at all** — both call `scan_layer_families` with
|
||||
`Selection::All` hardcoded, and `args.subsample` isn't even threaded
|
||||
into their function signatures (`mod.rs:212`, `mod.rs:226`). Only
|
||||
`snp_pseudo_alignment(args.subsample)` builds a real reservoir-sampled
|
||||
`Selection::Some(set)` (via `compute_selections`, `alignment.rs:89`).
|
||||
So today, `--subsample 500000` only bounds the pseudo-alignment step —
|
||||
the SNP-distance matrix, the Sankoff base-pair calibration, and the
|
||||
cardinality histogram are always computed over the **full, unsampled**
|
||||
index regardless of the flag. This is not merely "different subsamples
|
||||
per stage" (which would already be a problem worth fixing) — it's that
|
||||
three of the four stages never subsample, which explains most of the
|
||||
~10x runtime gap against `snp_pseudo_alignment` on its own.
|
||||
|
||||
**Decided requirement**: all four stages consume **one shared selection**,
|
||||
computed once, not each stage either scanning everything or drawing its
|
||||
own independent sample. Per-genome-pair SNP counts, the Sankoff base-pair
|
||||
calibration, the cardinality histogram, and the pseudo-alignment all
|
||||
describe the same set of families — the calibration and the alignment it
|
||||
calibrates are now guaranteed to agree on which sites exist.
|
||||
|
||||
**Correction to the "single pass" framing above**: `base_pair_tally`/
|
||||
`cardinality_tally` both need `raw_snp_distance`'s *complete* aggregate
|
||||
SNP/shared counts before they can derive `included[i,j]` (the
|
||||
`ratio_ceiling` filter) — a genuine sequential dependency (`included`
|
||||
can't be known until every pair's aggregate count is final), not an
|
||||
artifact of the old code's structure. So the fix is **two** passes over
|
||||
the shared selection, not one: pass A computes the aggregate counts (and
|
||||
derives `included`); pass B fuses `base_pair_tally` + `cardinality_tally`
|
||||
+ the pseudo-alignment (mutually independent once `included` is known)
|
||||
into a single scan. Still a 4→2 reduction, and — per the clarification
|
||||
that settled this — pass A itself now runs over the *same shared
|
||||
selection* pass B uses (not the full unsampled index): "les ratios, on
|
||||
les fait sur les sites sélectionnés, c'est tout, les autres sites
|
||||
n'existent pas" — once a selection is chosen, both passes are bounded by
|
||||
it, so on a real `--subsample`/`--entropy` run pass A is cheap too, not
|
||||
just pass B.
|
||||
|
||||
**Entropy-biased selection's own resolution to the "forward-looking
|
||||
complication"** (entropy must be known before selection, but selection
|
||||
happens during the same scan that would resolve it): see "Entropy-biased
|
||||
selection" below — resolved via a persisted per-layer entropy annex, not
|
||||
by restructuring the scan into an inline pre-pass.
|
||||
|
||||
## Entropy-biased selection: soft Gaussian weighting, not a hard cutoff (decided and implemented 2026-08-15)
|
||||
|
||||
Refines the "forward-looking complication" above with a concrete
|
||||
mechanism. Instead of a hard `[low, high]` entropy band (or any other
|
||||
exact cutoff) deciding which non-monomorphic minorant families are
|
||||
eligible, selection is weighted by an **unnormalized Gaussian kernel**
|
||||
centered on a target entropy: `w(entropy) = exp(-(entropy - μ)² / (2σ²))`
|
||||
— deliberately not the normalized Gaussian density (which would peak
|
||||
below 1 and complicate the "probability" reading) — this kernel form
|
||||
equals 1 exactly at `entropy = μ` and decays smoothly to 0 away from it,
|
||||
so it reads directly as an acceptance weight: the further a family's
|
||||
entropy from the target, the less likely it is picked, with no hard
|
||||
in/out boundary — a few "bad" sites can still get in, by design. `μ`
|
||||
(default ~1.0) and `σ` (default ~0.5) are meant to be user-tunable.
|
||||
|
||||
**Mechanism: joint probability, not a weighted reservoir**. Not
|
||||
Efraimidis–Spirakis weighted reservoir sampling (an earlier, more complex
|
||||
proposal, superseded before implementation) — instead, a single
|
||||
independent accept/reject draw per qualifying candidate: draw
|
||||
`u ~ Uniform(0,1)`, accept iff `u < p₀ · w(entropy)`. `p₀` is a single
|
||||
**index-wide** rate, `N / total_count` (`total_count` = the sum of
|
||||
`non_monomorphic_counts` across every layer, `N` = `--subsample`'s
|
||||
target), applied identically at every layer — this alone already gives
|
||||
each layer its proportional share (the same effect the old uniform
|
||||
reservoir's explicit per-layer `n_layer = N · count_layer / total_count`
|
||||
computation achieved, but without needing to compute it: applying one
|
||||
rate uniformly is mathematically the same as proportioning per layer).
|
||||
`p₀ = 1.0` when there is no `--subsample` at all — `--entropy` alone is a
|
||||
pure soft entropy filter over the whole index, no size target. Properties:
|
||||
(1) **strictly generalizes the existing uniform sampler** — with `σ` large
|
||||
enough that `w ≈ 1` everywhere, this reduces to the old uniform `N/total_count`
|
||||
draw; (2) yields "approximately N", not exactly N — expected accepted
|
||||
count is `N · mean(w)`, always `≤ N` — an intentional relaxation, matching
|
||||
"sous-échantillonnage à environ n" rather than the old reservoir's exact-N
|
||||
guarantee; (3) one streaming pass, one random draw per candidate, no
|
||||
reservoir state.
|
||||
|
||||
**Resolving "entropy must be known before selection, but selection
|
||||
happens during the same resolving scan"**: solved with a **persisted,
|
||||
per-layer entropy annex** (`obikphylo/src/siblings/entropy_annex.rs`,
|
||||
`EntropyAnnex`/`EntropyAnnexBuilder`), not by restructuring the scan.
|
||||
Mirrors `SiblingAnnex`'s mmap-backed, read-only-after-build convention,
|
||||
but indexed by `family_idx` (every minorant of the layer, monomorphic
|
||||
included — the same numbering `Selection`/`scan_layer_families` already
|
||||
use), one `f32` entropy15 value per entry, `-1.0` sentinel for monomorphic/
|
||||
not-yet-computed. First use of `--entropy`/`--entropy-sd` on an index
|
||||
pays a one-time cost (`ensure_entropy_annexes` in `entropy.rs`: a full,
|
||||
unsampled `Selection::All` scan, resolving every non-monomorphic
|
||||
minorant's `genome_mask` once to compute and persist its entropy) — every
|
||||
later run (any `μ`/`σ`, any command) reads the file positionally, no
|
||||
re-scan, restoring the usual `Selection::Some` "skip resolving excluded
|
||||
families" speedup that a naive "weigh during the resolving scan" design
|
||||
would have permanently forfeited.
|
||||
|
||||
**Resolved**: the existing hard "non-monomorphic minorant" eligibility
|
||||
filter stays a hard gate upstream of the Gaussian weighting — only
|
||||
qualifying families ever get a stored entropy value or a weighted draw.
|
||||
|
||||
**CLI, implemented**: two `phylo` options, `--entropy <μ>` and
|
||||
`--entropy-sd <σ>` (`obikmer/src/cmd/phylo/args.rs`). The entropic filter
|
||||
activates as soon as *either* is given (`mod.rs`, computed once into an
|
||||
`Option<EntropyBias>` threaded through `--snp`/`--family-overlap`/
|
||||
`--shannon`/the fused sankoff pipeline below). If active but one or both
|
||||
are unset, defaults are `μ = 1.0`, `σ = 0.5`.
|
||||
|
||||
## Two entropy definitions kept side by side, for comparison (2026-08-15)
|
||||
|
||||
`--shannon`'s CSV carries both `entropy15` (`family_entropy` — the settled
|
||||
15-non-empty-state definition, see above) and `entropy4` (`family_entropy_4`
|
||||
— plain nucleotide reduction), computed from the same already-resolved
|
||||
`genome_mask`, not from the marginal approximation above. A genome carrying
|
||||
several bases at once contributes to *each* base's count (counted once per
|
||||
base present, not fractionally split, not folded into one combined state)
|
||||
— a genome polymorphic for the family is present at more than one base by
|
||||
construction, so it is expected to count more than once; the denominator is
|
||||
the total base-occurrence count, not the genome count (the two coincide
|
||||
only when no genome carries more than one base). Kept side by side
|
||||
specifically to measure, on real data, how much the two diverge — not yet
|
||||
analyzed.
|
||||
|
||||
## `PersistentSparseBitMatrix` — implemented and measured (2026-08-15)
|
||||
|
||||
A row-major (k-mer-major), deduplicated sparse alternative to
|
||||
`obicompactvec::PersistentBitMatrix`, motivated by the same sparsity that
|
||||
drove `--subsample`/`--shannon` above, but pursued as a foundational
|
||||
storage-layer change rather than an index-level workaround. Full design
|
||||
history, rationale, and rejected alternatives (external Elias-Fano crates,
|
||||
`cacheline-ef`, a single unsplit `dict_id` array) are in the dedicated
|
||||
implementation plan (`vivid-mapping-tiger.md` at the time of writing — the
|
||||
content below is the durable summary, not a pointer to a session-scoped
|
||||
file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi,
|
||||
"Compact Data Structures for Collections of Sets" (2025,
|
||||
`biblio/Alanko et al. - Compact Data Structures for Collections of
|
||||
Sets.pdf`) — this design implements only their exact-duplicate special
|
||||
case (a plain dedup dictionary), not their full subset-containment
|
||||
hierarchy.
|
||||
|
||||
**Design**: four on-disk components, each mmap-backed, built once per
|
||||
layer (matching how the rest of the build pipeline already works — never
|
||||
the whole multi-billion-row index at once): an `is_multi` rank-capable
|
||||
flag per row (singleton vs. multi-genome), a fixed-bit-width array for
|
||||
singleton rows (genome index directly, `ceil(log2(n_cols))` bits), a
|
||||
separate fixed-bit-width array for multi-genome rows (`dict_id`,
|
||||
`ceil(log2(n_distinct_multi_sets))` bits — kept apart from the singleton
|
||||
array specifically because `n_distinct_multi_sets` can be large in
|
||||
absolute terms even when multi-genome rows are a small *fraction* of all
|
||||
rows, and a single shared array would force every row, singletons
|
||||
included, to pay the wider width), and a deduplicated dictionary of
|
||||
distinct multi-genome sets (Elias-Fano-encoded byte offsets + a
|
||||
varint-encoded values blob). New low-level primitives added to
|
||||
`obicompactvec` to build this: `PersistentFixedIntVec` (arbitrary,
|
||||
runtime-parameterized bit width, width 0 included — needed once a real
|
||||
bug surfaced, see below), `PersistentRankSelectBitVec` (rank1/rank0/select1
|
||||
on top of the crate's existing `count_ones`, using
|
||||
`common_traits::SelectInWord`), `EliasFano` (composes the two). A new
|
||||
`BinaryMatrix` trait (`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`,
|
||||
`count_ones`) unifies dense and sparse at the one call site that needs
|
||||
both interchangeably (`obikphylo::siblings::cache::Mat`) — column-oriented
|
||||
methods (`col`, `col_view`, the `partial_*_dist_matrix` family) stay
|
||||
dense-only.
|
||||
|
||||
**Two real bugs caught by tests, not by inspection**: (1) `EliasFano::open`
|
||||
re-derived its low-bits width from the persisted low-vector file's own
|
||||
width byte; the zero-width case was built with a dummy 1-bit placeholder
|
||||
(the builder rejected true width 0), so every reopened value silently
|
||||
doubled. Fixed by making `PersistentFixedIntVec` genuinely support width 0
|
||||
(no storage, `get` always 0) instead of working around the limitation in
|
||||
`EliasFano`. (2) An empty row (cardinality 0 — not expected on a real
|
||||
built index, but not guarded against either) was recorded as a singleton
|
||||
at genome 0, indistinguishable on read-back from a *real* singleton at
|
||||
genome 0. Fixed by routing cardinality-0 rows through the dictionary path
|
||||
(a genuine empty entry) instead of the singleton shortcut. Both caught by
|
||||
`obicompactvec`'s test suite (142 tests, including disk-reopen round-trips
|
||||
that drop every builder/mmap before reopening fresh), not by manual
|
||||
review — worth remembering next time a "this edge case can't happen in
|
||||
practice" shortcut is tempting.
|
||||
|
||||
**Measured on real data** (`layer_1` of `phyloskims_sal_vac`'s
|
||||
`part_00018`, 30,246,774 rows, 91 genomes — `#[ignore]`d benchmarks in
|
||||
`obikphylo/src/siblings/tests.rs`):
|
||||
|
||||
| | dense | sparse | ratio |
|
||||
|---|---|---|---|
|
||||
| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
|
||||
| build time / peak RSS | — | 4.26s / 628MB | (per-layer, in-memory construction — comfortable) |
|
||||
| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse **faster** (smaller structure, better cache fit) |
|
||||
| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** (the real `--shannon`/family-lookup shape) |
|
||||
| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** (no native column method — every read decodes a full row to keep one bit) |
|
||||
|
||||
The row-access wins (both directions) weren't the design's stated goal —
|
||||
compactness was — but turn out real: dense's genome-major layout scatters
|
||||
a single row read across a much bigger file, which costs more than
|
||||
sparse's rank/select/varint decode once the file is this much smaller.
|
||||
The column-access cost is the flip side of the same layout choice, and is
|
||||
exactly what the next item below exists to fix.
|
||||
|
||||
**Next, not yet planned**: rewrite `partial_jaccard_dist_matrix`/
|
||||
`partial_hamming_dist_matrix`/etc. (`obicompactvec/src/bitmatrix/pairwise.rs`)
|
||||
as a row-major co-occurrence accumulation (`O(Σ_rows k²)`, per-row
|
||||
increments into an `NxN` genome-pair counter — the known alternative to
|
||||
today's column-fold, plausibly cheaper on data this sparse, not just a
|
||||
fallback) so `obikindex`'s `--metric`/distance-matrix path can use the
|
||||
sparse type without the measured 86x column-access penalty. Needs its own
|
||||
design pass (in particular how it plugs into the `BitPartials`/
|
||||
`ColumnWeights` traits so both matrix types keep serving `--metric`)
|
||||
before implementation — not just "port the loop", a genuinely different
|
||||
algorithm.
|
||||
|
||||
Also still deferred, unchanged from the implementation plan: full Alanko
|
||||
et al. subset-hierarchy compression (only the exact-duplicate special case
|
||||
is built), a sparse `PersistentCompactIntMatrix` (count matrices), and
|
||||
BRWT-style column-correlation exploitation.
|
||||
|
||||
## Wired into `pack` and the sibling-annex build path (2026-08-15)
|
||||
|
||||
`PersistentSparseBitMatrix` went from a validated but unused type to a
|
||||
real, selectable on-disk format:
|
||||
|
||||
- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
|
||||
(`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
|
||||
`D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
|
||||
`PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
|
||||
`LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
|
||||
already generic over any `D: LayerData`, so they needed no change.
|
||||
Verified by `obilayeredmap`'s
|
||||
`presence_layer_generic_over_sparse_matches_dense` test: build a dense
|
||||
presence layer, convert it to sparse via `build_from_dense`, open both
|
||||
as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
|
||||
the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree.
|
||||
(This test must stay at `k=4` with mutually non-colliding canonical
|
||||
4-mers across its input sequences — `K`/`M` are process-wide
|
||||
`AtomicUsize`s in test builds, not thread-local, so a test using a
|
||||
different `k` races every other test in the same crate binary; a k=11
|
||||
version of this test passed alone but failed under the full
|
||||
`obilayeredmap` suite for exactly that reason before being fixed.)
|
||||
- **`obikphylo::siblings::cache::Mat`** gained a third variant,
|
||||
`SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
|
||||
and `Presence` — every method (`find_slot`, `index_batch`,
|
||||
`iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches
|
||||
to it identically to `Presence`, since both go through the same generic
|
||||
`Layer<D>` code. `PartitionCache::build` picks the variant per layer by
|
||||
checking for `presence/is_multi.prsb` (the sparse format's own marker
|
||||
file, see the design section above) before falling back to the dense
|
||||
open path.
|
||||
- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`):
|
||||
`pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb`
|
||||
first); packs to dense `matrix.pbmx` first if that hasn't happened yet
|
||||
(the dense→sparse transpose needs random row access, which only the
|
||||
packed/columnar dense forms give), then `build_from_dense`s the sparse
|
||||
form into the same directory and deletes `matrix.pbmx` — old-format
|
||||
files are removed only after the new format is fully written, mirroring
|
||||
`pack_bit_matrix`'s own crash-safety convention.
|
||||
- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through
|
||||
`KmerIndex::pack_matrices` (all other call sites — `select`, `merge`,
|
||||
`finalize_indexed` — pass `false`, unchanged dense behaviour). Count
|
||||
matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix`
|
||||
— see "still deferred" above).
|
||||
- **End-to-end coverage**: `obikphylo::siblings::tests::
|
||||
sibling_annex_works_after_pack_sparse` builds a two-genome index, packs
|
||||
it `--sparse`, asserts `is_multi.prsb` exists, then runs
|
||||
`build_sibling_annex` and checks the resulting `FamilyMask`s match the
|
||||
dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
|
||||
sparse format round-trips through the real build pipeline
|
||||
(`PartitionCache` sparse-detection included), not just the
|
||||
`obicompactvec`/`obilayeredmap` unit layers below it.
|
||||
|
||||
Full workspace `cargo test` (all crates, unit + doc tests) green after
|
||||
this change.
|
||||
|
||||
## Sankoff pipeline fusion + entropy-biased selection — implemented (2026-08-15)
|
||||
|
||||
Replaces the "four independent scans" problem above and implements
|
||||
"Entropy-biased selection" above, end to end:
|
||||
|
||||
- **`SankoffBundleExt::sankoff_bundle`** (new,
|
||||
`obikphylo/src/siblings/sankoff_bundle.rs`) — the `--sankoff`/`--tnt`/
|
||||
`--phyg`/`--iqtree` block in `obikmer/src/cmd/phylo/mod.rs` now calls
|
||||
this once instead of three separate `raw_snp_distance`/
|
||||
`base_pair_tally`/`cardinality_tally` calls. One `PartitionCache`, one
|
||||
shared (possibly subsampled/entropy-biased) selection computed once via
|
||||
`compute_selections`, two scans over it: pass A (aggregate SNP/shared
|
||||
counts, `--exclude-genome` zeroing, then `included[i,j]`), pass B
|
||||
(`base_pair_tally` + `cardinality_tally` + the pseudo-alignment, fused
|
||||
into one `scan_layer_families` call per layer, all three read off the
|
||||
same resolved `genome_mask`). `snp_pseudo_alignment`/
|
||||
`shannon_entropy_csv` (still used standalone by `--snp`/
|
||||
`--family-overlap`/`--shannon`) both gained an `entropy_bias` parameter
|
||||
too, so entropy-biased selection isn't sankoff-specific.
|
||||
- **Regression proof**: `sankoff_bundle_matches_old_separate_calls`
|
||||
(`obikphylo/src/siblings/tests.rs`) asserts `sankoff_bundle`'s four
|
||||
outputs are bit-identical to calling the old, separate
|
||||
`raw_snp_distance`/`base_pair_tally`/`cardinality_tally`/
|
||||
`snp_pseudo_alignment` on the same fixture with no subsample — the
|
||||
fusion is a performance change, not a behavior change.
|
||||
- **`EntropyAnnex`/`EntropyAnnexBuilder`** (new,
|
||||
`obikphylo/src/siblings/entropy_annex.rs`) and **`ensure_entropy_annexes`**
|
||||
(`entropy.rs`) implement the persisted-entropy mechanism from
|
||||
"Entropy-biased selection" above. `entropy_annex_builds_on_demand_and_biases_selection`
|
||||
(`tests.rs`) proves, on a fixture with one known-entropy family: the
|
||||
annex file doesn't exist before any entropy-biased call; `compute_selections`
|
||||
builds it on first use; `μ` set to the family's exact entropy with
|
||||
`p₀ = 1.0` selects it deterministically (`u < 1.0` always, for
|
||||
`u ∈ [0,1)`); `μ` set far away with tiny `σ` deterministically excludes
|
||||
it (`w` underflows to exactly `0.0`); the persisted value matches
|
||||
`--shannon`'s own `family_entropy` computation to `1e-6`.
|
||||
- **`EntropyBias`** (`pub`, `obikphylo::siblings::EntropyBias { mu, sigma }`)
|
||||
is the one new public type threading `--entropy`/`--entropy-sd` through
|
||||
every `Option<EntropyBias>`-accepting method — resolved once in
|
||||
`obikmer/src/cmd/phylo/mod.rs` from `args.entropy`/`args.entropy_sd`
|
||||
(activation: either given; defaults `1.0`/`0.5` for whichever is unset).
|
||||
|
||||
Full workspace `cargo test` green after this change (167 unit tests in
|
||||
`obicompactvec`+`obilayeredmap`+`obikphylo` alone, plus every other
|
||||
crate's suite, no regressions).
|
||||
|
||||
**Still open, not part of this change** (per "Correction to the 'single
|
||||
pass' framing" above): `--raw-snp-distance`/`--raw-snp-counts` (the
|
||||
standalone diagnostic flags, not the `--sankoff` pipeline) still always
|
||||
scan the full unsampled index — never threaded `--subsample`/`--entropy`,
|
||||
out of scope here since the reported problem was specifically about the
|
||||
`--sankoff`/`--tnt` pipeline's redundant/inconsistent scans, not these
|
||||
two standalone flags.
|
||||
Reference in New Issue
Block a user