feat: add sparse on-disk format for presence matrices

The index packing API now accepts a `sparse` parameter to generate `PersistentSparseBitMatrix` files alongside existing dense matrices. The sibling cache automatically detects this format via an `is_multi.prsb` marker file and routes queries identically to the dense variant. A new `--sparse` CLI flag exposes the option, with tests verifying end-to-end pipeline correctness and storage equivalence.
This commit is contained in:
Eric Coissac
2026-08-16 21:51:02 +02:00
parent 50f4820cb9
commit 3ba26b3dc1
14 changed files with 279 additions and 24 deletions
+56
View File
@@ -496,3 +496,59 @@ Also still deferred, unchanged from the implementation plan: full Alanko
et al. subset-hierarchy compression (only the exact-duplicate special case
is built), a sparse `PersistentCompactIntMatrix` (count matrices), and
BRWT-style column-correlation exploitation.
## Wired into `pack` and the sibling-annex build path (2026-08-15)
`PersistentSparseBitMatrix` went from a validated but unused type to a
real, selectable on-disk format:
- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
(`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
`D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
`PersistentBitMatrix``PersistentSparseBitMatrix` implements
`LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
already generic over any `D: LayerData`, so they needed no change.
Verified by `obilayeredmap`'s
`presence_layer_generic_over_sparse_matches_dense` test: build a dense
presence layer, convert it to sparse via `build_from_dense`, open both
as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree.
(This test must stay at `k=4` with mutually non-colliding canonical
4-mers across its input sequences — `K`/`M` are process-wide
`AtomicUsize`s in test builds, not thread-local, so a test using a
different `k` races every other test in the same crate binary; a k=11
version of this test passed alone but failed under the full
`obilayeredmap` suite for exactly that reason before being fixed.)
- **`obikphylo::siblings::cache::Mat`** gained a third variant,
`SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
and `Presence` — every method (`find_slot`, `index_batch`,
`iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches
to it identically to `Presence`, since both go through the same generic
`Layer<D>` code. `PartitionCache::build` picks the variant per layer by
checking for `presence/is_multi.prsb` (the sparse format's own marker
file, see the design section above) before falling back to the dense
open path.
- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`):
`pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb`
first); packs to dense `matrix.pbmx` first if that hasn't happened yet
(the dense→sparse transpose needs random row access, which only the
packed/columnar dense forms give), then `build_from_dense`s the sparse
form into the same directory and deletes `matrix.pbmx` — old-format
files are removed only after the new format is fully written, mirroring
`pack_bit_matrix`'s own crash-safety convention.
- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through
`KmerIndex::pack_matrices` (all other call sites — `select`, `merge`,
`finalize_indexed` — pass `false`, unchanged dense behaviour). Count
matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix`
— see "still deferred" above).
- **End-to-end coverage**: `obikphylo::siblings::tests::
sibling_annex_works_after_pack_sparse` builds a two-genome index, packs
it `--sparse`, asserts `is_multi.prsb` exists, then runs
`build_sibling_annex` and checks the resulting `FamilyMask`s match the
dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
sparse format round-trips through the real build pipeline
(`PartitionCache` sparse-detection included), not just the
`obicompactvec`/`obilayeredmap` unit layers below it.
Full workspace `cargo test` (all crates, unit + doc tests) green after
this change.