feat: add sparse on-disk format for presence matrices
The index packing API now accepts a `sparse` parameter to generate `PersistentSparseBitMatrix` files alongside existing dense matrices. The sibling cache automatically detects this format via an `is_multi.prsb` marker file and routes queries identically to the dense variant. A new `--sparse` CLI flag exposes the option, with tests verifying end-to-end pipeline correctness and storage equivalence.
This commit is contained in:
@@ -496,3 +496,59 @@ Also still deferred, unchanged from the implementation plan: full Alanko
|
||||
et al. subset-hierarchy compression (only the exact-duplicate special case
|
||||
is built), a sparse `PersistentCompactIntMatrix` (count matrices), and
|
||||
BRWT-style column-correlation exploitation.
|
||||
|
||||
## Wired into `pack` and the sibling-annex build path (2026-08-15)
|
||||
|
||||
`PersistentSparseBitMatrix` went from a validated but unused type to a
|
||||
real, selectable on-disk format:
|
||||
|
||||
- **Generic `Layer<D>`**: `obilayeredmap::Layer<D>`'s presence-only methods
|
||||
(`n_cols`, `sub_matrix`, `fill_sub_matrix`) are generic over any
|
||||
`D: LayerData<Item = Box<[bool]>> + BinaryMatrix`, not hardcoded to
|
||||
`PersistentBitMatrix` — `PersistentSparseBitMatrix` implements
|
||||
`LayerData` (`open`/`read`) the same way. `find_slot`/`index_batch` were
|
||||
already generic over any `D: LayerData`, so they needed no change.
|
||||
Verified by `obilayeredmap`'s
|
||||
`presence_layer_generic_over_sparse_matches_dense` test: build a dense
|
||||
presence layer, convert it to sparse via `build_from_dense`, open both
|
||||
as `Layer<PersistentBitMatrix>`/`Layer<PersistentSparseBitMatrix>` on
|
||||
the same directory, assert `n_cols`/`sub_matrix`/`find_slot` agree.
|
||||
(This test must stay at `k=4` with mutually non-colliding canonical
|
||||
4-mers across its input sequences — `K`/`M` are process-wide
|
||||
`AtomicUsize`s in test builds, not thread-local, so a test using a
|
||||
different `k` races every other test in the same crate binary; a k=11
|
||||
version of this test passed alone but failed under the full
|
||||
`obilayeredmap` suite for exactly that reason before being fixed.)
|
||||
- **`obikphylo::siblings::cache::Mat`** gained a third variant,
|
||||
`SparsePresence(Layer<PersistentSparseBitMatrix>)`, alongside `Count`
|
||||
and `Presence` — every method (`find_slot`, `index_batch`,
|
||||
`iter_minorants_batch`, `n_cols`, `fill_sub_matrix_carries`) dispatches
|
||||
to it identically to `Presence`, since both go through the same generic
|
||||
`Layer<D>` code. `PartitionCache::build` picks the variant per layer by
|
||||
checking for `presence/is_multi.prsb` (the sparse format's own marker
|
||||
file, see the design section above) before falling back to the dense
|
||||
open path.
|
||||
- **`pack_sparse_bit_matrix`** (new, `obicompactvec::bitmatrix::sparse`):
|
||||
`pack --sparse`'s entry point. Idempotent (checks `is_multi.prsb`
|
||||
first); packs to dense `matrix.pbmx` first if that hasn't happened yet
|
||||
(the dense→sparse transpose needs random row access, which only the
|
||||
packed/columnar dense forms give), then `build_from_dense`s the sparse
|
||||
form into the same directory and deletes `matrix.pbmx` — old-format
|
||||
files are removed only after the new format is fully written, mirroring
|
||||
`pack_bit_matrix`'s own crash-safety convention.
|
||||
- **CLI**: `obikmer pack --sparse` threads a `sparse: bool` through
|
||||
`KmerIndex::pack_matrices` (all other call sites — `select`, `merge`,
|
||||
`finalize_indexed` — pass `false`, unchanged dense behaviour). Count
|
||||
matrices are untouched by `--sparse` (no sparse `PersistentCompactIntMatrix`
|
||||
— see "still deferred" above).
|
||||
- **End-to-end coverage**: `obikphylo::siblings::tests::
|
||||
sibling_annex_works_after_pack_sparse` builds a two-genome index, packs
|
||||
it `--sparse`, asserts `is_multi.prsb` exists, then runs
|
||||
`build_sibling_annex` and checks the resulting `FamilyMask`s match the
|
||||
dense-path test (`sibling_annex_one_sibling_each`) exactly — proves the
|
||||
sparse format round-trips through the real build pipeline
|
||||
(`PartitionCache` sparse-detection included), not just the
|
||||
`obicompactvec`/`obilayeredmap` unit layers below it.
|
||||
|
||||
Full workspace `cargo test` (all crates, unit + doc tests) green after
|
||||
this change.
|
||||
|
||||
Reference in New Issue
Block a user