feat(obicompactvec): introduce sparse bit matrix with supporting primitives

Implements a compact, row-major sparse bit matrix backed by memory-mapped components, introducing EliasFano, PersistentFixedIntVec, and PersistentRankSelectBitVec primitives for efficient storage and decoding. Adds a BinaryMatrix trait to unify row-level operations across dense and sparse implementations. Corrects edge-case behaviors for zero-width bit storage and cardinality-0 rows. Delivers reduced on-disk size and faster random row access, with column reads remaining dense-only. Test suites and benchmarks are included but currently marked as ignored.

docmd/architecture/siblings.md

@@ -403,3 +403,96 @@ the total base-occurrence count, not the genome count (the two coincide
 only when no genome carries more than one base). Kept side by side
 specifically to measure, on real data, how much the two diverge — not yet
 analyzed.
+
+## `PersistentSparseBitMatrix` — implemented and measured (2026-08-15)
+
+A row-major (k-mer-major), deduplicated sparse alternative to
+`obicompactvec::PersistentBitMatrix`, motivated by the same sparsity that
+drove `--subsample`/`--shannon` above, but pursued as a foundational
+storage-layer change rather than an index-level workaround. Full design
+history, rationale, and rejected alternatives (external Elias-Fano crates,
+`cacheline-ef`, a single unsplit `dict_id` array) are in the dedicated
+implementation plan (`vivid-mapping-tiger.md` at the time of writing — the
+content below is the durable summary, not a pointer to a session-scoped
+file). Also directly informed by Alanko, Bille, Gørtz, Navarro, Puglisi,
+"Compact Data Structures for Collections of Sets" (2025,
+`biblio/Alanko et al. - Compact Data Structures for Collections of
+Sets.pdf`) — this design implements only their exact-duplicate special
+case (a plain dedup dictionary), not their full subset-containment
+hierarchy.
+
+**Design**: four on-disk components, each mmap-backed, built once per
+layer (matching how the rest of the build pipeline already works — never
+the whole multi-billion-row index at once): an `is_multi` rank-capable
+flag per row (singleton vs. multi-genome), a fixed-bit-width array for
+singleton rows (genome index directly, `ceil(log2(n_cols))` bits), a
+separate fixed-bit-width array for multi-genome rows (`dict_id`,
+`ceil(log2(n_distinct_multi_sets))` bits — kept apart from the singleton
+array specifically because `n_distinct_multi_sets` can be large in
+absolute terms even when multi-genome rows are a small *fraction* of all
+rows, and a single shared array would force every row, singletons
+included, to pay the wider width), and a deduplicated dictionary of
+distinct multi-genome sets (Elias-Fano-encoded byte offsets + a
+varint-encoded values blob). New low-level primitives added to
+`obicompactvec` to build this: `PersistentFixedIntVec` (arbitrary,
+runtime-parameterized bit width, width 0 included — needed once a real
+bug surfaced, see below), `PersistentRankSelectBitVec` (rank1/rank0/select1
+on top of the crate's existing `count_ones`, using
+`common_traits::SelectInWord`), `EliasFano` (composes the two). A new
+`BinaryMatrix` trait (`n`, `n_cols`, `row`/`fill_row`, `fill_sub_matrix`,
+`count_ones`) unifies dense and sparse at the one call site that needs
+both interchangeably (`obikphylo::siblings::cache::Mat`) — column-oriented
+methods (`col`, `col_view`, the `partial_*_dist_matrix` family) stay
+dense-only.
+
+**Two real bugs caught by tests, not by inspection**: (1) `EliasFano::open`
+re-derived its low-bits width from the persisted low-vector file's own
+width byte; the zero-width case was built with a dummy 1-bit placeholder
+(the builder rejected true width 0), so every reopened value silently
+doubled. Fixed by making `PersistentFixedIntVec` genuinely support width 0
+(no storage, `get` always 0) instead of working around the limitation in
+`EliasFano`. (2) An empty row (cardinality 0 — not expected on a real
+built index, but not guarded against either) was recorded as a singleton
+at genome 0, indistinguishable on read-back from a *real* singleton at
+genome 0. Fixed by routing cardinality-0 rows through the dictionary path
+(a genuine empty entry) instead of the singleton shortcut. Both caught by
+`obicompactvec`'s test suite (142 tests, including disk-reopen round-trips
+that drop every builder/mmap before reopening fresh), not by manual
+review — worth remembering next time a "this edge case can't happen in
+practice" shortcut is tempting.
+
+**Measured on real data** (`layer_1` of `phyloskims_sal_vac`'s
+`part_00018`, 30,246,774 rows, 91 genomes — `#[ignore]`d benchmarks in
+`obikphylo/src/siblings/tests.rs`):
+
+| | dense | sparse | ratio |
+|---|---|---|---|
+| on-disk size | 328.1MB | 43.7MB | **7.5x** smaller |
+| build time / peak RSS | — | 4.26s / 628MB | (per-layer, in-memory construction — comfortable) |
+| row access, sequential (2M reads) | 43ns/row | 32ns/row | sparse **faster** (smaller structure, better cache fit) |
+| row access, random (2M reads) | 409ns/row | 85ns/row | sparse **~4.8x faster** (the real `--shannon`/family-lookup shape) |
+| column access, one full column (30.2M rows) | 11.5ms | 993ms | sparse **86x slower** (no native column method — every read decodes a full row to keep one bit) |
+
+The row-access wins (both directions) weren't the design's stated goal —
+compactness was — but turn out real: dense's genome-major layout scatters
+a single row read across a much bigger file, which costs more than
+sparse's rank/select/varint decode once the file is this much smaller.
+The column-access cost is the flip side of the same layout choice, and is
+exactly what the next item below exists to fix.
+
+**Next, not yet planned**: rewrite `partial_jaccard_dist_matrix`/
+`partial_hamming_dist_matrix`/etc. (`obicompactvec/src/bitmatrix/pairwise.rs`)
+as a row-major co-occurrence accumulation (`O(Σ_rows k²)`, per-row
+increments into an `NxN` genome-pair counter — the known alternative to
+today's column-fold, plausibly cheaper on data this sparse, not just a
+fallback) so `obikindex`'s `--metric`/distance-matrix path can use the
+sparse type without the measured 86x column-access penalty. Needs its own
+design pass (in particular how it plugs into the `BitPartials`/
+`ColumnWeights` traits so both matrix types keep serving `--metric`)
+before implementation — not just "port the loop", a genuinely different
+algorithm.
+
+Also still deferred, unchanged from the implementation plan: full Alanko
+et al. subset-hierarchy compression (only the exact-duplicate special case
+is built), a sparse `PersistentCompactIntMatrix` (count matrices), and
+BRWT-style column-correlation exploitation.