Add sparse storage variant to PersistentCompactIntMatrix
Introduce a new `Sparse` format alongside existing `Columnar` and `Packed` variants, enabling optimized row-major pairwise counting for distance and similarity metrics via the `CountPartials` trait. Update storage detection priorities, extend matrix dispatch logic to sparse backends, and correct diagonal/off-diagonal formulas in bit matrix partial computations. Expand layer APIs with format-agnostic `nonzero_iter`, update usage documentation for the `--sparse` flag, and add comprehensive tests verifying roundtrip integrity and metric equivalence against dense implementations.
This commit is contained in:
@@ -16,14 +16,14 @@ obikmer pack INDEX [--sparse]
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--sparse` | off | Pack presence/absence matrices into a sparse, deduplicated format instead of the dense one |
|
||||
| `--sparse` | off | Pack presence/absence and count matrices into a sparse, deduplicated format instead of the dense one |
|
||||
|
||||
The index directory is locked for exclusive access while packing.
|
||||
|
||||
## `--sparse`
|
||||
|
||||
Presence/absence data (which genomes carry each kmer) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one bit per genome for every kmer regardless of how many genomes actually carry it; `--sparse` instead stores each kmer's genome list directly, and deduplicates identical lists shared by many kmers (common in real data, since kmers from the same conserved region tend to be carried by the same genomes).
|
||||
Matrix data (which genomes carry each kmer, or with what count) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one entry per genome for every kmer regardless of how many genomes actually carry it; `--sparse` instead stores each kmer's genome list directly. For presence/absence matrices, identical genome lists shared by many kmers are also deduplicated (common in real data, since kmers from the same conserved region tend to be carried by the same genomes); for count matrices, the genome list is deduplicated the same way but each kmer's actual counts are kept per-kmer, since two kmers sharing the same genome list rarely carry the same counts.
|
||||
|
||||
On real genome collections this has measured at roughly 7x smaller on disk than the dense format, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--metric` distance-matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s plain `--metric` distance matrices.
|
||||
On real genome collections this has measured at roughly 7x smaller on disk than the dense format for presence/absence, and single-kmer lookups (the shape `phylo`'s sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by `--metric` distance-matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no `--sparse`) for indexes you mainly query with `phylo`'s plain `--metric` distance matrices.
|
||||
|
||||
Count matrices (`--metric` on a count index) are not affected by `--sparse` — only presence/absence matrices are.
|
||||
`--sparse` applies to both presence/absence and count matrices — a count index (`--metric` distance matrices included) is packed sparse the same as a presence index.
|
||||
|
||||
Reference in New Issue
Block a user