pack

Pack an index's per-column matrix files into a single-file format to reduce query-time I/O (fewer file opens per query).

obikmer pack INDEX [--sparse]

Arguments

Argument	Description
INDEX	Index directory to pack (modified in place)

Options

Option	Default	Description
--sparse	off	Pack presence/absence and count matrices into a sparse, deduplicated format instead of the dense one

The index directory is locked for exclusive access while packing.

--sparse

Matrix data (which genomes carry each kmer, or with what count) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one entry per genome for every kmer regardless of how many genomes actually carry it; --sparse instead stores each kmer's genome list directly. For presence/absence matrices, identical genome lists shared by many kmers are also deduplicated (common in real data, since kmers from the same conserved region tend to be carried by the same genomes); for count matrices, the genome list is deduplicated the same way but each kmer's actual counts are kept per-kmer, since two kmers sharing the same genome list rarely carry the same counts.

On real genome collections this has measured at roughly 7x smaller on disk than the dense format for presence/absence, and single-kmer lookups (the shape phylo's sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by --distance matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no --sparse) for indexes you mainly query with phylo's --distance matrices.

--sparse applies to both presence/absence and count matrices — a count index (--distance matrix computations included) is packed sparse the same as a presence index.