pack

Pack an index's per-column matrix files into a single-file format to reduce query-time I/O (fewer file opens per query).

obikmer pack INDEX [--sparse]

Arguments

Argument	Description
INDEX	Index directory to pack (modified in place)

Options

Option	Default	Description
--sparse	off	Pack presence/absence matrices into a sparse, deduplicated format instead of the dense one

The index directory is locked for exclusive access while packing.

--sparse

Presence/absence data (which genomes carry each kmer) is often mostly empty — most kmers are present in only a handful of genomes out of the whole collection. The default (dense) packed format stores one bit per genome for every kmer regardless of how many genomes actually carry it; --sparse instead stores each kmer's genome list directly, and deduplicates identical lists shared by many kmers (common in real data, since kmers from the same conserved region tend to be carried by the same genomes).

On real genome collections this has measured at roughly 7x smaller on disk than the dense format, and single-kmer lookups (the shape phylo's sibling-annex/entropy/Sankoff computations use) are typically faster too, since the smaller files mean less data to read from disk. The trade-off: reading a whole genome column at once (used by --metric distance-matrix computations) is much slower on the sparse format than on the dense one, since there is no native column layout to read sequentially — prefer the dense format (the default, no --sparse) for indexes you mainly query with phylo's plain --metric distance matrices.

Count matrices (--metric on a count index) are not affected by --sparse — only presence/absence matrices are.