Eric Coissac
93fe838f97
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
3.1 KiB
select
Project and/or aggregate the genome columns of an index into a new index. Where filter selects rows (kmers), select operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.
obikmer select SOURCE --output OUTPUT [OPTIONS]
Arguments
| Argument | Description |
|---|---|
SOURCE |
Source index directory |
Options
| Option | Default | Description |
|---|---|---|
-o, --output |
— | Output index directory (required) |
-f, --force |
off | Overwrite an existing output directory |
--group NAME:PRED |
none | Define a named group of genomes by predicate (repeatable; mutually exclusive with --aggregate-by) |
--group-op NAME:OP |
none | Aggregation operator for a named group |
--aggregate-by KEY |
none | Automatically create one group per distinct value of a metadata key (mutually exclusive with --group) |
--aggregate-op OP |
none | Aggregation operator applied to every auto-generated group |
--select COL,... |
all columns | Output columns, in order (group names or genome labels) |
--presence-threshold |
0 |
Minimum count for a genome to be considered a carrier (logical operators only) |
--dense |
off | Pack the output's presence matrices in the dense format instead of the default sparse one |
--force-copy |
off | Copy each layer's unchanged kmer-identity files (mphf/unitigs/evidence/fingerprint) instead of hard-linking them |
Aggregation operators
any, all, none (logical, evaluated against --presence-threshold), sum, min, max (numeric, count index only). If a group's operator is left unspecified, it defaults to any when the source is a presence/absence index and sum when it stores counts.
A select never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in --select) is a cheap copy.
At least one output column must be defined; every name listed in --select must resolve to either a defined group or an existing genome label. See Genome predicates and taxonomy paths for the predicate syntax used by --group.
Disk usage
select always writes to a new output directory — there is no in-place mode. Each layer's kmer-identity files (MPHF, unitigs, evidence, fingerprint) never change under a column projection/aggregation, so they are hard-linked into the output rather than copied: no extra disk is used for them, even on a very large index. Linking falls back to a real copy automatically if it fails (e.g. SOURCE/OUTPUT on different filesystems). Use --force-copy to always copy instead — needed when the output must be able to survive independently of the source on disk (a hard link shares the same underlying data, so overwriting one path outside select itself would affect the other).
To replace an index with a selected version of itself, select to a temporary directory and swap it in:
obikmer select INDEX --output INDEX.tmp --group ... --group-op ... --select ...
rm -rf INDEX
mv INDEX.tmp INDEX