OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Remove in-place flag, optimize presence counting, and handle NUMA panics

Browse Source 
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
This commit is contained in:
Eric Coissac
2026-08-28 23:16:37 +02:00
parent 7363370065
commit 93fe838f97
9 changed files with 330 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
UserDocMD/usage/select.md
+18 -5
View File
@@ -1,9 +1,9 @@
# select
Project and/or aggregate the genome columns of an index into a new (or in-place) index. Where [`filter`](filter.md) selects rows (kmers), `select` operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.
Project and/or aggregate the genome columns of an index into a new index. Where [`filter`](filter.md) selects rows (kmers), `select` operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.
```bash
obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
obikmer select SOURCE --output OUTPUT [OPTIONS]
```
## Arguments
@@ -16,8 +16,7 @@ obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
| Option | Default | Description |
|---|---|---|
| `--output` | — | Output index directory (mutually exclusive with `--in-place`) |
| `--in-place` | off | Rewrite the source index in place (mutually exclusive with `--output`) |
| `-o, --output` | — | Output index directory (required) |
| `-f, --force` | off | Overwrite an existing output directory |
| `--group NAME:PRED` | none | Define a named group of genomes by predicate (repeatable; mutually exclusive with `--aggregate-by`) |
| `--group-op NAME:OP` | none | Aggregation operator for a named group |
@@ -25,6 +24,8 @@ obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
| `--aggregate-op OP` | none | Aggregation operator applied to every auto-generated group |
| `--select COL,...` | all columns | Output columns, in order (group names or genome labels) |
| `--presence-threshold` | `0` | Minimum count for a genome to be considered a carrier (logical operators only) |
| `--dense` | off | Pack the output's presence matrices in the dense format instead of the default sparse one |
| `--force-copy` | off | Copy each layer's unchanged kmer-identity files (mphf/unitigs/evidence/fingerprint) instead of hard-linking them |
## Aggregation operators
@@ -32,4 +33,16 @@ obikmer select SOURCE (--output OUTPUT | --in-place) [OPTIONS]
A `select` never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in `--select`) is a cheap copy.
At least one of `--output`/`--in-place` is required, and at least one output column must be defined; every name listed in `--select` must resolve to either a defined group or an existing genome label. See [Genome predicates and taxonomy paths](predicates.md) for the predicate syntax used by `--group`.
At least one output column must be defined; every name listed in `--select` must resolve to either a defined group or an existing genome label. See [Genome predicates and taxonomy paths](predicates.md) for the predicate syntax used by `--group`.
## Disk usage
`select` always writes to a new output directory — there is no in-place mode. Each layer's kmer-identity files (MPHF, unitigs, evidence, fingerprint) never change under a column projection/aggregation, so they are hard-linked into the output rather than copied: no extra disk is used for them, even on a very large index. Linking falls back to a real copy automatically if it fails (e.g. `SOURCE`/`OUTPUT` on different filesystems). Use `--force-copy` to always copy instead — needed when the output must be able to survive independently of the source on disk (a hard link shares the same underlying data, so overwriting one path outside `select` itself would affect the other).
To replace an index with a selected version of itself, select to a temporary directory and swap it in:
```bash
obikmer select INDEX --output INDEX.tmp --group ... --group-op ... --select ...
rm -rf INDEX
mv INDEX.tmp INDEX
```