OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Remove in-place flag, optimize presence counting, and handle NUMA panics

Browse Source 
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
This commit is contained in:
Eric Coissac
2026-08-28 23:16:37 +02:00
parent 7363370065
commit 93fe838f97
9 changed files with 330 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmark/filter_one_count.sh
+11 -2
View File
@@ -1,7 +1,10 @@
#!/usr/bin/env bash
# Usage: filter_one_count.sh SPECIES
# Filters global_index_count to keep only kmers specific to SPECIES,
# then selects the SPECIES column in-place.
# then selects the SPECIES column (to a temp dir, swapped over the
# original — obikmer select has no --in-place mode, but its output hard-
# links each layer's unchanged kmer-identity files rather than copying
# them, so this costs no extra disk for those).
# Outputs:
# specific_index_count/SPECIES/index.done (written by obikmer select)
# stats/specific_kmer_count/SPECIES.stats (one CSV data row, no header)
@@ -37,8 +40,11 @@ trap 'rm -f "${LOG_FILTER}" "${LOG_SELECT}"' EXIT
cat "${LOG_FILTER}" >&2
SELECT_TMP="${OUTPUT}.select_tmp"
rm -rf "${SELECT_TMP}"
"${BINARY}" select \
--in-place \
--output "${SELECT_TMP}" \
--group "${SPECIES}:species=${SPECIES}" \
--group-op "${SPECIES}:any" \
--select "${SPECIES}" \
@@ -47,6 +53,9 @@ cat "${LOG_FILTER}" >&2
cat "${LOG_SELECT}" >&2
rm -rf "${OUTPUT}"
mv "${SELECT_TMP}" "${OUTPUT}"
python3 - "${SPECIES}" "${LOG_FILTER}" "${LOG_SELECT}" <<'PYEOF' >"${STATS_FILE}"
import sys, re
benchmark/filter_one_presence.sh
+11 -2
View File
@@ -1,7 +1,10 @@
#!/usr/bin/env bash
# Usage: filter_one_presence.sh SPECIES
# Filters global_index_presence to keep only kmers specific to SPECIES,
# then selects the SPECIES column in-place.
# then selects the SPECIES column (to a temp dir, swapped over the
# original — obikmer select has no --in-place mode, but its output hard-
# links each layer's unchanged kmer-identity files rather than copying
# them, so this costs no extra disk for those).
# Outputs:
# specific_index_presence/SPECIES/index.done (written by obikmer select)
# stats/specific_kmer_presence/SPECIES.stats (one CSV data row, no header)
@@ -37,8 +40,11 @@ trap 'rm -f "${LOG_FILTER}" "${LOG_SELECT}"' EXIT
cat "${LOG_FILTER}" >&2
SELECT_TMP="${OUTPUT}.select_tmp"
rm -rf "${SELECT_TMP}"
"${BINARY}" select \
--in-place \
--output "${SELECT_TMP}" \
--group "${SPECIES}:species=${SPECIES}" \
--group-op "${SPECIES}:any" \
--select "${SPECIES}" \
@@ -47,6 +53,9 @@ cat "${LOG_FILTER}" >&2
cat "${LOG_SELECT}" >&2
rm -rf "${OUTPUT}"
mv "${SELECT_TMP}" "${OUTPUT}"
python3 - "${SPECIES}" "${LOG_FILTER}" "${LOG_SELECT}" <<'PYEOF' >"${STATS_FILE}"
import sys, re