OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Add benchmark pipeline for dense and sparse query testing

Browse Source 
Introduces a complete query benchmark track to evaluate performance and verify consistency between dense and sparse index formats. Adds scripts to simulate fixed-size paired-end reads, pack a sparse presence index, execute queries in both modes, and capture wall time and RSS metrics. Includes a verification step that compares outputs by read ID to ensure content identity across parallel processing. Updates build configuration, documentation, and ignore patterns to support the new pipeline for two microbial specimens.
This commit is contained in:
Eric Coissac
2026-08-20 13:59:12 +02:00
parent 5a9d903e51
commit 89ea077456
14 changed files with 565 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmark/README.md
+50 -2
View File
@@ -75,10 +75,41 @@ flowchart TD
verify_merge_count --> vmc[("stats/verify_merge_count/")]
end
subgraph query ["Query track (2 specimens: E. coli + archaeon)"]
GENOMES --> simulate_query
simulate_query --> qdata[("query_data/")]
gpres --> pack_sparse
BIN --> pack_sparse
pack_sparse --> gsparse[("global_index_presence_sparse/")]
qdata --> query_dense
gpres --> query_dense
BIN --> query_dense
query_dense --> qd[("query_dense/")]
query_dense --> qd_stats[("stats/query_dense/")]
qd_stats --> aggregate_query_dense
qdata --> query_sparse
gsparse --> query_sparse
BIN --> query_sparse
query_sparse --> qs[("query_sparse/")]
query_sparse --> qs_stats[("stats/query_sparse/")]
qs_stats --> aggregate_query_sparse
qd --> verify_query
qs --> verify_query
verify_query --> vq_stats[("stats/verify_query/")]
vq_stats --> aggregate_verify_query
end
aggregate_verify_presence --> all
aggregate_verify_count --> all
vmp --> all
vmc --> all
aggregate_query_dense --> all
aggregate_query_sparse --> all
aggregate_verify_query --> all
all -. "$(MAKE) re-eval" .-> aggregate_filter_presence
all -. "$(MAKE) re-eval" .-> aggregate_filter_count
```
@@ -105,6 +136,14 @@ flowchart TD
| `aggregate_filter_count` | `aggregate_stats.sh` | Aggregate species-specific kmer stats (count) |
| `verify_merge_presence` | `verify_merge_presence.sh` | Verify global presence index against all reference sets |
| `verify_merge_count` | `verify_merge_count.sh` | Verify global count index against all reference sets |
| `simulate_query` | `simulate_query_one.sh` | Simulate a fixed-size (100k pairs) read set per query specimen |
| `pack_sparse` | `pack_sparse.sh` | Build `global_index_presence_sparse/` from `global_index_presence/` |
| `query_dense` | `query_one.sh dense` | Query each query specimen's reads against the dense global index |
| `query_sparse` | `query_one.sh sparse` | Query each query specimen's reads against the sparse global index |
| `aggregate_query_dense` | `aggregate_stats.sh` | Aggregate dense query wall/RSS stats |
| `aggregate_query_sparse` | `aggregate_stats.sh` | Aggregate sparse query wall/RSS stats |
| `verify_query` | `verify_query_one.sh` | Diff dense vs sparse query output per specimen (regression check) |
| `aggregate_verify_query` | `aggregate_stats.sh` | Aggregate dense/sparse query regression stats |
## Directory layout
@@ -113,13 +152,18 @@ benchmark/
├── genomes/ # input reference genomes (.fna.gz)
├── simulated_data/ # generated by simulate
│ └── <species>/<specimen>/
├── query_data/ # generated by simulate_query (2 specimens, fixed 100k pairs)
│ └── <species>/<specimen>/
├── reference_index/ # reference kmer sets (.npz)
├── specimen_index_presence/ # per-specimen presence indexes
├── specimen_index_count/ # per-specimen count indexes
├── global_index_presence/ # merged global presence index
├── global_index_presence/ # merged global presence index (dense-packed)
├── global_index_presence_sparse/ # global presence index, sparse-packed (query benchmark)
├── global_index_count/ # merged global count index
├── specific_index_presence/ # species-specific presence indexes
├── specific_index_count/ # species-specific count indexes
├── query_dense/ # query output against global_index_presence
├── query_sparse/ # query output against global_index_presence_sparse
└── stats/ # all benchmark statistics
├── indexing_presence/
├── indexing_count/
@@ -128,5 +172,9 @@ benchmark/
├── specific_kmer_presence/
├── specific_kmer_count/
├── verify_merge_presence/
└── verify_merge_count/
├── verify_merge_count/
├── pack_sparse/
├── query_dense/
├── query_sparse/
└── verify_query/
```