Eric Coissac
89ea077456
Introduces a complete query benchmark track to evaluate performance and verify consistency between dense and sparse index formats. Adds scripts to simulate fixed-size paired-end reads, pack a sparse presence index, execute queries in both modes, and capture wall time and RSS metrics. Includes a verification step that compares outputs by read ID to ensure content identity across parallel processing. Updates build configuration, documentation, and ignore patterns to support the new pipeline for two microbial specimens.
181 lines
8.0 KiB
Markdown
181 lines
8.0 KiB
Markdown
# Benchmark pipeline
|
|
|
|
Requires **GNU Make ≥ 4.3** (grouped targets `&:`). On macOS use `gmake`.
|
|
|
|
```
|
|
gmake all # full pipeline
|
|
gmake simulate # simulation only
|
|
gmake reference # reference kmer sets only
|
|
```
|
|
|
|
## Pipeline overview
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
GENOMES["genomes/*.fna.gz"]
|
|
BIN["obikmer binary"]
|
|
|
|
GENOMES --> simulate
|
|
simulate --> simdata[("simulated_data/")]
|
|
|
|
simdata --> reference
|
|
reference --> refnpz[("reference_index/*.npz")]
|
|
|
|
subgraph presence ["Presence track"]
|
|
simdata --> index_presence
|
|
BIN --> index_presence
|
|
index_presence --> pres_done[("specimen_index_presence/")]
|
|
index_presence --> pres_istats[("stats/indexing_presence/")]
|
|
pres_istats --> aggregate_index_presence
|
|
|
|
pres_done --> merge_presence
|
|
BIN --> merge_presence
|
|
merge_presence --> gpres[("global_index_presence/")]
|
|
|
|
refnpz --> verify_presence
|
|
pres_done --> verify_presence
|
|
verify_presence --> vpres_stats[("stats/verify_presence/")]
|
|
vpres_stats --> aggregate_verify_presence
|
|
|
|
gpres --> filter_presence
|
|
BIN --> filter_presence
|
|
filter_presence --> spec_pres[("specific_index_presence/")]
|
|
filter_presence --> spec_pres_stats[("stats/specific_kmer_presence/")]
|
|
spec_pres_stats --> aggregate_filter_presence
|
|
|
|
refnpz --> verify_merge_presence
|
|
gpres --> verify_merge_presence
|
|
verify_merge_presence --> vmp[("stats/verify_merge_presence/")]
|
|
end
|
|
|
|
subgraph count ["Count track"]
|
|
simdata --> index_count
|
|
BIN --> index_count
|
|
index_count --> count_done[("specimen_index_count/")]
|
|
index_count --> count_istats[("stats/indexing_count/")]
|
|
count_istats --> aggregate_index_count
|
|
|
|
count_done --> merge_count
|
|
BIN --> merge_count
|
|
merge_count --> gcount[("global_index_count/")]
|
|
|
|
refnpz --> verify_count
|
|
count_done --> verify_count
|
|
verify_count --> vcount_stats[("stats/verify_count/")]
|
|
vcount_stats --> aggregate_verify_count
|
|
|
|
gcount --> filter_count
|
|
BIN --> filter_count
|
|
filter_count --> spec_count[("specific_index_count/")]
|
|
filter_count --> spec_count_stats[("stats/specific_kmer_count/")]
|
|
spec_count_stats --> aggregate_filter_count
|
|
|
|
refnpz --> verify_merge_count
|
|
gcount --> verify_merge_count
|
|
verify_merge_count --> vmc[("stats/verify_merge_count/")]
|
|
end
|
|
|
|
subgraph query ["Query track (2 specimens: E. coli + archaeon)"]
|
|
GENOMES --> simulate_query
|
|
simulate_query --> qdata[("query_data/")]
|
|
|
|
gpres --> pack_sparse
|
|
BIN --> pack_sparse
|
|
pack_sparse --> gsparse[("global_index_presence_sparse/")]
|
|
|
|
qdata --> query_dense
|
|
gpres --> query_dense
|
|
BIN --> query_dense
|
|
query_dense --> qd[("query_dense/")]
|
|
query_dense --> qd_stats[("stats/query_dense/")]
|
|
qd_stats --> aggregate_query_dense
|
|
|
|
qdata --> query_sparse
|
|
gsparse --> query_sparse
|
|
BIN --> query_sparse
|
|
query_sparse --> qs[("query_sparse/")]
|
|
query_sparse --> qs_stats[("stats/query_sparse/")]
|
|
qs_stats --> aggregate_query_sparse
|
|
|
|
qd --> verify_query
|
|
qs --> verify_query
|
|
verify_query --> vq_stats[("stats/verify_query/")]
|
|
vq_stats --> aggregate_verify_query
|
|
end
|
|
|
|
aggregate_verify_presence --> all
|
|
aggregate_verify_count --> all
|
|
vmp --> all
|
|
vmc --> all
|
|
aggregate_query_dense --> all
|
|
aggregate_query_sparse --> all
|
|
aggregate_verify_query --> all
|
|
all -. "$(MAKE) re-eval" .-> aggregate_filter_presence
|
|
all -. "$(MAKE) re-eval" .-> aggregate_filter_count
|
|
```
|
|
|
|
## Steps
|
|
|
|
| Target | Script | Description |
|
|
|---|---|---|
|
|
| `simulate` | `simulate.sh` | Simulate sequencing reads from the reference genomes |
|
|
| `reference` | `build_reference.sh` | Build reference kmer sets (`.npz`) from simulation truth |
|
|
| `index_presence` | `index_one_presence.sh` | Index each specimen (presence mode) |
|
|
| `index_count` | `index_one_count.sh` | Index each specimen (count mode) |
|
|
| `aggregate_index_presence` | `aggregate_stats.sh` | Aggregate per-specimen indexing stats (presence) |
|
|
| `aggregate_index_count` | `aggregate_stats.sh` | Aggregate per-specimen indexing stats (count) |
|
|
| `merge_presence` | `merge_presence.sh` | Merge all specimen presence indexes into a global index |
|
|
| `merge_count` | `merge_count.sh` | Merge all specimen count indexes into a global index |
|
|
| `verify_presence` | `verify_one_presence.sh` | Verify each specimen presence index against reference |
|
|
| `verify_count` | `verify_one_count.sh` | Verify each specimen count index against reference |
|
|
| `aggregate_verify_presence` | `aggregate_stats.sh` | Aggregate per-specimen verification stats (presence) |
|
|
| `aggregate_verify_count` | `aggregate_stats.sh` | Aggregate per-specimen verification stats (count) |
|
|
| `filter_presence` | `filter_one_presence.sh` | Extract species-specific presence indexes from global index |
|
|
| `filter_count` | `filter_one_count.sh` | Extract species-specific count indexes from global index |
|
|
| `aggregate_filter_presence` | `aggregate_stats.sh` | Aggregate species-specific kmer stats (presence) |
|
|
| `aggregate_filter_count` | `aggregate_stats.sh` | Aggregate species-specific kmer stats (count) |
|
|
| `verify_merge_presence` | `verify_merge_presence.sh` | Verify global presence index against all reference sets |
|
|
| `verify_merge_count` | `verify_merge_count.sh` | Verify global count index against all reference sets |
|
|
| `simulate_query` | `simulate_query_one.sh` | Simulate a fixed-size (100k pairs) read set per query specimen |
|
|
| `pack_sparse` | `pack_sparse.sh` | Build `global_index_presence_sparse/` from `global_index_presence/` |
|
|
| `query_dense` | `query_one.sh dense` | Query each query specimen's reads against the dense global index |
|
|
| `query_sparse` | `query_one.sh sparse` | Query each query specimen's reads against the sparse global index |
|
|
| `aggregate_query_dense` | `aggregate_stats.sh` | Aggregate dense query wall/RSS stats |
|
|
| `aggregate_query_sparse` | `aggregate_stats.sh` | Aggregate sparse query wall/RSS stats |
|
|
| `verify_query` | `verify_query_one.sh` | Diff dense vs sparse query output per specimen (regression check) |
|
|
| `aggregate_verify_query` | `aggregate_stats.sh` | Aggregate dense/sparse query regression stats |
|
|
|
|
## Directory layout
|
|
|
|
```
|
|
benchmark/
|
|
├── genomes/ # input reference genomes (.fna.gz)
|
|
├── simulated_data/ # generated by simulate
|
|
│ └── <species>/<specimen>/
|
|
├── query_data/ # generated by simulate_query (2 specimens, fixed 100k pairs)
|
|
│ └── <species>/<specimen>/
|
|
├── reference_index/ # reference kmer sets (.npz)
|
|
├── specimen_index_presence/ # per-specimen presence indexes
|
|
├── specimen_index_count/ # per-specimen count indexes
|
|
├── global_index_presence/ # merged global presence index (dense-packed)
|
|
├── global_index_presence_sparse/ # global presence index, sparse-packed (query benchmark)
|
|
├── global_index_count/ # merged global count index
|
|
├── specific_index_presence/ # species-specific presence indexes
|
|
├── specific_index_count/ # species-specific count indexes
|
|
├── query_dense/ # query output against global_index_presence
|
|
├── query_sparse/ # query output against global_index_presence_sparse
|
|
└── stats/ # all benchmark statistics
|
|
├── indexing_presence/
|
|
├── indexing_count/
|
|
├── verify_presence/
|
|
├── verify_count/
|
|
├── specific_kmer_presence/
|
|
├── specific_kmer_count/
|
|
├── verify_merge_presence/
|
|
├── verify_merge_count/
|
|
├── pack_sparse/
|
|
├── query_dense/
|
|
├── query_sparse/
|
|
└── verify_query/
|
|
```
|