Eric Coissac
89ea077456
Introduces a complete query benchmark track to evaluate performance and verify consistency between dense and sparse index formats. Adds scripts to simulate fixed-size paired-end reads, pack a sparse presence index, execute queries in both modes, and capture wall time and RSS metrics. Includes a verification step that compares outputs by read ID to ensure content identity across parallel processing. Updates build configuration, documentation, and ignore patterns to support the new pipeline for two microbial specimens.
8.0 KiB
8.0 KiB
Benchmark pipeline
Requires GNU Make ≥ 4.3 (grouped targets &:). On macOS use gmake.
gmake all # full pipeline
gmake simulate # simulation only
gmake reference # reference kmer sets only
Pipeline overview
flowchart TD
GENOMES["genomes/*.fna.gz"]
BIN["obikmer binary"]
GENOMES --> simulate
simulate --> simdata[("simulated_data/")]
simdata --> reference
reference --> refnpz[("reference_index/*.npz")]
subgraph presence ["Presence track"]
simdata --> index_presence
BIN --> index_presence
index_presence --> pres_done[("specimen_index_presence/")]
index_presence --> pres_istats[("stats/indexing_presence/")]
pres_istats --> aggregate_index_presence
pres_done --> merge_presence
BIN --> merge_presence
merge_presence --> gpres[("global_index_presence/")]
refnpz --> verify_presence
pres_done --> verify_presence
verify_presence --> vpres_stats[("stats/verify_presence/")]
vpres_stats --> aggregate_verify_presence
gpres --> filter_presence
BIN --> filter_presence
filter_presence --> spec_pres[("specific_index_presence/")]
filter_presence --> spec_pres_stats[("stats/specific_kmer_presence/")]
spec_pres_stats --> aggregate_filter_presence
refnpz --> verify_merge_presence
gpres --> verify_merge_presence
verify_merge_presence --> vmp[("stats/verify_merge_presence/")]
end
subgraph count ["Count track"]
simdata --> index_count
BIN --> index_count
index_count --> count_done[("specimen_index_count/")]
index_count --> count_istats[("stats/indexing_count/")]
count_istats --> aggregate_index_count
count_done --> merge_count
BIN --> merge_count
merge_count --> gcount[("global_index_count/")]
refnpz --> verify_count
count_done --> verify_count
verify_count --> vcount_stats[("stats/verify_count/")]
vcount_stats --> aggregate_verify_count
gcount --> filter_count
BIN --> filter_count
filter_count --> spec_count[("specific_index_count/")]
filter_count --> spec_count_stats[("stats/specific_kmer_count/")]
spec_count_stats --> aggregate_filter_count
refnpz --> verify_merge_count
gcount --> verify_merge_count
verify_merge_count --> vmc[("stats/verify_merge_count/")]
end
subgraph query ["Query track (2 specimens: E. coli + archaeon)"]
GENOMES --> simulate_query
simulate_query --> qdata[("query_data/")]
gpres --> pack_sparse
BIN --> pack_sparse
pack_sparse --> gsparse[("global_index_presence_sparse/")]
qdata --> query_dense
gpres --> query_dense
BIN --> query_dense
query_dense --> qd[("query_dense/")]
query_dense --> qd_stats[("stats/query_dense/")]
qd_stats --> aggregate_query_dense
qdata --> query_sparse
gsparse --> query_sparse
BIN --> query_sparse
query_sparse --> qs[("query_sparse/")]
query_sparse --> qs_stats[("stats/query_sparse/")]
qs_stats --> aggregate_query_sparse
qd --> verify_query
qs --> verify_query
verify_query --> vq_stats[("stats/verify_query/")]
vq_stats --> aggregate_verify_query
end
aggregate_verify_presence --> all
aggregate_verify_count --> all
vmp --> all
vmc --> all
aggregate_query_dense --> all
aggregate_query_sparse --> all
aggregate_verify_query --> all
all -. "$(MAKE) re-eval" .-> aggregate_filter_presence
all -. "$(MAKE) re-eval" .-> aggregate_filter_count
Steps
| Target | Script | Description |
|---|---|---|
simulate |
simulate.sh |
Simulate sequencing reads from the reference genomes |
reference |
build_reference.sh |
Build reference kmer sets (.npz) from simulation truth |
index_presence |
index_one_presence.sh |
Index each specimen (presence mode) |
index_count |
index_one_count.sh |
Index each specimen (count mode) |
aggregate_index_presence |
aggregate_stats.sh |
Aggregate per-specimen indexing stats (presence) |
aggregate_index_count |
aggregate_stats.sh |
Aggregate per-specimen indexing stats (count) |
merge_presence |
merge_presence.sh |
Merge all specimen presence indexes into a global index |
merge_count |
merge_count.sh |
Merge all specimen count indexes into a global index |
verify_presence |
verify_one_presence.sh |
Verify each specimen presence index against reference |
verify_count |
verify_one_count.sh |
Verify each specimen count index against reference |
aggregate_verify_presence |
aggregate_stats.sh |
Aggregate per-specimen verification stats (presence) |
aggregate_verify_count |
aggregate_stats.sh |
Aggregate per-specimen verification stats (count) |
filter_presence |
filter_one_presence.sh |
Extract species-specific presence indexes from global index |
filter_count |
filter_one_count.sh |
Extract species-specific count indexes from global index |
aggregate_filter_presence |
aggregate_stats.sh |
Aggregate species-specific kmer stats (presence) |
aggregate_filter_count |
aggregate_stats.sh |
Aggregate species-specific kmer stats (count) |
verify_merge_presence |
verify_merge_presence.sh |
Verify global presence index against all reference sets |
verify_merge_count |
verify_merge_count.sh |
Verify global count index against all reference sets |
simulate_query |
simulate_query_one.sh |
Simulate a fixed-size (100k pairs) read set per query specimen |
pack_sparse |
pack_sparse.sh |
Build global_index_presence_sparse/ from global_index_presence/ |
query_dense |
query_one.sh dense |
Query each query specimen's reads against the dense global index |
query_sparse |
query_one.sh sparse |
Query each query specimen's reads against the sparse global index |
aggregate_query_dense |
aggregate_stats.sh |
Aggregate dense query wall/RSS stats |
aggregate_query_sparse |
aggregate_stats.sh |
Aggregate sparse query wall/RSS stats |
verify_query |
verify_query_one.sh |
Diff dense vs sparse query output per specimen (regression check) |
aggregate_verify_query |
aggregate_stats.sh |
Aggregate dense/sparse query regression stats |
Directory layout
benchmark/
├── genomes/ # input reference genomes (.fna.gz)
├── simulated_data/ # generated by simulate
│ └── <species>/<specimen>/
├── query_data/ # generated by simulate_query (2 specimens, fixed 100k pairs)
│ └── <species>/<specimen>/
├── reference_index/ # reference kmer sets (.npz)
├── specimen_index_presence/ # per-specimen presence indexes
├── specimen_index_count/ # per-specimen count indexes
├── global_index_presence/ # merged global presence index (dense-packed)
├── global_index_presence_sparse/ # global presence index, sparse-packed (query benchmark)
├── global_index_count/ # merged global count index
├── specific_index_presence/ # species-specific presence indexes
├── specific_index_count/ # species-specific count indexes
├── query_dense/ # query output against global_index_presence
├── query_sparse/ # query output against global_index_presence_sparse
└── stats/ # all benchmark statistics
├── indexing_presence/
├── indexing_count/
├── verify_presence/
├── verify_count/
├── specific_kmer_presence/
├── specific_kmer_count/
├── verify_merge_presence/
├── verify_merge_count/
├── pack_sparse/
├── query_dense/
├── query_sparse/
└── verify_query/