obikmer/DevDocMD/implementation/benchmark_query_testing.md

Benchmark: query-path testing (discussion)

benchmark/Makefile exercises indexing, merge, and phylo distance reconstruction against simulated bacterial genomes, but has no coverage of obikmer query — the read-matching path — nor of the sparse packed presence-matrix format (obikmer pack --sparse). This note captures the planned extension.

Motivation

• query is untested end-to-end. A regression there would not be caught by the existing verify_presence/verify_merge_presence branches, which only check index content against the .npz truth, never the query API.
• pack --sparse produces a presence-matrix format documented (see siblings.md) as faster for single-row access (query) and slower for column-oriented access (phylo --metric). global_index_presence/ built by merge_presence.sh is currently always packed dense (packing is a stage inside merge, not a separate pack invocation). There is no dense/sparse regression check.

Plan

New read source, independent of simulated_data/. Reusing simulated_data/<species>/<strain>/reads_R1.fastq.gz for queries would bias the test: those reads were already folded into the index being queried, with the same sequencing-error draw. Query reads must come from a second, independent iss generate run against the same reference genome(s) — new random error draw, same underlying sequence — landing in a separate tree: query_data/<species>/<strain>/reads_R1.fastq.gz, built by the existing simulate_one.sh (unseeded, so a second invocation naturally draws different reads).

Two specimens chosen as query sources (enough to catch a dense/sparse regression without duplicating the exhaustive per-specimen coverage verify_merge_presence already provides across all SPECIMENS): Escherichia_coli--K-12_MG1655 (common, well-represented bacterium) and Saccharolobus_islandicus--M.16.4 (the only archaeon in SPECIES — distant lineage, different GC content, stresses the query path differently from a close-relative match).

make_deps.py needs a QUERY_SPECIMENS list (explicit, short) and, for each, an extra dependency line:

query_data/<species>/<strain>/reads_R1.fastq.gz: genomes/<genome>.fna.gz

distinct from the simulated_data/... rule for the same specimen.

Read count fixed at 100,000 read pairs per genome, independent of genome size — unlike simulate_one.sh's simulated_data/ runs, which derive n_reads from a fixed 15x coverage target. A query benchmark does not need coverage-proportional depth; a fixed pair count keeps the two query runs comparable to each other and keeps wall/RSS numbers meaningful across genomes of very different sizes (bacterium vs archaeon). This likely needs a dedicated simulate_query_one.sh (or a parameter to simulate_one.sh) rather than reusing it unchanged, since n_reads is currently computed in-script from genome size.

Phase 1 — sparse global index. New target global_index_presence_sparse/index.done, built from global_index_presence/ via obikmer pack --sparse. Open question, to verify against the pack implementation before writing the rule: does pack --sparse accept an already dense-packed index in place (cp -r + repack), or does it require the pre-pack column layout, forcing a dedicated merge run instead of reusing global_index_presence/?

Phase 2 — query runs. For each of the two QUERY_SPECIMENS, run obikmer query against both global_index_presence and global_index_presence_sparse, capturing Reporter wall/RSS stats the same way merge_presence.sh does (stderr capture + parse_reporter).

Phase 3 — dense/sparse regression. verify_query.py diffs the two query JSON outputs per specimen (same matches, same per-genome presence annotations) → .stats CSV (run,specimen,mismatches,pct), aggregated by aggregate_stats.sh under a new query case. Any mismatch is a real regression — dense and sparse must be content-identical, only I/O access pattern differs.

Phase 4 — performance comparison. No dedicated script: the wall/RSS columns from Phase 2's .stats files, aggregated, are the dense-vs-sparse performance comparison (the expected win for query on sparse, per the pack --sparse help text).

count track is out of scope for the sparse branch: pack --sparse targets presence matrices only (per CLI help), no count equivalent confirmed.