OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity

Add benchmark pipeline for dense and sparse query testing

Browse Source 
Introduces a complete query benchmark track to evaluate performance and verify consistency between dense and sparse index formats. Adds scripts to simulate fixed-size paired-end reads, pack a sparse presence index, execute queries in both modes, and capture wall time and RSS metrics. Includes a verification step that compares outputs by read ID to ensure content identity across parallel processing. Updates build configuration, documentation, and ignore patterns to support the new pipeline for two microbial specimens.
This commit is contained in:
Eric Coissac
2026-08-20 13:59:12 +02:00
parent 5a9d903e51
commit 89ea077456
14 changed files with 565 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DevDocMD/implementation/benchmark_query_testing.md
+75 -66
View File
@@ -1,84 +1,93 @@
# Benchmark: query-path testing (discussion)
# Benchmark: query-path testing
`benchmark/Makefile` exercises indexing, merge, and phylo distance
reconstruction against simulated bacterial genomes, but has no coverage of
`obikmer query` — the read-matching path — nor of the sparse packed
presence-matrix format (`obikmer pack --sparse`). This note captures the
planned extension.
reconstruction against simulated bacterial genomes. It now also covers
`obikmer query` — the read-matching path — and the sparse packed
presence-matrix format (`obikmer pack --sparse`), previously untested by
this pipeline.
## Motivation
- `query` is untested end-to-end. A regression there would not be caught by
the existing `verify_presence`/`verify_merge_presence` branches, which only
check index *content* against the `.npz` truth, never the query API.
- `query` had no end-to-end coverage. A regression there would not be caught
by `verify_presence`/`verify_merge_presence`, which only check index
*content* against the `.npz` truth, never the query API.
- `pack --sparse` produces a presence-matrix format documented (see
[siblings.md](../architecture/siblings.md)) as faster for single-row
access (query) and slower for column-oriented access (phylo `--metric`).
`global_index_presence/` built by `merge_presence.sh` is currently always
packed dense (packing is a stage inside `merge`, not a separate `pack`
invocation). There is no dense/sparse regression check.
`global_index_presence/` built by `merge_presence.sh` is always packed
dense (packing is a stage inside `merge`, not a separate `pack`
invocation) — there was no dense/sparse regression check.
## Plan
## Query read source
**New read source, independent of `simulated_data/`.** Reusing
`simulated_data/<species>/<strain>/reads_R1.fastq.gz` for queries would bias
the test: those reads were already folded into the index being queried, with
the same sequencing-error draw. Query reads must come from a *second*,
independent `iss generate` run against the same reference genome(s) — new
random error draw, same underlying sequence — landing in a separate tree:
`query_data/<species>/<strain>/reads_R1.fastq.gz`, built by the existing
`simulate_one.sh` (unseeded, so a second invocation naturally draws different
reads).
Query reads are independent of `simulated_data/` (which is folded into the
index being queried): reusing those reads would test against the exact
error draw the index was built from. `query_data/<species>/<strain>/` holds
a *second*, independent `iss generate` run against the same reference
genome, via `simulate_query_one.sh` — unseeded, so a second draw picks up
different sequencing errors than `simulate_one.sh`'s draw for the same
genome. Fixed at 100,000 read pairs per genome (not coverage-proportional
like the 15x used for `simulated_data/`), so wall/RSS numbers stay
comparable across genomes of very different sizes.
Two specimens chosen as query sources (enough to catch a dense/sparse
regression without duplicating the exhaustive per-specimen coverage
`verify_merge_presence` already provides across all `SPECIMENS`):
`Escherichia_coli--K-12_MG1655` (common, well-represented bacterium) and
`Saccharolobus_islandicus--M.16.4` (the only archaeon in `SPECIES` — distant
lineage, different GC content, stresses the query path differently from a
close-relative match).
Two query-source specimens, hardcoded as `QUERY_SPECIMENS` in
`make_deps.py`: `Escherichia_coli--K-12_MG1655` (common, well-represented
bacterium) and `Saccharolobus_islandicus--M.16.4` (the only archaeon in
`SPECIES` — distant lineage, stresses the query path differently from a
close-relative match). Two is enough to catch a dense/sparse regression
without duplicating the exhaustive per-specimen coverage
`verify_merge_presence` already provides across all `SPECIMENS`.
`make_deps.py` needs a `QUERY_SPECIMENS` list (explicit, short) and, for each,
an extra dependency line:
```
query_data/<species>/<strain>/reads_R1.fastq.gz: genomes/<genome>.fna.gz
```
distinct from the `simulated_data/...` rule for the same specimen.
## Sparse global index
Read count fixed at 100,000 read pairs per genome, independent of genome
size — unlike `simulate_one.sh`'s `simulated_data/` runs, which derive
`n_reads` from a fixed 15x coverage target. A query benchmark does not need
coverage-proportional depth; a fixed pair count keeps the two query runs
comparable to each other and keeps wall/RSS numbers meaningful across
genomes of very different sizes (bacterium vs archaeon). This likely needs a
dedicated `simulate_query_one.sh` (or a parameter to `simulate_one.sh`)
rather than reusing it unchanged, since `n_reads` is currently computed
in-script from genome size.
`global_index_presence_sparse/` is built by `pack_sparse.sh`: copy
`global_index_presence/` wholesale, then `obikmer pack --sparse` in place.
This works directly because `merge`'s pack stage (`merge.rs:252`,
`pack_matrices(false)`) keeps the per-genome column files on disk after
dense-packing — `pack_sparse_bit_matrix` (`obicompactvec/src/bitmatrix/sparse.rs:447`)
reads those, is idempotent, and removes `matrix.pbmx` once the sparse form
is written, so `Persistent::open` falls through to the sparse format
afterward. No separate merge run needed.
**Phase 1 — sparse global index.** New target
`global_index_presence_sparse/index.done`, built from `global_index_presence/`
via `obikmer pack --sparse`. Open question, to verify against the `pack`
implementation before writing the rule: does `pack --sparse` accept an
already dense-packed index in place (`cp -r` + repack), or does it require
the pre-pack column layout, forcing a dedicated merge run instead of reusing
`global_index_presence/`?
## Query runs
**Phase 2 — query runs.** For each of the two `QUERY_SPECIMENS`, run
`obikmer query` against both `global_index_presence` and
`global_index_presence_sparse`, capturing Reporter wall/RSS stats the same
way `merge_presence.sh` does (stderr capture + `parse_reporter`).
`query_one.sh dense|sparse SPECIMEN` runs `obikmer query --count-missing`
against `global_index_presence` or `global_index_presence_sparse`, output
gzipped to `query_{dense,sparse}/SPECIMEN.fasta.gz`, Reporter wall/RSS
captured to `stats/query_{dense,sparse}/SPECIMEN.stats` (same
stderr-parsing convention as `merge_presence.sh`).
**Phase 3 — dense/sparse regression.** `verify_query.py` diffs the two query
JSON outputs per specimen (same matches, same per-genome presence
annotations) → `.stats` CSV (`run,specimen,mismatches,pct`), aggregated by
`aggregate_stats.sh` under a new `query` case. Any mismatch is a real
regression — dense and sparse must be content-identical, only I/O access
pattern differs.
Flags: `--count-missing` only. `--mismatch` is a no-op today
(`query/mod.rs:212-213`, prints "not yet implemented, ignored") — left off
rather than tested for a feature that doesn't exist yet.
**Phase 4 — performance comparison.** No dedicated script: the wall/RSS
columns from Phase 2's `.stats` files, aggregated, are the dense-vs-sparse
performance comparison (the expected win for query on sparse, per the `pack
--sparse` help text).
## Dense/sparse regression
`count` track is out of scope for the sparse branch: `pack --sparse` targets
presence matrices only (per CLI help), no count equivalent confirmed.
`verify_query.py` compares the two query outputs per specimen, matched by
read id (not stream position — the query pipeline chunks input across
worker threads and doesn't guarantee output order). Compares `kmer_count`,
`kmer_missing`, and the full `kmer_strict_matches` map per read. Any
mismatch is a real regression: dense and sparse must be content-identical,
only I/O access pattern differs. `.stats` → `stats/verify_query/`,
aggregated by `aggregate_stats.sh query|verify_query`-style cases
(`query_dense`, `query_sparse`, `verify_query`).
## Performance comparison
No dedicated script: the wall/RSS columns from the `query_dense` and
`query_sparse` aggregated `.stats` CSVs are the dense-vs-sparse performance
comparison — the expected win for query on sparse, per the `pack --sparse`
help text.
## Scope
`count` track excluded from the sparse branch: `pack --sparse` targets
presence matrices only (per CLI help); `pack_matrices` leaves count
matrices untouched regardless of the `sparse` flag
(`obikindex/src/index.rs:308`).
## New Makefile targets
`simulate_query`, `pack_sparse`, `query_dense`, `query_sparse`,
`aggregate_query_dense`, `aggregate_query_sparse`, `verify_query`,
`aggregate_verify_query` — the last three folded into `all`.