OBIKmers/obikmer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 34 Wiki Activity
Files
6c860f120fdadf9b15cb07c0554842e2ca8cc8ef
obikmer/DevDoc/implementation/benchmark_query_testing/index.html
T

1582 lines
28 KiB
HTML
Raw Normal View History

add some doc about optimisation for query
2026-08-20 03:32:37 +02:00
<!doctype html>
<html lang="en" class="no-js">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<link rel="prev" href="../obitaxonomy/">
<link rel="next" href="../../architecture/sequences/invariant/">
<link rel="icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
<title>Benchmark: query-path testing - obikmer</title>
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
<style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
</head>
<body dir="ltr">
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
<label class="md-overlay" for="__drawer"></label>
<div data-md-component="skip">
<a href="#benchmark-query-path-testing" class="md-skip">
Skip to content
</a>
</div>
<div data-md-component="announce">
</div>
<header class="md-header md-header--shadow" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header">
<a href="../.." title="obikmer" class="md-header__button md-logo" aria-label="obikmer" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
</a>
<label class="md-header__button md-icon" for="__drawer">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3zm0 5h18v2H3zm0 5h18v2H3z"/></svg>
</label>
<div class="md-header__title" data-md-component="header-title">
<div class="md-header__ellipsis">
<div class="md-header__topic">
<span class="md-ellipsis">
obikmer
</span>
</div>
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</div>
</div>
</div>
<script>var palette=__md_get("__palette");if(palette&&palette.color){if("(prefers-color-scheme)"===palette.color.media){var media=matchMedia("(prefers-color-scheme: light)"),input=document.querySelector(media.matches?"[data-md-color-media='(prefers-color-scheme: light)']":"[data-md-color-media='(prefers-color-scheme: dark)']");palette.color.media=input.getAttribute("data-md-color-media"),palette.color.scheme=input.getAttribute("data-md-color-scheme"),palette.color.primary=input.getAttribute("data-md-color-primary"),palette.color.accent=input.getAttribute("data-md-color-accent")}for(var[key,value]of Object.entries(palette.color))document.body.setAttribute("data-md-color-"+key,value)}</script>
</nav>
</header>
<div class="md-container" data-md-component="container">
<main class="md-main" data-md-component="main">
<div class="md-main__inner md-grid">
<div class="md-sidebar md-sidebar--primary" data-md-component="sidebar" data-md-type="navigation" >
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer">
<a href="../.." title="obikmer" class="md-nav__button md-logo" aria-label="obikmer" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
</a>
obikmer
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../.." class="md-nav__link">
<span class="md-ellipsis">
Home
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../installation/" class="md-nav__link">
<span class="md-ellipsis">
Installation
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3" >
<label class="md-nav__link" for="__nav_3" id="__nav_3_label" tabindex="0">
<span class="md-ellipsis">
Theory
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_3_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3">
<span class="md-nav__icon md-icon"></span>
Theory
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../kmers/" class="md-nav__link">
<span class="md-ellipsis">
Kmers and super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link">
<span class="md-ellipsis">
DNA encoding
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/entropy/" class="md-nav__link">
<span class="md-ellipsis">
Entropy filter
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/minimizer/" class="md-nav__link">
<span class="md-ellipsis">
Minimizer selection
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/indexing/" class="md-nav__link">
<span class="md-ellipsis">
Partitioning architecture
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/evolutionary_distances/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP distance (discussion)
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_4" checked>
<label class="md-nav__link" for="__nav_4" id="__nav_4_label" tabindex="0">
<span class="md-ellipsis">
Implementation
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_4_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_4">
<span class="md-nav__icon md-icon"></span>
Implementation
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../superkmer/" class="md-nav__link">
<span class="md-ellipsis">
SuperKmer
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../kmer/" class="md-nav__link">
<span class="md-ellipsis">
Kmer
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../chunkreader/" class="md-nav__link">
<span class="md-ellipsis">
Chunk reader
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../pipeline/" class="md-nav__link">
<span class="md-ellipsis">
Construction pipeline
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obipipeline/" class="md-nav__link">
<span class="md-ellipsis">
obipipeline library
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../storage/" class="md-nav__link">
<span class="md-ellipsis">
On-disk storage
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../mphf/" class="md-nav__link">
<span class="md-ellipsis">
MPHF selection
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../unitig_evidence/" class="md-nav__link">
<span class="md-ellipsis">
Unitig evidence encoding
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../evidence_elimination/" class="md-nav__link">
<span class="md-ellipsis">
Evidence elimination (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obilayeredmap/" class="md-nav__link">
<span class="md-ellipsis">
obilayeredmap crate
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_compact_int_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentCompactIntVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../persistent_bit_vec/" class="md-nav__link">
<span class="md-ellipsis">
PersistentBitVec
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge/" class="md-nav__link">
<span class="md-ellipsis">
Merge command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../merge_parallelism/" class="md-nav__link">
<span class="md-ellipsis">
Merge parallelism & memory
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../filtering/" class="md-nav__link">
<span class="md-ellipsis">
Kmer filtering
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../select/" class="md-nav__link">
<span class="md-ellipsis">
Select command
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../obitaxonomy/" class="md-nav__link">
<span class="md-ellipsis">
obitaxonomy crate
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--active">
<input class="md-nav__toggle md-toggle" type="checkbox" id="__toc">
<label class="md-nav__link md-nav__link--active" for="__toc">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<a href="./" class="md-nav__link md-nav__link--active">
<span class="md-ellipsis">
Benchmark: query-path testing
</span>
</a>
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#motivation" class="md-nav__link">
<span class="md-ellipsis">
Motivation
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#query-read-source" class="md-nav__link">
<span class="md-ellipsis">
Query read source
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sparse-global-index" class="md-nav__link">
<span class="md-ellipsis">
Sparse global index
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#query-runs" class="md-nav__link">
<span class="md-ellipsis">
Query runs
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#densesparse-regression" class="md-nav__link">
<span class="md-ellipsis">
Dense/sparse regression
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#performance-comparison" class="md-nav__link">
<span class="md-ellipsis">
Performance comparison
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#scope" class="md-nav__link">
<span class="md-ellipsis">
Scope
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#new-makefile-targets" class="md-nav__link">
<span class="md-ellipsis">
New Makefile targets
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_5" >
<label class="md-nav__link" for="__nav_5" id="__nav_5_label" tabindex="0">
<span class="md-ellipsis">
Architecture
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="1" aria-labelledby="__nav_5_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_5">
<span class="md-nav__icon md-icon"></span>
Architecture
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../architecture/sequences/invariant/" class="md-nav__link">
<span class="md-ellipsis">
Sequences
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/index_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Kmer index
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/siblings/" class="md-nav__link">
<span class="md-ellipsis">
Sibling annex (discussion)
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_worker_pools/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware worker pools
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../architecture/numa_partition_runner/" class="md-nav__link">
<span class="md-ellipsis">
NUMA-aware partition runner
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-sidebar md-sidebar--secondary" data-md-component="sidebar" data-md-type="toc" >
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#motivation" class="md-nav__link">
<span class="md-ellipsis">
Motivation
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#query-read-source" class="md-nav__link">
<span class="md-ellipsis">
Query read source
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#sparse-global-index" class="md-nav__link">
<span class="md-ellipsis">
Sparse global index
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#query-runs" class="md-nav__link">
<span class="md-ellipsis">
Query runs
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#densesparse-regression" class="md-nav__link">
<span class="md-ellipsis">
Dense/sparse regression
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#performance-comparison" class="md-nav__link">
<span class="md-ellipsis">
Performance comparison
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#scope" class="md-nav__link">
<span class="md-ellipsis">
Scope
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#new-makefile-targets" class="md-nav__link">
<span class="md-ellipsis">
New Makefile targets
</span>
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-content" data-md-component="content">
<article class="md-content__inner md-typeset">
<h1 id="benchmark-query-path-testing">Benchmark: query-path testing</h1>
<p><code>benchmark/Makefile</code> exercises indexing, merge, and phylo distance
reconstruction against simulated bacterial genomes. It now also covers
<code>obikmer query</code> — the read-matching path — and the sparse packed
presence-matrix format (<code>obikmer pack --sparse</code>), previously untested by
this pipeline.</p>
<h2 id="motivation">Motivation</h2>
<ul>
<li><code>query</code> had no end-to-end coverage. A regression there would not be caught
by <code>verify_presence</code>/<code>verify_merge_presence</code>, which only check index
<em>content</em> against the <code>.npz</code> truth, never the query API.</li>
<li><code>pack --sparse</code> produces a presence-matrix format documented (see
<a href="../../architecture/siblings/">siblings.md</a>) as faster for single-row
access (query) and slower for column-oriented access (phylo <code>--metric</code>).
<code>global_index_presence/</code> built by <code>merge_presence.sh</code> is always packed
dense (packing is a stage inside <code>merge</code>, not a separate <code>pack</code>
invocation) — there was no dense/sparse regression check.</li>
</ul>
<h2 id="query-read-source">Query read source</h2>
<p>Query reads are independent of <code>simulated_data/</code> (which is folded into the
index being queried): reusing those reads would test against the exact
error draw the index was built from. <code>query_data/&lt;species&gt;/&lt;strain&gt;/</code> holds
a <em>second</em>, independent <code>iss generate</code> run against the same reference
genome, via <code>simulate_query_one.sh</code> — unseeded, so a second draw picks up
different sequencing errors than <code>simulate_one.sh</code>'s draw for the same
genome. Fixed at 100,000 read pairs per genome (not coverage-proportional
like the 15x used for <code>simulated_data/</code>), so wall/RSS numbers stay
comparable across genomes of very different sizes.</p>
<p>Two query-source specimens, hardcoded as <code>QUERY_SPECIMENS</code> in
<code>make_deps.py</code>: <code>Escherichia_coli--K-12_MG1655</code> (common, well-represented
bacterium) and <code>Saccharolobus_islandicus--M.16.4</code> (the only archaeon in
<code>SPECIES</code> — distant lineage, stresses the query path differently from a
close-relative match). Two is enough to catch a dense/sparse regression
without duplicating the exhaustive per-specimen coverage
<code>verify_merge_presence</code> already provides across all <code>SPECIMENS</code>.</p>
<h2 id="sparse-global-index">Sparse global index</h2>
<p><code>global_index_presence_sparse/</code> is built by <code>pack_sparse.sh</code>: copy
<code>global_index_presence/</code> wholesale, then <code>obikmer pack --sparse</code> in place.
This works directly because <code>merge</code>'s pack stage (<code>merge.rs:252</code>,
<code>pack_matrices(false)</code>) keeps the per-genome column files on disk after
dense-packing — <code>pack_sparse_bit_matrix</code> (<code>obicompactvec/src/bitmatrix/sparse.rs:447</code>)
reads those, is idempotent, and removes <code>matrix.pbmx</code> once the sparse form
is written, so <code>Persistent::open</code> falls through to the sparse format
afterward. No separate merge run needed.</p>
<h2 id="query-runs">Query runs</h2>
<p><code>query_one.sh dense|sparse SPECIMEN</code> runs <code>obikmer query --count-missing</code>
against <code>global_index_presence</code> or <code>global_index_presence_sparse</code>, output
gzipped to <code>query_{dense,sparse}/SPECIMEN.fasta.gz</code>, Reporter wall/RSS
captured to <code>stats/query_{dense,sparse}/SPECIMEN.stats</code> (same
stderr-parsing convention as <code>merge_presence.sh</code>).</p>
<p>Flags: <code>--count-missing</code> only. <code>--mismatch</code> is a no-op today
(<code>query/mod.rs:212-213</code>, prints "not yet implemented, ignored") — left off
rather than tested for a feature that doesn't exist yet.</p>
<h2 id="densesparse-regression">Dense/sparse regression</h2>
<p><code>verify_query.py</code> compares the two query outputs per specimen, matched by
read id (not stream position — the query pipeline chunks input across
worker threads and doesn't guarantee output order). Compares <code>kmer_count</code>,
<code>kmer_missing</code>, and the full <code>kmer_strict_matches</code> map per read. Any
mismatch is a real regression: dense and sparse must be content-identical,
only I/O access pattern differs. <code>.stats</code> → <code>stats/verify_query/</code>,
aggregated by <code>aggregate_stats.sh query|verify_query</code>-style cases
(<code>query_dense</code>, <code>query_sparse</code>, <code>verify_query</code>).</p>
<h2 id="performance-comparison">Performance comparison</h2>
<p>No dedicated script: the wall/RSS columns from the <code>query_dense</code> and
<code>query_sparse</code> aggregated <code>.stats</code> CSVs are the dense-vs-sparse performance
comparison — the expected win for query on sparse, per the <code>pack --sparse</code>
help text.</p>
<h2 id="scope">Scope</h2>
<p><code>count</code> track excluded from the sparse branch: <code>pack --sparse</code> targets
presence matrices only (per CLI help); <code>pack_matrices</code> leaves count
matrices untouched regardless of the <code>sparse</code> flag
(<code>obikindex/src/index.rs:308</code>).</p>
<h2 id="new-makefile-targets">New Makefile targets</h2>
<p><code>simulate_query</code>, <code>pack_sparse</code>, <code>query_dense</code>, <code>query_sparse</code>,
<code>aggregate_query_dense</code>, <code>aggregate_query_sparse</code>, <code>verify_query</code>,
<code>aggregate_verify_query</code> — the last three folded into <code>all</code>.</p>
</article>
</div>
<script>var target=document.getElementById(location.hash.slice(1));target&&target.name&&(target.checked=target.name.startsWith("__tabbed_"))</script>
</div>
</main>
<footer class="md-footer">
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-copyright">
Made with
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
Material for MkDocs
</a>
</div>
</div>
</div>
</footer>
</div>
<div class="md-dialog" data-md-component="dialog">
<div class="md-dialog__inner md-typeset"></div>
</div>
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script>
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
</body>
</html>
Reference in New Issue Copy Permalink