.gitignore

@@ -10,13 +10,23 @@ src/target
 data-stress
 *.fasta
 *.fasta.gz
+*.fastq
+*.fastq.gz
+*.vcf
 *.zst
 *.zst.meta
 *.pb
-./**/*.json
+**/*.json
 *.bin
 *.log
 *.csv
+*.meta
+*.pfiv
+*.done
+*.efh
+*.efl
+*.bin.idx
+*.prsb
 Betula_exilis--IGA-24-33
 benchmark/genomes
 benchmark/genomes_orig
@@ -34,6 +44,7 @@ benchmark/reference_dist
 benchmark/obikmer_dist
 benchmark/specific_index_count
 benchmark/specific_index_presence
+benchmark/query_data
 TNT
 phyg
 biblio

DevDocMD/implementation/benchmark_query_testing.md

@@ -1,84 +1,93 @@
-# Benchmark: query-path testing (discussion)
+# Benchmark: query-path testing
 
 `benchmark/Makefile` exercises indexing, merge, and phylo distance
-reconstruction against simulated bacterial genomes, but has no coverage of
+reconstruction against simulated bacterial genomes. It now also covers
-`obikmer query` — the read-matching path — nor of the sparse packed
+`obikmer query` — the read-matching path — and the sparse packed
-presence-matrix format (`obikmer pack --sparse`). This note captures the
+presence-matrix format (`obikmer pack --sparse`), previously untested by
-planned extension.
+this pipeline.
 
 ## Motivation
 
-- `query` is untested end-to-end. A regression there would not be caught by
+- `query` had no end-to-end coverage. A regression there would not be caught
-  the existing `verify_presence`/`verify_merge_presence` branches, which only
+  by `verify_presence`/`verify_merge_presence`, which only check index
-  check index *content* against the `.npz` truth, never the query API.
+  *content* against the `.npz` truth, never the query API.
 - `pack --sparse` produces a presence-matrix format documented (see
   [siblings.md](../architecture/siblings.md)) as faster for single-row
   access (query) and slower for column-oriented access (phylo `--metric`).
-  `global_index_presence/` built by `merge_presence.sh` is currently always
+  `global_index_presence/` built by `merge_presence.sh` is always packed
-  packed dense (packing is a stage inside `merge`, not a separate `pack`
+  dense (packing is a stage inside `merge`, not a separate `pack`
-  invocation). There is no dense/sparse regression check.
+  invocation) — there was no dense/sparse regression check.
 
-## Plan
+## Query read source
 
-**New read source, independent of `simulated_data/`.** Reusing
+Query reads are independent of `simulated_data/` (which is folded into the
-`simulated_data/<species>/<strain>/reads_R1.fastq.gz` for queries would bias
+index being queried): reusing those reads would test against the exact
-the test: those reads were already folded into the index being queried, with
+error draw the index was built from. `query_data/<species>/<strain>/` holds
-the same sequencing-error draw. Query reads must come from a *second*,
+a *second*, independent `iss generate` run against the same reference
-independent `iss generate` run against the same reference genome(s) — new
+genome, via `simulate_query_one.sh` — unseeded, so a second draw picks up
-random error draw, same underlying sequence — landing in a separate tree:
+different sequencing errors than `simulate_one.sh`'s draw for the same
-`query_data/<species>/<strain>/reads_R1.fastq.gz`, built by the existing
+genome. Fixed at 100,000 read pairs per genome (not coverage-proportional
-`simulate_one.sh` (unseeded, so a second invocation naturally draws different
+like the 15x used for `simulated_data/`), so wall/RSS numbers stay
-reads).
+comparable across genomes of very different sizes.
 
-Two specimens chosen as query sources (enough to catch a dense/sparse
+Two query-source specimens, hardcoded as `QUERY_SPECIMENS` in
-regression without duplicating the exhaustive per-specimen coverage
+`make_deps.py`: `Escherichia_coli--K-12_MG1655` (common, well-represented
-`verify_merge_presence` already provides across all `SPECIMENS`):
+bacterium) and `Saccharolobus_islandicus--M.16.4` (the only archaeon in
-`Escherichia_coli--K-12_MG1655` (common, well-represented bacterium) and
+`SPECIES` — distant lineage, stresses the query path differently from a
-`Saccharolobus_islandicus--M.16.4` (the only archaeon in `SPECIES` — distant
+close-relative match). Two is enough to catch a dense/sparse regression
-lineage, different GC content, stresses the query path differently from a
+without duplicating the exhaustive per-specimen coverage
-close-relative match).
+`verify_merge_presence` already provides across all `SPECIMENS`.
 
-`make_deps.py` needs a `QUERY_SPECIMENS` list (explicit, short) and, for each,
+## Sparse global index
-an extra dependency line:
-```
-query_data/<species>/<strain>/reads_R1.fastq.gz: genomes/<genome>.fna.gz
-```
-distinct from the `simulated_data/...` rule for the same specimen.
 
-Read count fixed at 100,000 read pairs per genome, independent of genome
+`global_index_presence_sparse/` is built by `pack_sparse.sh`: copy
-size — unlike `simulate_one.sh`'s `simulated_data/` runs, which derive
+`global_index_presence/` wholesale, then `obikmer pack --sparse` in place.
-`n_reads` from a fixed 15x coverage target. A query benchmark does not need
+This works directly because `merge`'s pack stage (`merge.rs:252`,
-coverage-proportional depth; a fixed pair count keeps the two query runs
+`pack_matrices(false)`) keeps the per-genome column files on disk after
-comparable to each other and keeps wall/RSS numbers meaningful across
+dense-packing — `pack_sparse_bit_matrix` (`obicompactvec/src/bitmatrix/sparse.rs:447`)
-genomes of very different sizes (bacterium vs archaeon). This likely needs a
+reads those, is idempotent, and removes `matrix.pbmx` once the sparse form
-dedicated `simulate_query_one.sh` (or a parameter to `simulate_one.sh`)
+is written, so `Persistent::open` falls through to the sparse format
-rather than reusing it unchanged, since `n_reads` is currently computed
+afterward. No separate merge run needed.
-in-script from genome size.
 
-**Phase 1 — sparse global index.** New target
+## Query runs
-`global_index_presence_sparse/index.done`, built from `global_index_presence/`
-via `obikmer pack --sparse`. Open question, to verify against the `pack`
-implementation before writing the rule: does `pack --sparse` accept an
-already dense-packed index in place (`cp -r` + repack), or does it require
-the pre-pack column layout, forcing a dedicated merge run instead of reusing
-`global_index_presence/`?
 
-**Phase 2 — query runs.** For each of the two `QUERY_SPECIMENS`, run
+`query_one.sh dense|sparse SPECIMEN` runs `obikmer query --count-missing`
-`obikmer query` against both `global_index_presence` and
+against `global_index_presence` or `global_index_presence_sparse`, output
-`global_index_presence_sparse`, capturing Reporter wall/RSS stats the same
+gzipped to `query_{dense,sparse}/SPECIMEN.fasta.gz`, Reporter wall/RSS
-way `merge_presence.sh` does (stderr capture + `parse_reporter`).
+captured to `stats/query_{dense,sparse}/SPECIMEN.stats` (same
+stderr-parsing convention as `merge_presence.sh`).
 
-**Phase 3 — dense/sparse regression.** `verify_query.py` diffs the two query
+Flags: `--count-missing` only. `--mismatch` is a no-op today
-JSON outputs per specimen (same matches, same per-genome presence
+(`query/mod.rs:212-213`, prints "not yet implemented, ignored") — left off
-annotations) → `.stats` CSV (`run,specimen,mismatches,pct`), aggregated by
+rather than tested for a feature that doesn't exist yet.
-`aggregate_stats.sh` under a new `query` case. Any mismatch is a real
-regression — dense and sparse must be content-identical, only I/O access
-pattern differs.
 
-**Phase 4 — performance comparison.** No dedicated script: the wall/RSS
+## Dense/sparse regression
-columns from Phase 2's `.stats` files, aggregated, are the dense-vs-sparse
-performance comparison (the expected win for query on sparse, per the `pack
---sparse` help text).
 
-`count` track is out of scope for the sparse branch: `pack --sparse` targets
+`verify_query.py` compares the two query outputs per specimen, matched by
-presence matrices only (per CLI help), no count equivalent confirmed.
+read id (not stream position — the query pipeline chunks input across
+worker threads and doesn't guarantee output order). Compares `kmer_count`,
+`kmer_missing`, and the full `kmer_strict_matches` map per read. Any
+mismatch is a real regression: dense and sparse must be content-identical,
+only I/O access pattern differs. `.stats` → `stats/verify_query/`,
+aggregated by `aggregate_stats.sh query|verify_query`-style cases
+(`query_dense`, `query_sparse`, `verify_query`).
+
+## Performance comparison
+
+No dedicated script: the wall/RSS columns from the `query_dense` and
+`query_sparse` aggregated `.stats` CSVs are the dense-vs-sparse performance
+comparison — the expected win for query on sparse, per the `pack --sparse`
+help text.
+
+## Scope
+
+`count` track excluded from the sparse branch: `pack --sparse` targets
+presence matrices only (per CLI help); `pack_matrices` leaves count
+matrices untouched regardless of the `sparse` flag
+(`obikindex/src/index.rs:308`).
+
+## New Makefile targets
+
+`simulate_query`, `pack_sparse`, `query_dense`, `query_sparse`,
+`aggregate_query_dense`, `aggregate_query_sparse`, `verify_query`,
+`aggregate_verify_query` — the last three folded into `all`.

benchmark/Makefile

@@ -38,6 +38,12 @@ SPECIFIC_PRESENCE_STATS := $(SPECIES:%=stats/specific_kmer_presence/%.stats)
 SPECIFIC_COUNT_DONE     := $(SPECIES:%=specific_index_count/%/index.done)
 SPECIFIC_COUNT_STATS    := $(SPECIES:%=stats/specific_kmer_count/%.stats)
 SIMULATED_READS := $(foreach s,$(SPECIMENS),simulated_data/$(subst --,/,$s)/reads_R1.fastq.gz)
+QUERY_READS      := $(foreach s,$(QUERY_SPECIMENS),query_data/$(subst --,/,$s)/reads_R1.fastq.gz)
+QUERY_DENSE_DONE  := $(QUERY_SPECIMENS:%=query_dense/%.fasta.gz)
+QUERY_DENSE_STATS := $(QUERY_SPECIMENS:%=stats/query_dense/%.stats)
+QUERY_SPARSE_DONE  := $(QUERY_SPECIMENS:%=query_sparse/%.fasta.gz)
+QUERY_SPARSE_STATS := $(QUERY_SPECIMENS:%=stats/query_sparse/%.stats)
+VERIFY_QUERY_STATS := $(QUERY_SPECIMENS:%=stats/verify_query/%.stats)
 
 .NOTPARALLEL:
 
@@ -51,7 +57,11 @@ SIMULATED_READS := $(foreach s,$(SPECIMENS),simulated_data/$(subst --,/,$s)/read
         aggregate_verify_presence aggregate_verify_count \
         verify_merge_presence verify_merge_count \
         filter_presence filter_count \
-        aggregate_filter_presence aggregate_filter_count
+        aggregate_filter_presence aggregate_filter_count \
+        pack_sparse simulate_query \
+        query_dense query_sparse \
+        aggregate_query_dense aggregate_query_sparse \
+        verify_query aggregate_verify_query
 
 verify_merge_presence: stats/verify_merge_presence/current.csv
 verify_merge_count:    stats/verify_merge_count/current.csv
@@ -59,12 +69,13 @@ verify_merge_count:    stats/verify_merge_count/current.csv
 all: aggregate_verify_presence aggregate_verify_count \
      verify_merge_presence verify_merge_count \
      aggregate_filter_presence aggregate_filter_count \
-     dist_comparison
+     dist_comparison \
+     aggregate_query_dense aggregate_query_sparse aggregate_verify_query
 
 # ── dependency file ───────────────────────────────────────────────────────────
 
-deps.mk: $(GENOMES)
+deps.mk: $(GENOMES) make_deps.py
-	$(VENV_PY) make_deps.py $^ > $@
+	$(VENV_PY) make_deps.py $(GENOMES) > $@
 
 # ── simulation ────────────────────────────────────────────────────────────────
 # Prerequisites (genome → reads) are in deps.mk; $< is the genome file.
@@ -74,6 +85,14 @@ $(SIMULATED_READS):
 
 simulate: $(SIMULATED_READS)
 
+# ── query read simulation (fixed size, independent draw) ───────────────────────
+# Prerequisites (genome → reads) are in deps.mk; $< is the genome file.
+
+$(QUERY_READS):
+	bash simulate_query_one.sh $< $(dir $@)
+
+simulate_query: $(QUERY_READS)
+
 # ── reference kmer sets ───────────────────────────────────────────────────────
 # Prerequisites (reads → npz) are in deps.mk.
 
@@ -228,3 +247,40 @@ stats/verify_merge_presence/current.csv: $(REF_NPZS) global_index_presence/index
 
 stats/verify_merge_count/current.csv: $(REF_NPZS) global_index_count/index.done
 	bash verify_merge_count.sh
+
+# ── sparse presence index (query benchmark) ─────────────────────────────────────
+
+global_index_presence_sparse/index.done: global_index_presence/index.done $(BINARY)
+	bash pack_sparse.sh
+
+pack_sparse: global_index_presence_sparse/index.done
+
+# ── query: dense vs sparse ───────────────────────────────────────────────────────
+# Prerequisites (reads + index → output + .stats) are in deps.mk.
+
+query_dense/%.fasta.gz \
+stats/query_dense/%.stats &: $(BINARY)
+	bash query_one.sh dense $*
+
+query_sparse/%.fasta.gz \
+stats/query_sparse/%.stats &: $(BINARY)
+	bash query_one.sh sparse $*
+
+query_dense:  $(QUERY_DENSE_DONE)
+query_sparse: $(QUERY_SPARSE_DONE)
+
+aggregate_query_dense: $(QUERY_DENSE_STATS)
+	bash aggregate_stats.sh query_dense
+
+aggregate_query_sparse: $(QUERY_SPARSE_STATS)
+	bash aggregate_stats.sh query_sparse
+
+# ── query: dense/sparse regression ──────────────────────────────────────────────
+
+stats/verify_query/%.stats:
+	bash verify_query_one.sh $*
+
+verify_query: $(VERIFY_QUERY_STATS)
+
+aggregate_verify_query: $(VERIFY_QUERY_STATS)
+	bash aggregate_stats.sh verify_query

benchmark/README.md

@@ -75,10 +75,41 @@ flowchart TD
         verify_merge_count --> vmc[("stats/verify_merge_count/")]
     end
 
+    subgraph query ["Query track (2 specimens: E. coli + archaeon)"]
+        GENOMES --> simulate_query
+        simulate_query --> qdata[("query_data/")]
+
+        gpres --> pack_sparse
+        BIN   --> pack_sparse
+        pack_sparse --> gsparse[("global_index_presence_sparse/")]
+
+        qdata  --> query_dense
+        gpres  --> query_dense
+        BIN    --> query_dense
+        query_dense --> qd[("query_dense/")]
+        query_dense --> qd_stats[("stats/query_dense/")]
+        qd_stats --> aggregate_query_dense
+
+        qdata   --> query_sparse
+        gsparse --> query_sparse
+        BIN     --> query_sparse
+        query_sparse --> qs[("query_sparse/")]
+        query_sparse --> qs_stats[("stats/query_sparse/")]
+        qs_stats --> aggregate_query_sparse
+
+        qd --> verify_query
+        qs --> verify_query
+        verify_query --> vq_stats[("stats/verify_query/")]
+        vq_stats --> aggregate_verify_query
+    end
+
     aggregate_verify_presence  --> all
     aggregate_verify_count     --> all
     vmp                        --> all
     vmc                        --> all
+    aggregate_query_dense      --> all
+    aggregate_query_sparse     --> all
+    aggregate_verify_query     --> all
     all -. "$(MAKE) re-eval" .-> aggregate_filter_presence
     all -. "$(MAKE) re-eval" .-> aggregate_filter_count
 ```
@@ -105,6 +136,14 @@ flowchart TD
 | `aggregate_filter_count` | `aggregate_stats.sh` | Aggregate species-specific kmer stats (count) |
 | `verify_merge_presence` | `verify_merge_presence.sh` | Verify global presence index against all reference sets |
 | `verify_merge_count` | `verify_merge_count.sh` | Verify global count index against all reference sets |
+| `simulate_query` | `simulate_query_one.sh` | Simulate a fixed-size (100k pairs) read set per query specimen |
+| `pack_sparse` | `pack_sparse.sh` | Build `global_index_presence_sparse/` from `global_index_presence/` |
+| `query_dense` | `query_one.sh dense` | Query each query specimen's reads against the dense global index |
+| `query_sparse` | `query_one.sh sparse` | Query each query specimen's reads against the sparse global index |
+| `aggregate_query_dense` | `aggregate_stats.sh` | Aggregate dense query wall/RSS stats |
+| `aggregate_query_sparse` | `aggregate_stats.sh` | Aggregate sparse query wall/RSS stats |
+| `verify_query` | `verify_query_one.sh` | Diff dense vs sparse query output per specimen (regression check) |
+| `aggregate_verify_query` | `aggregate_stats.sh` | Aggregate dense/sparse query regression stats |
 
 ## Directory layout
 
@@ -113,13 +152,18 @@ benchmark/
 ├── genomes/                        # input reference genomes (.fna.gz)
 ├── simulated_data/                 # generated by simulate
 │   └── <species>/<specimen>/
+├── query_data/                     # generated by simulate_query (2 specimens, fixed 100k pairs)
+│   └── <species>/<specimen>/
 ├── reference_index/                # reference kmer sets (.npz)
 ├── specimen_index_presence/        # per-specimen presence indexes
 ├── specimen_index_count/           # per-specimen count indexes
-├── global_index_presence/          # merged global presence index
+├── global_index_presence/          # merged global presence index (dense-packed)
+├── global_index_presence_sparse/   # global presence index, sparse-packed (query benchmark)
 ├── global_index_count/             # merged global count index
 ├── specific_index_presence/        # species-specific presence indexes
 ├── specific_index_count/           # species-specific count indexes
+├── query_dense/                    # query output against global_index_presence
+├── query_sparse/                   # query output against global_index_presence_sparse
 └── stats/                          # all benchmark statistics
     ├── indexing_presence/
     ├── indexing_count/
@@ -128,5 +172,9 @@ benchmark/
     ├── specific_kmer_presence/
     ├── specific_kmer_count/
     ├── verify_merge_presence/
-    └── verify_merge_count/
+    ├── verify_merge_count/
+    ├── pack_sparse/
+    ├── query_dense/
+    ├── query_sparse/
+    └── verify_query/
 ```

benchmark/aggregate_stats.sh

@@ -24,6 +24,12 @@ case "${TYPE}" in
     specific_kmer_presence|specific_kmer_count)
         HEADER="run,species,rebuild_wall_s,rebuild_rss_b,pack_wall_s,pack_rss_b,filter_total_wall_s,filter_total_rss_b,select_wall_s,select_rss_b,select_total_wall_s,select_total_rss_b"
         ;;
+    query_dense|query_sparse)
+        HEADER="run,species,strain,query_wall_s,query_rss_b,total_wall_s,total_rss_b"
+        ;;
+    verify_query)
+        HEADER="run,species,strain,n_reads,n_common,missing_in_dense,missing_in_sparse,mismatched,mismatch_pct"
+        ;;
     *)
         echo "ERROR: unknown stats type '${TYPE}'" >&2
         exit 1

benchmark/deps.mk

@@ -197,3 +197,17 @@ specific_index_count/Yersinia_ruckeri/index.done stats/specific_kmer_count/Yersi
 # Candidozyma_auris
 specific_index_presence/Candidozyma_auris/index.done stats/specific_kmer_presence/Candidozyma_auris.stats: global_index_presence/index.done
 specific_index_count/Candidozyma_auris/index.done stats/specific_kmer_count/Candidozyma_auris.stats: global_index_count/index.done
+
+QUERY_SPECIMENS := Escherichia_coli--K-12_MG1655 Saccharolobus_islandicus--M.16.4
+
+# query: Escherichia_coli--K-12_MG1655
+query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz: genomes/GCF_000005845.2_ASM584v2_genomic.fna.gz
+query_dense/Escherichia_coli--K-12_MG1655.fasta.gz stats/query_dense/Escherichia_coli--K-12_MG1655.stats: query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz global_index_presence/index.done
+query_sparse/Escherichia_coli--K-12_MG1655.fasta.gz stats/query_sparse/Escherichia_coli--K-12_MG1655.stats: query_data/Escherichia_coli/K-12_MG1655/reads_R1.fastq.gz global_index_presence_sparse/index.done
+stats/verify_query/Escherichia_coli--K-12_MG1655.stats: query_dense/Escherichia_coli--K-12_MG1655.fasta.gz query_sparse/Escherichia_coli--K-12_MG1655.fasta.gz
+
+# query: Saccharolobus_islandicus--M.16.4
+query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz: genomes/GCF_000022445.1_ASM2244v1_genomic.fna.gz
+query_dense/Saccharolobus_islandicus--M.16.4.fasta.gz stats/query_dense/Saccharolobus_islandicus--M.16.4.stats: query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz global_index_presence/index.done
+query_sparse/Saccharolobus_islandicus--M.16.4.fasta.gz stats/query_sparse/Saccharolobus_islandicus--M.16.4.stats: query_data/Saccharolobus_islandicus/M.16.4/reads_R1.fastq.gz global_index_presence_sparse/index.done
+stats/verify_query/Saccharolobus_islandicus--M.16.4.stats: query_dense/Saccharolobus_islandicus--M.16.4.fasta.gz query_sparse/Saccharolobus_islandicus--M.16.4.fasta.gz

benchmark/make_deps.py

@@ -13,6 +13,11 @@ STOP_WORDS    = {'complete', 'chromosome', 'whole', 'sequence', 'genome',
                  'endosymbiont', 'of'}
 STOP_PREFIXES = ('scaffold', 'contig', 'plasmid')
 
+# Specimens used as read sources for the query benchmark (see
+# DevDocMD/implementation/benchmark_query_testing.md): one common bacterium,
+# one distant lineage (the only archaeon in SPECIES).
+QUERY_SPECIMENS = ['Escherichia_coli--K-12_MG1655', 'Saccharolobus_islandicus--M.16.4']
+
 
 def is_stop(tok):
     t = tok.lower()
@@ -113,6 +118,27 @@ def main():
         print(f'{sp_done} {sp_stats}: global_index_presence/index.done')
         print(f'{sc_done} {sc_stats}: global_index_count/index.done')
 
+    print()
+    print('QUERY_SPECIMENS :=', ' '.join(QUERY_SPECIMENS))
+
+    by_specimen = {e[0]: e for e in entries}
+    for specimen in QUERY_SPECIMENS:
+        _, species, sim_dir, genome = by_specimen[specimen]
+        query_dir = sim_dir.replace('simulated_data/', 'query_data/', 1)
+        reads      = f'{query_dir}/reads_R1.fastq.gz'
+        dense_out  = f'query_dense/{specimen}.fasta.gz'
+        dense_stat = f'stats/query_dense/{specimen}.stats'
+        sparse_out  = f'query_sparse/{specimen}.fasta.gz'
+        sparse_stat = f'stats/query_sparse/{specimen}.stats'
+        vq_stat     = f'stats/verify_query/{specimen}.stats'
+
+        print()
+        print(f'# query: {specimen}')
+        print(f'{reads}: {genome}')
+        print(f'{dense_out} {dense_stat}: {reads} global_index_presence/index.done')
+        print(f'{sparse_out} {sparse_stat}: {reads} global_index_presence_sparse/index.done')
+        print(f'{vq_stat}: {dense_out} {sparse_out}')
+
 
 if __name__ == '__main__':
     main()

benchmark/pack_sparse.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Builds global_index_presence_sparse/ from global_index_presence/ by
+# copying the index (column files are kept in place after merge's dense
+# pack — see obikindex::KmerIndex::pack_matrices) and repacking in place
+# with --sparse.
+# Outputs:
+#   global_index_presence_sparse/index.done  (copied from source)
+#   stats/pack_sparse/current.stats          (one CSV data row, no header)
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
+SOURCE="${SCRIPT_DIR}/global_index_presence"
+OUTPUT="${SCRIPT_DIR}/global_index_presence_sparse"
+STATS_DIR="${SCRIPT_DIR}/stats/pack_sparse"
+STATS_FILE="${STATS_DIR}/current.stats"
+
+mkdir -p "${STATS_DIR}"
+
+echo "[pack_sparse] ${SOURCE} → ${OUTPUT}"
+
+rm -rf "${OUTPUT}"
+cp -r "${SOURCE}" "${OUTPUT}"
+
+STDERR_LOG=$(mktemp)
+trap 'rm -f "${STDERR_LOG}"' EXIT
+
+"${BINARY}" pack --sparse "${OUTPUT}" 2>"${STDERR_LOG}"
+
+cat "${STDERR_LOG}" >&2
+
+python3 - "${STDERR_LOG}" <<'PYEOF' >"${STATS_FILE}"
+import sys, re
+
+logfile = sys.argv[1]
+
+def strip_ansi(s):
+    return re.sub(r'\x1b\[[\x30-\x3f]*[\x20-\x2f]*[\x40-\x7e]', '', s)
+
+def parse_wall(s):
+    s = s.strip()
+    if s.endswith('ms'): return float(s[:-2]) / 1000.0
+    if s.endswith('s'):  return float(s[:-1])
+    return 0.0
+
+def parse_rss(s):
+    m = re.match(r'([\d.]+)\s*(GB|MB|KB|B)', s.strip())
+    if not m: return 0
+    return int(float(m.group(1)) * {'GB': 1<<30, 'MB': 1<<20, 'KB': 1024, 'B': 1}[m.group(2)])
+
+def is_sep(s):
+    return bool(s) and not re.search(r'[A-Za-z0-9]', s)
+
+stats = {}
+state = 'scan'
+with open(logfile, errors='replace') as fh:
+    for raw in fh:
+        line = strip_ansi(raw.rstrip('\n'))
+        s    = line.strip()
+        if state == 'scan':
+            if re.search(r'\bstage\b.*\bwall\b', line):
+                state = 'in_header'
+        elif state == 'in_header':
+            if is_sep(s): state = 'rows'
+        elif state == 'rows':
+            if is_sep(s): state = 'total'
+            elif s:
+                parts = re.split(r'  +', s)
+                if len(parts) >= 4:
+                    stats[parts[0]] = (parse_wall(parts[1]), parse_rss(parts[3]))
+        elif state == 'total':
+            if s:
+                parts = re.split(r'  +', s)
+                if len(parts) >= 3:
+                    stats['TOTAL'] = (parse_wall(parts[1]),
+                                      parse_rss(parts[3]) if len(parts) > 3 else 0)
+            break
+
+w, r = stats.get('pack', ('', ''))
+tw, tr = stats.get('TOTAL', ('', ''))
+row = [f'{w:.3f}' if isinstance(w, float) else '', str(r),
+       f'{tw:.3f}' if isinstance(tw, float) else '', str(tr)]
+print(','.join(row))
+PYEOF
+
+echo "Done → ${OUTPUT}"

benchmark/query_one.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Usage: query_one.sh MODE SPECIMEN
+# MODE = dense | sparse
+# SPECIMEN = "species--strain" (Make pattern stem), reads from query_data/
+# Outputs:
+#   query_MODE/SPECIMEN.fasta.gz     (obikmer query output, --count-missing)
+#   stats/query_MODE/SPECIMEN.stats  (one CSV data row, no header)
+set -euo pipefail
+
+MODE="$1"
+SPECIMEN="$2"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+BINARY="${SCRIPT_DIR}/../src/target/release/obikmer"
+
+case "${MODE}" in
+    dense)  INDEX="${SCRIPT_DIR}/global_index_presence" ;;
+    sparse) INDEX="${SCRIPT_DIR}/global_index_presence_sparse" ;;
+    *) echo "ERROR: unknown mode '${MODE}' (expected dense|sparse)" >&2; exit 1 ;;
+esac
+
+species="${SPECIMEN%%--*}"
+strain="${SPECIMEN#*--}"
+
+READS_DIR="${SCRIPT_DIR}/query_data/${species}/${strain}"
+OUT_DIR="${SCRIPT_DIR}/query_${MODE}"
+STATS_DIR="${SCRIPT_DIR}/stats/query_${MODE}"
+OUT_FILE="${OUT_DIR}/${SPECIMEN}.fasta.gz"
+STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
+
+mkdir -p "${OUT_DIR}" "${STATS_DIR}"
+
+r1="${READS_DIR}/reads_R1.fastq.gz"
+r2="${READS_DIR}/reads_R2.fastq.gz"
+if [[ ! -f "${r1}" || ! -f "${r2}" ]]; then
+    echo "ERROR: reads not found in ${READS_DIR}" >&2
+    exit 1
+fi
+
+echo "[${SPECIMEN}] query (${MODE}) → ${OUT_FILE}"
+
+STDERR_LOG=$(mktemp)
+trap 'rm -f "${STDERR_LOG}"' EXIT
+
+"${BINARY}" query \
+    --count-missing \
+    "${INDEX}" "${r1}" "${r2}" \
+    2>"${STDERR_LOG}" \
+    | gzip >"${OUT_FILE}"
+
+cat "${STDERR_LOG}" >&2
+
+python3 - "${species}" "${strain}" "${STDERR_LOG}" <<'PYEOF' >"${STATS_FILE}"
+import sys, re
+
+species, strain, logfile = sys.argv[1], sys.argv[2], sys.argv[3]
+
+def strip_ansi(s):
+    return re.sub(r'\x1b\[[\x30-\x3f]*[\x20-\x2f]*[\x40-\x7e]', '', s)
+
+def parse_wall(s):
+    s = s.strip()
+    if s.endswith('ms'): return float(s[:-2]) / 1000.0
+    if s.endswith('s'):  return float(s[:-1])
+    return 0.0
+
+def parse_rss(s):
+    m = re.match(r'([\d.]+)\s*(GB|MB|KB|B)', s.strip())
+    if not m: return 0
+    return int(float(m.group(1)) * {'GB': 1<<30, 'MB': 1<<20, 'KB': 1024, 'B': 1}[m.group(2)])
+
+def is_sep(s):
+    return bool(s) and not re.search(r'[A-Za-z0-9]', s)
+
+stats = {}
+state = 'scan'
+with open(logfile, errors='replace') as fh:
+    for raw in fh:
+        line = strip_ansi(raw.rstrip('\n'))
+        s    = line.strip()
+        if state == 'scan':
+            if re.search(r'\bstage\b.*\bwall\b', line):
+                state = 'in_header'
+        elif state == 'in_header':
+            if is_sep(s): state = 'rows'
+        elif state == 'rows':
+            if is_sep(s): state = 'total'
+            elif s:
+                parts = re.split(r'  +', s)
+                if len(parts) >= 4:
+                    stats[parts[0]] = (parse_wall(parts[1]), parse_rss(parts[3]))
+        elif state == 'total':
+            if s:
+                parts = re.split(r'  +', s)
+                if len(parts) >= 3:
+                    stats['TOTAL'] = (parse_wall(parts[1]),
+                                      parse_rss(parts[3]) if len(parts) > 3 else 0)
+            break
+
+qw, qr = stats.get('query', ('', ''))
+tw, tr = stats.get('TOTAL', ('', ''))
+row = [species, strain,
+       f'{qw:.3f}' if isinstance(qw, float) else '', str(qr),
+       f'{tw:.3f}' if isinstance(tw, float) else '', str(tr)]
+print(','.join(row))
+PYEOF

benchmark/simulate_query_one.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Usage: simulate_query_one.sh genome.fna.gz output_dir
+# Simulates a fixed-size paired-end HiSeq read set for the query benchmark.
+# Unlike simulate_one.sh (coverage-proportional, used to build the indexed
+# specimens), this always draws N_READS pairs regardless of genome size —
+# query benchmark numbers (wall/RSS) must stay comparable across genomes of
+# very different sizes. Independent iss run (unseeded), so error draw
+# differs from any simulated_data/ reads for the same genome.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ISS="${SCRIPT_DIR}/../.venv/bin/iss"
+N_READS=100000
+CPUS="${CPUS:-$(sysctl -n hw.logicalcpu 2>/dev/null || nproc 2>/dev/null || echo 2)}"
+
+genome_file="$1"
+out_dir="$2"
+
+mkdir -p "${out_dir}"
+
+tmp_fasta=$(mktemp "${TMPDIR:-/tmp}/obikmer_XXXXXX.fna")
+trap 'rm -f "${tmp_fasta}"' EXIT
+
+gzip -dc "${genome_file}" > "${tmp_fasta}"
+
+echo "[${out_dir}]  ${N_READS} read pairs (query benchmark, fixed size)"
+
+"${ISS}" generate \
+    --genomes   "${tmp_fasta}" \
+    --model     HiSeq \
+    --n_reads   "${N_READS}" \
+    --cpus      "${CPUS}" \
+    --compress \
+    --output    "${out_dir}/reads"

benchmark/verify_query.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Compare dense vs sparse obikmer query output for one specimen.
+
+Both files are `obikmer query --count-missing` output: OBITools4-style
+FASTA, one record per read — `>id {"kmer_count":N,"kmer_missing":M,
+"kmer_strict_matches":{"label":count,...}}`. Packing format (dense vs
+sparse presence matrix) must not change query results — only I/O access
+pattern differs. Matched by read id rather than by stream position: the
+query pipeline processes input in chunks across worker threads and does
+not guarantee output order matches input order.
+
+Output to stdout: one CSV row
+  species, strain, n_reads, n_common, missing_in_dense, missing_in_sparse, mismatched, mismatch_pct
+"""
+import argparse
+import gzip
+import json
+import sys
+
+
+def load(path):
+    records = {}
+    opener = gzip.open if path.endswith('.gz') else open
+    with opener(path, 'rt') as fh:
+        for line in fh:
+            if not line.startswith('>'):
+                continue
+            header = line[1:].rstrip('\n')
+            read_id, _, json_part = header.partition(' ')
+            records[read_id] = json.loads(json_part) if json_part else {}
+    return records
+
+
+def annotations_equal(a, b):
+    return (
+        a.get('kmer_count') == b.get('kmer_count')
+        and a.get('kmer_missing') == b.get('kmer_missing')
+        and a.get('kmer_strict_matches', {}) == b.get('kmer_strict_matches', {})
+    )
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument('--species', required=True)
+    ap.add_argument('--strain', required=True)
+    ap.add_argument('dense_fasta')
+    ap.add_argument('sparse_fasta')
+    args = ap.parse_args()
+
+    dense = load(args.dense_fasta)
+    sparse = load(args.sparse_fasta)
+
+    dense_ids, sparse_ids = set(dense), set(sparse)
+    common = dense_ids & sparse_ids
+    missing_in_dense = len(sparse_ids - dense_ids)
+    missing_in_sparse = len(dense_ids - sparse_ids)
+
+    mismatched = sum(1 for rid in common if not annotations_equal(dense[rid], sparse[rid]))
+
+    n_reads = len(dense_ids | sparse_ids)
+    mismatch_pct = 100.0 * (mismatched + missing_in_dense + missing_in_sparse) / n_reads if n_reads else 0.0
+
+    print(f'{args.species},{args.strain},{n_reads},{len(common)},'
+          f'{missing_in_dense},{missing_in_sparse},{mismatched},{mismatch_pct:.6f}')
+
+
+if __name__ == '__main__':
+    main()

benchmark/verify_query_one.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Usage: verify_query_one.sh SPECIMEN
+# SPECIMEN = "species--strain" (Make pattern stem)
+# Output: stats/verify_query/SPECIMEN.stats (one CSV data row, no header)
+set -euo pipefail
+
+SPECIMEN="$1"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PYTHON="${SCRIPT_DIR}/../.venv/bin/python3"
+VERIFY_PY="${SCRIPT_DIR}/verify_query.py"
+
+species="${SPECIMEN%%--*}"
+strain="${SPECIMEN#*--}"
+
+DENSE="${SCRIPT_DIR}/query_dense/${SPECIMEN}.fasta.gz"
+SPARSE="${SCRIPT_DIR}/query_sparse/${SPECIMEN}.fasta.gz"
+STATS_DIR="${SCRIPT_DIR}/stats/verify_query"
+STATS_FILE="${STATS_DIR}/${SPECIMEN}.stats"
+
+mkdir -p "${STATS_DIR}"
+
+echo "[${SPECIMEN}] verifying query (dense vs sparse)"
+
+"${PYTHON}" "${VERIFY_PY}" \
+    --species "${species}" \
+    --strain  "${strain}" \
+    "${DENSE}" "${SPARSE}" \
+    >"${STATS_FILE}"

mkdocs.yml

@@ -55,7 +55,7 @@ nav:
       - Kmer filtering: implementation/filtering.md
       - Select command: implementation/select.md
       - obitaxonomy crate: implementation/obitaxonomy.md
-      - "Benchmark: query-path testing (discussion)": implementation/benchmark_query_testing.md
+      - "Benchmark: query-path testing": implementation/benchmark_query_testing.md
   - Architecture:
       - Sequences: architecture/sequences/invariant.md
       - Kmer index: architecture/index_architecture.md