chore: update dependencies and adapt to updated crate APIs

Bumps core dependencies including ndarray, rand, hashbrown, niffler, ureq, sysinfo, indicatif, lru, and remove_dir_all. Adapts source code to accommodate breaking changes by migrating RNG initialization, adjusting HTTP response handling, and replacing the fs4 crate with standard library file locking. Adds a planning document for query benchmarking and sparse index regression tests.

DevDocMD/implementation/benchmark_query_testing.md

@@ -0,0 +1,84 @@
+# Benchmark: query-path testing (discussion)
+
+`benchmark/Makefile` exercises indexing, merge, and phylo distance
+reconstruction against simulated bacterial genomes, but has no coverage of
+`obikmer query` — the read-matching path — nor of the sparse packed
+presence-matrix format (`obikmer pack --sparse`). This note captures the
+planned extension.
+
+## Motivation
+
+- `query` is untested end-to-end. A regression there would not be caught by
+  the existing `verify_presence`/`verify_merge_presence` branches, which only
+  check index *content* against the `.npz` truth, never the query API.
+- `pack --sparse` produces a presence-matrix format documented (see
+  [siblings.md](../architecture/siblings.md)) as faster for single-row
+  access (query) and slower for column-oriented access (phylo `--metric`).
+  `global_index_presence/` built by `merge_presence.sh` is currently always
+  packed dense (packing is a stage inside `merge`, not a separate `pack`
+  invocation). There is no dense/sparse regression check.
+
+## Plan
+
+**New read source, independent of `simulated_data/`.** Reusing
+`simulated_data/<species>/<strain>/reads_R1.fastq.gz` for queries would bias
+the test: those reads were already folded into the index being queried, with
+the same sequencing-error draw. Query reads must come from a *second*,
+independent `iss generate` run against the same reference genome(s) — new
+random error draw, same underlying sequence — landing in a separate tree:
+`query_data/<species>/<strain>/reads_R1.fastq.gz`, built by the existing
+`simulate_one.sh` (unseeded, so a second invocation naturally draws different
+reads).
+
+Two specimens chosen as query sources (enough to catch a dense/sparse
+regression without duplicating the exhaustive per-specimen coverage
+`verify_merge_presence` already provides across all `SPECIMENS`):
+`Escherichia_coli--K-12_MG1655` (common, well-represented bacterium) and
+`Saccharolobus_islandicus--M.16.4` (the only archaeon in `SPECIES` — distant
+lineage, different GC content, stresses the query path differently from a
+close-relative match).
+
+`make_deps.py` needs a `QUERY_SPECIMENS` list (explicit, short) and, for each,
+an extra dependency line:
+```
+query_data/<species>/<strain>/reads_R1.fastq.gz: genomes/<genome>.fna.gz
+```
+distinct from the `simulated_data/...` rule for the same specimen.
+
+Read count fixed at 100,000 read pairs per genome, independent of genome
+size — unlike `simulate_one.sh`'s `simulated_data/` runs, which derive
+`n_reads` from a fixed 15x coverage target. A query benchmark does not need
+coverage-proportional depth; a fixed pair count keeps the two query runs
+comparable to each other and keeps wall/RSS numbers meaningful across
+genomes of very different sizes (bacterium vs archaeon). This likely needs a
+dedicated `simulate_query_one.sh` (or a parameter to `simulate_one.sh`)
+rather than reusing it unchanged, since `n_reads` is currently computed
+in-script from genome size.
+
+**Phase 1 — sparse global index.** New target
+`global_index_presence_sparse/index.done`, built from `global_index_presence/`
+via `obikmer pack --sparse`. Open question, to verify against the `pack`
+implementation before writing the rule: does `pack --sparse` accept an
+already dense-packed index in place (`cp -r` + repack), or does it require
+the pre-pack column layout, forcing a dedicated merge run instead of reusing
+`global_index_presence/`?
+
+**Phase 2 — query runs.** For each of the two `QUERY_SPECIMENS`, run
+`obikmer query` against both `global_index_presence` and
+`global_index_presence_sparse`, capturing Reporter wall/RSS stats the same
+way `merge_presence.sh` does (stderr capture + `parse_reporter`).
+
+**Phase 3 — dense/sparse regression.** `verify_query.py` diffs the two query
+JSON outputs per specimen (same matches, same per-genome presence
+annotations) → `.stats` CSV (`run,specimen,mismatches,pct`), aggregated by
+`aggregate_stats.sh` under a new `query` case. Any mismatch is a real
+regression — dense and sparse must be content-identical, only I/O access
+pattern differs.
+
+**Phase 4 — performance comparison.** No dedicated script: the wall/RSS
+columns from Phase 2's `.stats` files, aggregated, are the dense-vs-sparse
+performance comparison (the expected win for query on sparse, per the `pack
+--sparse` help text).
+
+`count` track is out of scope for the sparse branch: `pack --sparse` targets
+presence matrices only (per CLI help), no count equivalent confirmed.