chore: update dependencies and adapt to updated crate APIs

Bumps core dependencies including ndarray, rand, hashbrown, niffler, ureq, sysinfo, indicatif, lru, and remove_dir_all. Adapts source code to accommodate breaking changes by migrating RNG initialization, adjusting HTTP response handling, and replacing the fs4 crate with standard library file locking. Adds a planning document for query benchmarking and sparse index regression tests.

DevDocMD/implementation/benchmark_query_testing.md

@@ -0,0 +1,84 @@
+# Benchmark: query-path testing (discussion)
+
+`benchmark/Makefile` exercises indexing, merge, and phylo distance
+reconstruction against simulated bacterial genomes, but has no coverage of
+`obikmer query` — the read-matching path — nor of the sparse packed
+presence-matrix format (`obikmer pack --sparse`). This note captures the
+planned extension.
+
+## Motivation
+
+- `query` is untested end-to-end. A regression there would not be caught by
+  the existing `verify_presence`/`verify_merge_presence` branches, which only
+  check index *content* against the `.npz` truth, never the query API.
+- `pack --sparse` produces a presence-matrix format documented (see
+  [siblings.md](../architecture/siblings.md)) as faster for single-row
+  access (query) and slower for column-oriented access (phylo `--metric`).
+  `global_index_presence/` built by `merge_presence.sh` is currently always
+  packed dense (packing is a stage inside `merge`, not a separate `pack`
+  invocation). There is no dense/sparse regression check.
+
+## Plan
+
+**New read source, independent of `simulated_data/`.** Reusing
+`simulated_data/<species>/<strain>/reads_R1.fastq.gz` for queries would bias
+the test: those reads were already folded into the index being queried, with
+the same sequencing-error draw. Query reads must come from a *second*,
+independent `iss generate` run against the same reference genome(s) — new
+random error draw, same underlying sequence — landing in a separate tree:
+`query_data/<species>/<strain>/reads_R1.fastq.gz`, built by the existing
+`simulate_one.sh` (unseeded, so a second invocation naturally draws different
+reads).
+
+Two specimens chosen as query sources (enough to catch a dense/sparse
+regression without duplicating the exhaustive per-specimen coverage
+`verify_merge_presence` already provides across all `SPECIMENS`):
+`Escherichia_coli--K-12_MG1655` (common, well-represented bacterium) and
+`Saccharolobus_islandicus--M.16.4` (the only archaeon in `SPECIES` — distant
+lineage, different GC content, stresses the query path differently from a
+close-relative match).
+
+`make_deps.py` needs a `QUERY_SPECIMENS` list (explicit, short) and, for each,
+an extra dependency line:
+```
+query_data/<species>/<strain>/reads_R1.fastq.gz: genomes/<genome>.fna.gz
+```
+distinct from the `simulated_data/...` rule for the same specimen.
+
+Read count fixed at 100,000 read pairs per genome, independent of genome
+size — unlike `simulate_one.sh`'s `simulated_data/` runs, which derive
+`n_reads` from a fixed 15x coverage target. A query benchmark does not need
+coverage-proportional depth; a fixed pair count keeps the two query runs
+comparable to each other and keeps wall/RSS numbers meaningful across
+genomes of very different sizes (bacterium vs archaeon). This likely needs a
+dedicated `simulate_query_one.sh` (or a parameter to `simulate_one.sh`)
+rather than reusing it unchanged, since `n_reads` is currently computed
+in-script from genome size.
+
+**Phase 1 — sparse global index.** New target
+`global_index_presence_sparse/index.done`, built from `global_index_presence/`
+via `obikmer pack --sparse`. Open question, to verify against the `pack`
+implementation before writing the rule: does `pack --sparse` accept an
+already dense-packed index in place (`cp -r` + repack), or does it require
+the pre-pack column layout, forcing a dedicated merge run instead of reusing
+`global_index_presence/`?
+
+**Phase 2 — query runs.** For each of the two `QUERY_SPECIMENS`, run
+`obikmer query` against both `global_index_presence` and
+`global_index_presence_sparse`, capturing Reporter wall/RSS stats the same
+way `merge_presence.sh` does (stderr capture + `parse_reporter`).
+
+**Phase 3 — dense/sparse regression.** `verify_query.py` diffs the two query
+JSON outputs per specimen (same matches, same per-genome presence
+annotations) → `.stats` CSV (`run,specimen,mismatches,pct`), aggregated by
+`aggregate_stats.sh` under a new `query` case. Any mismatch is a real
+regression — dense and sparse must be content-identical, only I/O access
+pattern differs.
+
+**Phase 4 — performance comparison.** No dedicated script: the wall/RSS
+columns from Phase 2's `.stats` files, aggregated, are the dense-vs-sparse
+performance comparison (the expected win for query on sparse, per the `pack
+--sparse` help text).
+
+`count` track is out of scope for the sparse branch: `pack --sparse` targets
+presence matrices only (per CLI help), no count equivalent confirmed.

mkdocs.yml

@@ -55,6 +55,7 @@ nav:
       - Kmer filtering: implementation/filtering.md
       - Select command: implementation/select.md
       - obitaxonomy crate: implementation/obitaxonomy.md
+      - "Benchmark: query-path testing (discussion)": implementation/benchmark_query_testing.md
   - Architecture:
       - Sequences: architecture/sequences/invariant.md
       - Kmer index: architecture/index_architecture.md

src/obicompactvec/Cargo.toml

@@ -6,7 +6,7 @@ edition = "2024"
 [dependencies]
 common_traits = "0.11"
 memmap2  = "0.9"
-ndarray  = "0.16"
+ndarray  = "0.17"
 rayon    = "1"
 tempfile = "3"
 

src/obidebruinj/Cargo.toml

@@ -7,7 +7,7 @@ edition = "2021"
 obikseq = { path = "../obikseq" }
 obifastwrite = { path = "../obifastwrite" }
 ahash = "0.8"
-hashbrown = { version = "0.14", features = ["rayon"] }
+hashbrown = { version = "0.17", features = ["rayon"] }
 rayon              = "1"
 crossbeam-channel  = "0.5"
 xxhash-rust = { version = "0.8.15", features = ["xxh3", "const_xxh3"] }

src/obikindex/Cargo.toml

@@ -11,12 +11,12 @@ obiskio          = { path = "../obiskio" }
 obisys           = { path = "../obisys" }
 obicompactvec    = { path = "../obicompactvec" }
 obilayeredmap    = { path = "../obilayeredmap" }
-ndarray          = "0.16"
+ndarray          = "0.17"
 rayon             = "1"
 crossbeam-channel = "0.5"
 serde            = { version = "1", features = ["derive"] }
 serde_json       = "1"
-indicatif        = "0.17"
+indicatif        = "0.18"
 tracing          = "0.1.44"
 hwlocality       = { version = "1.0.0-alpha.11", features = ["vendored"], optional = true }
 

src/obikpartitionner/Cargo.toml

@@ -11,7 +11,7 @@ obikrope  = { path = "../obikrope" }
 
 [dependencies]
 niffler = "3.0.0"
-remove_dir_all = "0.8"
+remove_dir_all = "1.0"
 obikseq      = { path = "../obikseq" }
 obikentropy  = { path = "../obikentropy" }
 obiskbuilder = { path = "../obiskbuilder" }
@@ -19,7 +19,7 @@ obiskio      = { path = "../obiskio" }
 obidebruinj  = { path = "../obidebruinj" }
 obilayeredmap = { path = "../obilayeredmap" }
 rayon = "1"
-sysinfo = "0.33"
+sysinfo = "0.39"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 tracing = "0.1.44"
@@ -28,6 +28,6 @@ epserde = "0.8"
 memmap2 = "0.9.10"
 obicompactvec = { path = "../obicompactvec" }
 ptr_hash = "1.1"
-indicatif = "0.17"
+indicatif = "0.18"
 obisys      = { path = "../obisys" }
 obipipeline = { path = "../obipipeline" }

src/obikphylo/Cargo.toml

@@ -14,8 +14,8 @@ obilayeredmap    = { path = "../obilayeredmap" }
 obiskbuilder     = { path = "../obiskbuilder" }
 obipipeline      = { path = "../obipipeline" }
 memmap2          = "0.9"
-ndarray          = "0.16"
+ndarray          = "0.17"
-rand             = "0.8"
+rand             = "0.10"
 rayon            = "1"
 tracing          = "0.1.44"
 

src/obikphylo/src/siblings/subsample.rs

@@ -22,7 +22,7 @@
 use std::collections::HashSet;
 use std::path::{Path, PathBuf};
 
-use rand::Rng;
+use rand::RngExt;
 use rayon::prelude::*;
 
 use obikindex::{KmerIndex, OKIError, OKIResult};
@@ -97,7 +97,7 @@ fn reservoir_sample_layer(layer_dir: &Path, n_layer: usize) -> OKIResult<HashSet
     let mut reservoir: Vec<usize> = Vec::with_capacity(n_layer);
     let mut seen: u64 = 0;
     let mut family_idx: usize = 0;
-    let mut rng = rand::thread_rng();
+    let mut rng = rand::rng();
     for slot in 0..annex.len() {
         let Some(mask) = annex.get(slot) else { continue };
         if !mask.is_minorant() {
@@ -111,7 +111,7 @@ fn reservoir_sample_layer(layer_dir: &Path, n_layer: usize) -> OKIResult<HashSet
         if reservoir.len() < n_layer {
             reservoir.push(this_family_idx);
         } else {
-            let j = rng.gen_range(0..=seen);
+            let j = rng.random_range(0..=seen);
             if j < n_layer as u64 {
                 reservoir[j as usize] = this_family_idx;
             }
@@ -188,7 +188,7 @@ fn entropy_biased_sample_layer(layer_dir: &Path, p0: f64, bias: EntropyBias) ->
     let entropy_annex = EntropyAnnex::open(&layer_dir.join(ENTROPY_ANNEX_FILE_NAME)).map_err(OKIError::Io)?;
     let mut selected = HashSet::new();
     let mut family_idx: usize = 0;
-    let mut rng = rand::thread_rng();
+    let mut rng = rand::rng();
     for slot in 0..annex.len() {
         let Some(mask) = annex.get(slot) else { continue };
         if !mask.is_minorant() {
@@ -207,7 +207,7 @@ fn entropy_biased_sample_layer(layer_dir: &Path, p0: f64, bias: EntropyBias) ->
         let Some(entropy) = entropy_annex.get(this_family_idx) else { continue };
         let d = (entropy as f64 - bias.mu) / bias.sigma;
         let w = (-0.5 * d * d).exp();
-        if rng.r#gen::<f64>() < p0 * w {
+        if rng.random::<f64>() < p0 * w {
             selected.insert(this_family_idx);
         }
     }

src/obilayeredmap/Cargo.toml

@@ -11,7 +11,7 @@ ptr_hash     = "1.1"
 cacheline-ef = "1.1"
 epserde      = "0.8"
 rayon        = "1"
-ndarray      = "0.16"
+ndarray      = "0.17"
 bitvec       = "1"
 memmap2      = "0.9"
 serde        = { version = "1", features = ["derive"] }

src/obiread/Cargo.toml

@@ -5,11 +5,15 @@ edition = "2024"
 
 [dependencies]
 obikrope = { path = "../obikrope" }
-niffler    = { version = "2", default-features = false, features = ["gz", "bz2", "lzma", "zstd"] }
+niffler    = { version = "3", default-features = false, features = ["gz", "bz2", "lzma", "zstd"] }
-bzip2-sys  = { version = "0.1", features = ["static"] }
+# `niffler`'s plain "bz2" feature only pulls the `bzip2` crate, without
-liblzma-sys = { version = "0.3", features = ["static"] }
+# picking a backend — its own "default" feature is what forwards
-ureq   = "2"
+# "bzip2/default" (the pure-Rust libbz2-rs-sys backend). Depend on it
+# directly, default features on, purely to select that backend.
+bzip2      = "0.6"
+liblzma    = { version = "0.4", features = ["static"] }
+ureq   = "3"
 tracing = "0.1.44"
 tracing-subscriber = { version = "0.3.23", features = ["fmt", "env-filter"] }
-infer = "0.19.0"
+infer = "0.22.0"
 regex = "1"

src/obiread/src/xopen.rs

@@ -52,7 +52,7 @@ fn http_reader(url: &str) -> io::Result<Box<dyn Read + Send>> {
     ureq::get(url)
         .call()
         .map_err(|e| io::Error::new(io::ErrorKind::Other, e.to_string()))
-        .map(|resp| -> Box<dyn Read + Send> { Box::new(resp.into_reader()) })
+        .map(|resp| -> Box<dyn Read + Send> { Box::new(resp.into_body().into_reader()) })
 }
 
 fn decompress(raw: Box<dyn Read + Send>) -> io::Result<Box<dyn Read + Send>> {

src/obiskio/Cargo.toml

@@ -6,7 +6,7 @@ edition = "2024"
 [dependencies]
 niffler = "3.0.0"
 rustix = { version = "1.1.4", features = ["process"] }
-lru = "0.12"
+lru = "0.18"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 

src/obisys/Cargo.toml

@@ -5,7 +5,6 @@ edition = "2024"
 
 [dependencies]
 libc      = "0.2"
-sysinfo   = "0.33"
+sysinfo   = "0.39"
-indicatif = "0.17"
+indicatif = "0.18"
 tracing   = "0.1"
-fs4       = "0.9"

src/obisys/src/lock.rs

@@ -11,9 +11,9 @@ use tracing::info;
 /// only needs to lock the destination.
 ///
 /// Uses the OS's advisory file lock (`flock` on Unix, `LockFileEx` on
-/// Windows) via `fs4`, not a hand-rolled PID file: the OS releases it
+/// Windows) via `std::fs::File::lock`/`try_lock`, not a hand-rolled PID
-/// automatically on process exit, including a crash — no stale-lock cleanup
+/// file: the OS releases it automatically on process exit, including a
-/// logic needed.
+/// crash — no stale-lock cleanup logic needed.
 pub struct DirLock {
     _file: std::fs::File,
 }
@@ -23,8 +23,6 @@ impl DirLock {
     /// and the lock file within it if needed). Logs once if the wait is
     /// non-trivial, so a blocked command doesn't look silently hung.
     pub fn acquire(dir: &std::path::Path) -> std::io::Result<Self> {
-        use fs4::fs_std::FileExt;
-
         std::fs::create_dir_all(dir)?;
         let lock_path = dir.join(".obikmer.lock");
         let file = std::fs::OpenOptions::new()
@@ -33,9 +31,9 @@ impl DirLock {
             .write(true)
             .open(&lock_path)?;
 
-        if file.try_lock_exclusive().is_err() {
+        if file.try_lock().is_err() {
             info!(dir = %dir.display(), "waiting for another obikmer process to release this index");
-            file.lock_exclusive()?;
+            file.lock()?;
         }
         Ok(Self { _file: file })
     }