Push smluomvxpptv #59

2026-07-07T17:12:55Z

coissac commented

2026-07-07 17:12:55 +00:00

No description provided.

coissac added 9 commits 2026-07-07 17:12:56 +00:00

docs: add future work section for parallel gzip decompression 00b4b1fa51

Proposes replacing the single-threaded niffler/flate2 pipeline in `obiread::xopen` with `rapidgzip-rs` for local `.gz` files. Details constraints such as path dependencies, non-seekable streams, C++ toolchain requirements, and binding maturity. Marks the optimization as parked pending throughput and correctness validation.

docs: clarify query processing and add performance roadmap 8f0ceec784

Clarifies that query processing is fully inlined within `process_chunk` using flat allocations, refining monotone-deque sliding window semantics and `kmer_missing` tracking. Updates `QueryLayer::open` variant precedence and introduces a phased roadmap (Phases 0–6) to address performance bottlenecks through parallel I/O, genome-aware chunk sizing, k-mer dereplication, NUMA-aware matrix fetches, and sparse Findere rework.

feat(query): add throughput metering and --max-open-files flag 61c390503d

Introduces an EMA-based throughput meter that dynamically updates a spinner with MB/s rates, along with atomic counters for tracking cumulative bytes and active chunks. Adds final pipeline reporting and consolidates imports for cleaner performance instrumentation.

refactor(query): implement throttled per-file streaming pipeline 9d49929b0c

Replace the flat chunk iterator with a throttled, per-file streaming architecture using `obipipeline::throttle`. The new `GuardedChunkIter` binds file handles to their `ThrottleGuard`, enforcing concurrent open-file limits and tracking active files via an atomic counter. Pipeline stages and the progress spinner are updated to support this resource-aware, parallelized I/O flow.

perf(query): replace static divisor with dynamic overhead multiplier 9d7ced4493

Replaces the static 16 divisor with a dynamic overhead_multiplier that scales chunk size based on n_genomes, the --detail flag, and a safety factor. This bounds per-chunk memory usage to ≤50% of available RAM across concurrent workers by accounting for genome-scaled k-mer buffers and optional coverage data.

refactor(query): deduplicate k-mers upfront and split MPHF lookup a348637f3b

Refactor `QueryBatch` construction to perform canonical k-mer deduplication and partition routing during initialization, eliminating post-batch splitting. Split the `QueryLayer` MPHF lookup into separate `find_slot` and `fill_row` methods, updating callbacks to pass occurrence descriptors instead of indices. Introduce `QueryStats` for tracking MPHF calls and dereplication ratios, and add comprehensive unit tests for batch construction, stats arithmetic, and safe partition handling. Expose new query-layer types in the public API.

refactor(query): optimize mmap locality with column-major matrix fetch 040eff140c

Refactor the query pipeline into a two-stage MPHF hit-detection pass followed by a column-major matrix fetch to improve cache efficiency. Introduce a QueryHit enum for event-driven callbacks, decoupling hit detection from data population. Add scan/fetch metrics to QueryStats, update Phase 4 architecture docs, and align tests with the new callback signature.

perf: optimize k-mer queries with sparse index and run-based aggregation ae42a061bd

Replaces the dense `KmerResults` matrix with a sparse `SmerIndex` (`Vec<bool>` + offsets) that tracks k-mer presence independently of per-genome counts. Introduces a new aggregation pass, `sparse_findere_for_genome`, which sorts hits, detects contiguous runs, and applies monotone-deque scans to compute sliding-window minimums. This reduces query complexity from O(n_smers) to O(hits log hits), significantly lowering memory overhead and computational cost for low-density queries. Adds a deterministic PRNG and dense reference oracle in tests to validate correctness against randomized inputs without external property-testing crates.

ci: improve release workflow and bump obikmer to v1.1.37

Release / create-release (push) Successful in 2m27s

Details

CI / build (pull_request) Successful in 3m32s

Details

Release / build-linux-x86_64 (push) Successful in 8m5s

Details

Release / build-macos-arm64 (push) Successful in 1m41s

Details

eea884f393

Replace inline `docker run` with explicit container lifecycle management to improve isolation and verify exit statuses. Bump `obikmer` crate version to 1.1.37. Add debug logging for sparse hit structure memory tracking, update chunk memory documentation to reflect the Findere rework, and clarify `BYTES_PER_KMER_PER_GENOME` as a pathological bound for safety tuning.

coissac merged commit dff5d2f457 into main

2026-07-07 17:13:04 +00:00

coissac deleted branch push-smluomvxpptv

2026-07-07 17:13:04 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: OBIKmers/obikmer#59