Extends `KmerIndex::finalize_indexed` to accept a `sparse` boolean parameter, enabling control over whether output presence matrices are packed in a compact sparse format or stored densely. The merge command now exposes a `--dense` flag to invert this setting, defaulting to the existing sparse representation. Call sites in select and rebuild modules pass `false` to preserve current behavior. A benchmark lock file is also added for dependency tracking.
Extended `nonzero_triples` with a boolean flag to conditionally skip sorting when input slots are already ordered. Updated call sites to pass appropriate flags, enabling a single-pass columnar traversal in the sparse matrix builder. Refactored `build_from_dense` into a batched processing pipeline that reduces repeated file access and improves sequential read performance. Added a diagnostic example to validate the new builder against existing dense matrices.
Added a `col_weights` method to index layers for computing per-genome column sums or presence k-mer counts. Refactored `obikstats` to implement the `Algorithm` trait with a two-phase `new`/`run` model, replacing manual layer resolution with `IndexCache` for eager file I/O. Simplified per-genome counting logic and updated public exports and dependencies accordingly.
Introduces the `annotate` subcommand to apply genome metadata from an external CSV file to a pre-built k-mer index. The command supports configurable field separators, ID columns, and null markers, while providing a `--dump` option to export current index metadata as sorted CSV. Supporting changes include minor internal refactoring in `obikindex` to use object-level directory accessors and updates to the `csv` dependency.
Introduce `iter_unitigs` methods across the index cache, content layer, MphfLayer, and typed layer to stream whole reconstructed sequences directly from underlying storage without decomposing into k-mers. Add a corresponding low-level streaming iterator in obiskio for thread-safe, lazy reads from memory-mapped files. Include a new CLI subcommand to compute and display approximate false-positive rates based on provided indexing parameters.
Introduces a new `superkmer` command that implements a multi-stage pipeline for reading nucleotide pages and constructing superkmers with configurable parameters. The implementation uses partition-aware scatter writing via `obifastwrite` to distribute output batches across workers, and adds the necessary local dependencies to the project manifest.
Introduces directory-level locking to prevent concurrent index operations from corrupting shared directories, along with explicit APIs for acquiring, probing, and releasing locks. Restructures the index cache crate to use a layered store architecture that eagerly initializes metadata and provides fast hierarchical lookups. Updates dependent modules, test suites, and CLI commands to align with the refactored API surface, and adds an end-to-end smoke test for validation.
Replaces manual filesystem path handling and parallel iteration with dedicated internal utilities. Introduces `par_over_layer0` and `layer0()` to abstract partition layer access, enabling partition-driven initialization and centralized progress tracking. Removes disabled or internal methods (`rewrite_config`, `open_data`) no longer part of the active interface. Updates error propagation and metadata persistence to align with the new layer-centric workflow.
This commit partitions the obikindex crate into multiple focused subcrates (obikfilter, obikmerge, obikquery, obikrebuild, obikselect, obikstats, obikdump, and obikidxcache) to reduce coupling and clarify module boundaries. It standardizes error handling across the workspace using OKIError and OKIResult, updates index APIs to support lazy, disk-backed partition access, and migrates NUMA system utilities to a new obisys crate. All modifications are structural, focusing on dependency graph expansion, import path updates, and API surface reorganization without altering core runtime behavior.
The MphfLayer::open signature now accepts only a directory path. Mode configuration is resolved internally via disk-based auto-detection using LayerEvidence, eliminating the need for explicit arguments from callers. All internal query methods and external call sites have been updated accordingly, while TypedLayer::open temporarily retains the unused parameter for future cleanup.
Define a shared Algorithm trait with an associated Output type and a parameterless run(&mut self) method. Refactor PartitionRouter, Dereplicator, Counter, and LayerBuilder to implement the trait, standardizing receivers to &mut self and moving configuration and progress callbacks to dedicated builder setters. Decouple error handling using a generic boxed error type and update workspace dependencies accordingly.
Migrate index state tracking from filesystem sentinel files to an on-disk JSON schema within `index.meta`. The `IndexMeta` struct is now wrapped in an `Arc` with internal locking, exposing only fallible methods for genome and state access. In-memory mutation capabilities have been removed, requiring callers to handle I/O errors explicitly and pass immutable references to downstream components like `PartitionRouter`. Public sentinel constants have been removed from exports.
Extracts directory bookkeeping and construction methods into a new IndexBuilder trait, separating maintenance operations from scientific computation logic. Migrates relevant methods to the new trait, updates module imports across obikindex, and narrows the crate's public API surface. Adds an end-to-end CLI smoke test to verify multi-index merge workflows.
Moves pipeline state bookkeeping, including sentinel file marking and spectrum persistence, into the algorithms' run and close methods. Introduces a crate-private extension trait to satisfy Rust's orphan rule while implementing construction-only operations on KmerIndex. Updates helper function visibility for cross-crate access and removes the now-empty index_layer module.
This commit renames the `Layer` type to `KmerLayer` throughout the `obikindex` and `obikphylo` crates. All imports, struct fields, function signatures, pattern matches, and instantiation calls have been updated accordingly. The change is a purely structural refactor that tightens type constraints without altering runtime behavior or data models.
Introduce a dedicated LayerBuilder struct to orchestrate parallel layer 0 construction across partitions. The fluent API supports configurable abundance thresholds and intermediate artifact retention. Parallel execution is delegated to a NUMA-aware PartitionRunner scheduler, while progress reporting and stage timing are shifted to the command layer. A new mark_indexed method cleanly separates state tracking from orchestration by generating a completion sentinel.
The layer API has been updated to rename `index` and `index_batch` to `hash` and `hash_batch`, aligning with raw MPHF slot hashing behavior. New public methods have been added to directly expose kmer hashing and iteration, enforcing strict state validation on empty layers. All other modifications are consistent code formatting adjustments without functional changes.
Decouple k-mer counting from the partitioner by introducing a new `Counter` struct. The module exposes a fluent builder API with optional partial file retention, executes partition processing in parallel via Rayon with memory-aware chunk sizing, and integrates thread-safe progress callbacks. Update all callers to use the new counter, simplify test pipelines by removing serialization overhead, and clarify algorithm separation in module documentation.
The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
Move partition dereplication logic into a dedicated `obikderep` crate that implements a two-phase hash-split/merge strategy with Rayon for parallel processing. Centralize superkmer file path construction in `obilayeredmap` and update dependent crates to use the new pipeline and shared path helpers. Adjust test suites to explicitly invoke the dereplication step.
Invert dependencies by moving partition path primitives to a dedicated module and updating the index crate accordingly. Reshape the partition router to accept a mutable index reference, enabling chainable configuration and resolving lifetime issues with explicit drops. Shift orchestration logic from the index crate to the CLI, replacing monolithic scatter calls with discrete dereplication and counting steps. Introduce a generic progress callback API to decouple rate calculation from UI rendering, and correct file I/O paths to route layer-0 artifacts under the partition index directory.
Introduces the `obikpartition` crate containing the `KmerPartition` struct to manage open layers in sequential order. The `open` constructor eagerly initializes layers under a specified directory, while the `find` method returns the index of the first layer containing a given k-mer. Dependencies are strictly scoped to `obilayeredmap` and `obikseq`, with unit tests validating layer ingestion and single-kmer lookup behavior. Batch lookups and further migration are deferred to a subsequent phase.
The `Layer` enum is transformed into a lifecycle-aware state machine with an `Empty` variant representing an unconstructed directory. Read and query operations now explicitly panic when invoked on this state, enforcing explicit progression through `create()` before use. Iterator methods are updated to handle the new variant exhaustively, and module visibility constants are adjusted to support the refactored structure.
Renames `Layer<D>` to `TypedLayer<D>` to establish a distinct typed abstraction. Introduces a new heterogeneous `Layer` enum that unifies count and presence storage with runtime dispatch, delegating operations to the underlying typed variants. Updates cache, index, and phylo consumers to align with the renamed type and new extension traits, preserving full test suite stability while preparing the foundation for multi-partition caching.
Consolidates partition logic, metadata storage, and layer management directly into KmerIndex. Renames obikpartitionner to obikpartition, retaining only PartitionRouter for superkmer routing. Removes intermediate .partition() accessors in favor of direct methods on the index and updates PartitionCache::build to accept &KmerIndex directly. Derives n_partitions from config.n_bits and consolidates k-mer/minimizer sizes into IndexMeta.config. Fixes a regression where PartitionRouter::open incorrectly defaulted to closed.
Update all dependent crates and internal imports to reference the renamed module. Expose configuration parameters and genome metadata through new public accessor methods in the index API, and update documentation comments to align with the new package name.
Adjusts the visibility of the COUNTS_DIR constant to pub(crate), enabling internal crate-wide access without altering runtime behavior or data structures.
Update Cargo manifests, dependency paths, and Rust imports across the workspace to reflect the `obikpartitionner` to `obikpartition` rename. Synchronize architecture and implementation documentation with the new module structure. Fix minor syntax issues in test assertions to ensure compilation compatibility. No behavioral or API changes are introduced.
Rename the KmerPartition type to KmerPartitions across obikindex, obikpartitionner, and obikphylo/siblings to reflect an updated data model. Update the Mat enum in siblings/cache.rs to add a SparsePresence variant and simplify opening logic by delegating sparse versus dense detection to PersistentBitMatrix. Apply consistent code formatting, import reordering, and multi-line refactoring throughout the affected modules.
Replaced scattered direct metadata loading with centralized instance methods on `KmerPartition` to guarantee consistent error mapping and legacy recovery. Introduced `StorageKind`, `LayerContent`, and `EvidenceKind` enums alongside lightweight disk-probe methods that inspect file presence without opening heavy data structures. Updated callers across the index, partitioner, and phylo modules to use the new partition API, and added unit tests validating the introspection behavior.
Introduce a design specification outlining performance bottlenecks in layer data access and an agreed-upon implementation direction for caching. Refactor the codebase to centralize index and layer directory path resolution within the partition object, replacing manual string joining and external helper functions with dedicated accessor methods.
Standardizes layer directory path resolution across obikindex, obikpartitionner, and obikphylo by replacing inline string formatting and join calls with the centralized obilayeredmap::layer_dir utility. This refactoring removes redundant path construction logic while preserving existing iteration bounds, control flow, and public APIs.
Replace manual directory formatting and inline matrix opening with the newly introduced `layer_dir` and `open_data` helpers from the `obilayeredmap` crate. This standardizes filesystem access, encapsulates layer naming conventions, and simplifies error handling without altering computational behavior or test suites.
Refactor `pack_sparse_bit_matrix` to conditionally open either a packed or columnar dense representation based on file existence. This eliminates the previous unconditional pre-conversion step that forced a full packed copy. Update cleanup logic and add tests to verify direct transposition, proper file handling, and idempotency for both formats.
Consolidate row extraction and parallel column reduction into reusable generic utilities within the pairwise module. Replace manual iteration in builder and view methods with iterator-based zipped loops and deferred overflow processing to improve memory access patterns. Optimize distance functions by switching to bulk byte passes for branch-free SIMD vectorization, adding a merge-like helper to correct masked overflow values without secondary allocations. Add a comprehensive test validating the optimized distance paths against naive references across multiple thresholds and overflow scenarios.
Consolidate bit matrix operations by delegating pairwise metric calculations to IntSliceView and col_view implementations. Introduce chunked_presence_count in colgroup to handle threshold-based accumulation efficiently. Add a direct point lookup method to the sparse matrix representation to eliminate buffer allocations, and simplify match arms in persistent accessors accordingly. All public APIs and behavioral contracts remain unchanged.
Extract duplicated traversal logic into a shared `NonzeroSlotsView` trait and `nonzero_triples` helper. Update slice view iterators to consistently yield `(position, value)` tuples and use a temporary buffer with explicit permutation mapping for sorted access. Delegate manual slot sorting and column iteration to the new shared helper, eliminating per-format duplication and eager collection while preserving existing public API signatures.
Refactored `sub_matrix` to delegate column population to `fill_sub_matrix`, replacing manual buffer allocation and explicit permutation loops with direct value assignment via `enumerate_slots_values`. This eliminates intermediate allocations, reduces pipeline overhead, and simplifies control flow while preserving slot ordering and permutation semantics.
Renames `get_batch`, `fill_batch`, and `fill_batch_sorted` to `collect_slots_values`, `fill_slots_values`, and `fill_slots_values_sorted` to align with updated `common_traits 0.13` APIs. Introduces optimized batch retrieval that sorts input indices for sequential mmap access before reordering outputs to match the original query order. Updates test suites to reflect the new method signatures without altering validation logic or coverage.
Replaces nested column-major point lookups with a batched iterator that delegates to format-native traversal strategies. The implementation enforces a single pass per matrix type, using row-major iteration for sparse formats and eager collection for packed/columnar layouts while preserving original slot ordering. Memory allocation is optimized by removing `n_cols`-wide buffers in favor of per-row buffering or lazy iteration. Correctness tests verify iterator output against dense baselines across all supported layouts, and architecture documentation is updated to reflect the new format-agnostic query pattern.
Introduces a complete query benchmark track to evaluate performance and verify consistency between dense and sparse index formats. Adds scripts to simulate fixed-size paired-end reads, pack a sparse presence index, execute queries in both modes, and capture wall time and RSS metrics. Includes a verification step that compares outputs by read ID to ensure content identity across parallel processing. Updates build configuration, documentation, and ignore patterns to support the new pipeline for two microbial specimens.
Bumps core dependencies including ndarray, rand, hashbrown, niffler, ureq, sysinfo, indicatif, lru, and remove_dir_all. Adapts source code to accommodate breaking changes by migrating RNG initialization, adjusting HTTP response handling, and replacing the fs4 crate with standard library file locking. Adds a planning document for query benchmarking and sparse index regression tests.
Updated core libraries and ecosystem packages, including the full serde suite and regex dependencies. Restructured syn dependency resolution by pinning 2.0.117 while introducing a standalone 3.0.3 entry for derive macros. Downgraded windows-sys to 0.59.0 and replaced the anes dependency in criterion2 with bpaf and walkdir. Removed an unused IntSliceView import from test modules.
Introduces raw mapping and iteration APIs that bypass membership checks, clarifies variant-specific storage layouts and auto-detection logic, and documents optimized batch access patterns with caller-provided buffers. Removes the outdated obicompactvector_reflexion.md specification to consolidate architectural details into current implementation docs.
This change removes precomputed model parameters, phylogenetic tree datasets, k-mer spectrum data, and compressed profile archives. It also deletes runtime logs and detailed pipeline documentation. The repository ignore list is updated to exclude the sandbox directory.
Eric Coissac