This change introduces a comprehensive profiling system, allowing for periodic resource usage monitoring (CPU, I/O, memory) via a background thread. It also exposes new public APIs for budget management, locking, NUMA partitioning, and progress tracking.
This change introduces logic to detect NUMA topology, calculate per-node worker limits based on available CPU cores, and cap resource allocation to ensure worker counts respect the physical capacity of each node.
This change implements mechanisms to enforce resource limits based on the available CPU budget, including setting a hard CPU ceiling and calculating effective thread counts for CLI, query, and superkmer operations.
Update obisys dependency configuration across multiple crates to explicitly set default-features = false. This restricts compilation to only explicitly enabled feature flags, affecting build resolution and potentially compile-time paths or binary size. Also updates the obikmer package version to 1.3.1.
Restructure the benchmark pipeline to direct all simulated data, indices, statistics, and query outputs into a unified `run/` directory. Update Makefile targets, shell scripts, and Python utilities to resolve paths relative to this new base. Adjust documentation and dependency tracking to match the revised layout, and remove outdated temporary artifacts.
Introduces a `batch_int_group_stats` API for computing presence counts, sums, minimums, and maximums across sparse and dense matrix representations. The selection layer now utilizes this batched approach to optimize aggregation semantics for boolean and numeric operations. Additionally, reorganizes the benchmarking infrastructure to support querying across presence and count index variants in both dense and sparse formats, including new packing scripts and updated statistics aggregation.
Update documentation to reflect architectural fixes in the NUMA partition runner and implementation changes in the merge and select modules. The partition runner now wraps task execution in `catch_unwind`, emitting panic events that count toward completion thresholds to prevent controller deadlocks, while re-raising the first payload on return. CLI documentation for select has been updated to mandate output directories, introduce default hard-linking with automatic fallback, and document presence aggregation optimizations. Known issues such as self-merge crashes and sparse-packed index gaps are now documented as failing fast due to these improvements.
Introduces recursive directory replication functions that default to hard-linking for space efficiency, with automatic fallback to full file copying on cross-filesystem failures. Enforces mandatory real copies for presence and counts subdirectories to guarantee independent data matrices before merging, while preserving the existing separation between immutable index files and mutable matrix files.
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
Introduces the `--force-copy` CLI argument to bypass hard-linking during selection. When enabled, unchanged kmer-identity files are copied instead of hard-linked to ensure output independence from the source filesystem. The default behavior remains unchanged, continuing to use hard-links with automatic fallback to copying on failure.
Renames the reindex command to convert across all usage guides and navigation menus. Adds documentation for the newly introduced name-tree command. Updates flag references, such as replacing --metric with --distance in pack and phylo modules, and adjusts MkDocs configuration to match the revised structure. All changes are strictly limited to documentation and configuration files.
Restructure the workspace and rename the CLI application to obikmer2. Replace direct KmerIndex usage across all commands with Arc-wrapped IndexCache to enable shared ownership. Introduce builder patterns for algorithmic operations and integrate explicit progress tracking. Add new Phylo, NameTree, and Convert subcommands with expanded CLI flags. Consolidate module structure, update dependency specifications, and remove legacy directories.
Extends the entropy annex pipeline to optionally compute family overlap data in a single pass, introducing a memory-mapped binary format and an in-memory accumulator for efficient storage. Adds --min-shared-family to auto-exclude genomes below a threshold during SNP-family distance computation, and --family-overlap to export a shared family matrix as CSV. Centralizes upper-triangle indexing into a shared helper function to eliminate duplication.
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
Extends the phylogenetic pipeline with exporters for TNT, PhyG, and IQ-TREE that generate executable scripts, cost matrices, and recoded alignments. Adds internal helpers for state indexing, floating-point matrix scaling, and Floyd-Warshall metric closure to satisfy external tool constraints. Adjusts visibility modifiers for sibling iterators and entropy annex structs, and updates design documentation for evolutionary distance metrics.
Introduce the --sankoff flag to generate subsampled calibration bundles and pairwise cost matrices. Expose internal sibling algorithm types as public to support external consumption. Implement YAML, CSV, and FASTA output serialization using serde, gated by a configurable substitution saturation ceiling and requiring the --subsample flag.
Replaces the previous two-pass design with a single-pass reduction that computes per-genome-pair substitution counts, cardinality and base-composition tallies, and transition cost matrices. Introduces internal APIs to derive downstream n×n matrices from reduced data and applies configurable filtering based on SNP ratios and exclusion masks. Adds a fused pipeline entry point aggregating alignment and distance outputs for downstream phylogenetic commands. Adjusts visibility of existing reduction functions to restrict external access while permitting sibling module invocation.
Replace per-family callbacks with a per-layer batch callback that accumulates surviving families into an Arc<Vec<SurvivingFamily>>. Update the alignment reduction routine to iterate over this batched structure, enabling zero-copy sharing for parallel operations and structuring the code for future extensibility.
Introduces `--sibling-stats` and `--sibling-hist` flags to compute and export family-size distributions from pre-built annexes. The new algorithms module implements parallel and sequential scanning routines, while the CLI layer handles CSV export with deduplication and standardized error handling.
Introduces configurable phylogenetic sampling and masking controls via new CLI flags (`--subsample`, `--entropy`, `--exclude-genome`, etc.). Adds a complete SNP pseudo-alignment pipeline featuring entropy-biased Gaussian sampling, post-hoc state masking, and proportional per-layer filtering. Extends the FASTA writer with a `write_plain_record` API for bare-header output without JSON annotations.
Introduces an entropy.pent persisted format and algorithms to compute per-family Shannon entropy across genomic partitions. Refactors IndexCache to use Arc-based layer sharing for efficient iteration and restructures the siblings module with extension traits for annex generation. Adds a --shannon CLI flag to export partition-level entropy metrics to CSV.
Refactors the `SiblingAnnexBuildExt` trait to `SiblingExt` across all imports, implementations, and documentation. Introduces new modules for streaming and batched iteration over sibling layers, along with a two-phase pipeline for resolving minorant families.
Introduces a new `--sibling-annex` CLI flag that triggers the construction of a sibling-count/minorant annex for multi-genome indices. The implementation adds a `FamilyMask` data structure with memory-mapped I/O, enabling lock-free concurrent updates via atomic bitwise operations. Batched processing improves cache locality and parallelism, while helper functions derive family presence and minorant flags dynamically. The feature is exposed through an `IndexCache` extension trait, protected by an exclusive directory lock to prevent index corruption during construction.
Implement an `obipipeline::Throttle` with an RAII guard to acquire and release concurrency slots. Expose new bounded parallel methods on `IndexCache` to process cached layers with a configurable cap. Refactor downstream aggregation logic to use single-pass parallel map-reduce instead of manual collect-map-reduce sequences, enforcing a maximum of 8 concurrent layer scans to bound memory usage.
Refactor phylogenetic algorithms (`neighbor_joining`, `upgma`) to return an explicit `Tree` struct instead of a serialized Newick string. This change makes serialization an explicit step for downstream consumers via the new public `Tree::to_newick()` method, decoupling tree construction from output formatting. The `siblings` module has also been moved to `siblings_old`.
Introduce the `obikphylo` crate to support genome-vs-genome distance matrix computation and phylogenetic tree inference via Neighbor-Joining and UPGMA algorithms. Extend the `obikmer2` CLI with a new `phylo` command that exposes configurable metrics, presence thresholds, and CSV/Newick output options. Refactor `obikindex` to expose trait-based partial aggregation for efficient distance metric finalization without full matrix materialization. Update dependency graphs and remove obsolete storage modules.
Update all matrix type references, imports, and instantiations across
the codebase to use the new PersistentIntMatrix name. The change also
includes standardizing code formatting, such as multi-line statements
and import ordering, without altering any behavioral logic or public
API contracts.
Relocate cardcomp into the siblings directory and update internal import paths to reflect the new hierarchy. Adjust the crate's public API surface by updating module declarations and re-exports. This is a purely structural change with no modifications to logic, data models, or runtime behavior.
Introduce a new CLI command that modifies existing indices to support exact, approximate, or hybrid evidence modes. This change enables the reindex module in obikrebuild, exposing an IndexReindex trait for evidence persistence and layer processing. It also refactors obikindexer to publicly expose an IndexBuilder trait for post-build evidence construction, while adding a set_evidence API in obikindex to safely update configuration fields without altering core parameters.
Replaces lifetime-bound references with runtime reference counting across multiple crates. This enables safe concurrent access across parallel workers without explicit cloning or manual lifetime management. Introduces the `query` and `utils` CLI commands in obikmer2, along with supporting modules for batch processing, sparse indexing, sliding-window findere logic, and output formatting. Updates dependency manifests and aligns test suites with the new ownership model.
Change the return type of `nonzero_iter` from an opaque `impl Iterator` to a `Box<dyn Iterator>`. This enables direct return coercion in the `Count` layer branch, removing an explicit `Box::new()` wrapper and aligning it with the existing `Presence` implementation. The modification shifts iterator construction to runtime dynamic dispatch while preserving the public API contract and iteration semantics.
Introduce a new `Sparse` format alongside existing `Columnar` and `Packed` variants, enabling optimized row-major pairwise counting for distance and similarity metrics via the `CountPartials` trait. Update storage detection priorities, extend matrix dispatch logic to sparse backends, and correct diagonal/off-diagonal formulas in bit matrix partial computations. Expand layer APIs with format-agnostic `nonzero_iter`, update usage documentation for the `--sparse` flag, and add comprehensive tests verifying roundtrip integrity and metric equivalence against dense implementations.
Adds the `pack` subcommand to `obikmer2` for persisting presence and count matrices in sparse or dense formats. Introduces the `compact_layer` module in `obikrebuild` to merge multiple partition layers into a single layer in-place via atomic operations. Updates dependencies across `obikdump`, `obikmer2`, and `obikrebuild` to resolve new local crate references. Shifts the architecture from multi-layer accumulation to in-place compaction, removing legacy rebuild modules.
Introduces a new unitig extraction feature that reads k-mers from an index, filters them by group metadata, and partitions the computation using rayon. The implementation constructs per-partition de Bruijn graphs, merges them, computes node degrees, and writes the resulting sequences as FASTA. A corresponding CLI command is added to obikmer2 to expose this functionality with configurable filtering and progress reporting.
Introduces a new `dump` CLI command to export index k-mers as a CSV table. Adds a reusable `GroupFilterArgs` struct for ingroup/outgroup metadata-predicate quorum filtering with configurable count and fraction thresholds. Refactors the filter command to use explicit flags and a `GenomeSelector`-based pipeline, improving error handling and predicate application order. Exposes `IndexDump` from the `obikdump` crate and updates local dependencies accordingly.
Introduces a new `obikselect` module to handle k-mer selection and column aggregation. Adds a `select` CLI command in `obikmer2` that supports group predicates, aggregate operators, and output column filtering. Implements operator parsing with case-insensitive matching and default rules, along with a centralized module for resolving group specifications from metadata or genome filters. Updates dependencies to include the new crate and its prerequisites.
Introduces a unified MatrixBuilder abstraction for persistent bit and integer matrices, replacing custom dispatch enums and boolean flags across consumers. Refactors index, merge, and select layers to adopt explicit merge modes and centralized lifecycle management. Adds a complete k-mer filtering implementation with partition-level processing, progress tracking, and a new CLI subcommand supporting configurable predicates and thresholds.
Introduces `PersistentSparseCompactIntMatrix`, a row-major sparse integer matrix combining a bitmatrix support layer with dual value streams for singleton and multi partitions. Adds builder and packing utilities to convert dense count matrices into the new on-disk format, featuring idempotency guards and crash-safe persistence workflows. Exposes core accessors, iterators, and column weight calculations while adjusting internal visibility for companion modules. Includes unit tests and diagnostic examples to validate equivalence with dense representations and measure compaction metrics.
Move the partition iterator implementation from obikdump to obikfilter, updating imports and exposing the type publicly. Add obikidxcache as a local dependency for obikfilter. In obikindexer, expose the new build_layer_from_kmers function to enable shared graph-building logic across pipelines without code duplication.
Move partition iteration logic to obikdump, introducing a FilteredPartitionIter trait over IndexCache for batch-oriented scanning with configurable data retrieval and early termination. Consolidate ingroup and outgroup index storage in GroupQuorumFilter into a unified Selection struct driven by predicate matching. Update dependency manifests to include obikidxcache, rayon, and obikentropy, and remove the deprecated dump_layer module while adjusting public API re-exports.
Extracts file creation, opening, and memory mapping logic into a new `mmap_file` module. This replaces repetitive unsafe operations across persistent data structures with standardized helper functions, ensuring consistent header validation, error handling, and data layout while keeping public APIs unchanged.
Update PersistentSparseBitMatrix documentation for clarity. Introduce select0 to the rank/select module, and add next_geq, contains, successor, and predecessor methods to the Elias-Fano implementation. Include naive reference implementations and unit tests to validate correctness and boundary conditions for all new APIs. All changes are strictly additive.
Shifts the selection logic from a direct method into a dedicated `Select` struct implementing the `Algorithm` trait. Introduces a two-phase initialization pattern with explicit builder methods and closure-based progress callbacks. Updates the crate's dependency graph to include `obikalgorithm`, refactors layer processing with unified matrix dispatch, and implements dual directory routing for partition execution.
Eric Coissac
coissac