Commit Graph
75 Commits
Author SHA1 Message Date
Eric Coissac 354e6f9bf1 Add unified matrix builder abstraction and k-mer filtering CLI
Introduces a unified MatrixBuilder abstraction for persistent bit and integer matrices, replacing custom dispatch enums and boolean flags across consumers. Refactors index, merge, and select layers to adopt explicit merge modes and centralized lifecycle management. Adds a complete k-mer filtering implementation with partition-level processing, progress tracking, and a new CLI subcommand supporting configurable predicates and thresholds.
2026-08-26 14:19:15 +02:00
Eric Coissac b7a8b5e6cf feat: add sparse compact integer matrix implementation and utilities
Introduces `PersistentSparseCompactIntMatrix`, a row-major sparse integer matrix combining a bitmatrix support layer with dual value streams for singleton and multi partitions. Adds builder and packing utilities to convert dense count matrices into the new on-disk format, featuring idempotency guards and crash-safe persistence workflows. Exposes core accessors, iterators, and column weight calculations while adjusting internal visibility for companion modules. Includes unit tests and diagnostic examples to validate equivalence with dense representations and measure compaction metrics.
2026-08-26 14:05:54 +02:00
Eric Coissac 881b1532b5 refactor: consolidate mmap file operations into shared helper functions
Extracts file creation, opening, and memory mapping logic into a new `mmap_file` module. This replaces repetitive unsafe operations across persistent data structures with standardized helper functions, ensuring consistent header validation, error handling, and data layout while keeping public APIs unchanged.
2026-08-26 09:26:49 +02:00
Eric Coissac ad0b7173a3 Add select0 method and expand Elias-Fano query API
Update PersistentSparseBitMatrix documentation for clarity. Introduce select0 to the rank/select module, and add next_geq, contains, successor, and predecessor methods to the Elias-Fano implementation. Include naive reference implementations and unit tests to validate correctness and boundary conditions for all new APIs. All changes are strictly additive.
2026-08-26 09:21:52 +02:00
Eric Coissac bb380d0c7d Add sorted_slots parameter to nonzero_triples and batch build_from_dense
Extended `nonzero_triples` with a boolean flag to conditionally skip sorting when input slots are already ordered. Updated call sites to pass appropriate flags, enabling a single-pass columnar traversal in the sparse matrix builder. Refactored `build_from_dense` into a batched processing pipeline that reduces repeated file access and improves sequential read performance. Added a diagnostic example to validate the new builder against existing dense matrices.
2026-08-22 17:33:07 +02:00
Eric Coissac 5c1584967f feat: enable index resumption and enforce directory creation
The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.
2026-08-21 05:06:38 +02:00
Eric Coissac b5ec0122d0 Rename obikpartition crate to obikpartitionner
Update all dependent crates and internal imports to reference the renamed module. Expose configuration parameters and genome metadata through new public accessor methods in the index API, and update documentation comments to align with the new package name.
2026-08-20 19:43:00 +02:00
Eric Coissac 9379bbaad8 refactor: rename obikpartitionner crate to obikpartition
Update Cargo manifests, dependency paths, and Rust imports across the workspace to reflect the `obikpartitionner` to `obikpartition` rename. Synchronize architecture and implementation documentation with the new module structure. Fix minor syntax issues in test assertions to ensure compilation compatibility. No behavioral or API changes are introduced.
2026-08-20 16:41:22 +02:00
Eric Coissac 76cbd3a886 Centralize partition metadata access and add layer introspection APIs
Replaced scattered direct metadata loading with centralized instance methods on `KmerPartition` to guarantee consistent error mapping and legacy recovery. Introduced `StorageKind`, `LayerContent`, and `EvidenceKind` enums alongside lightweight disk-probe methods that inspect file presence without opening heavy data structures. Updated callers across the index, partitioner, and phylo modules to use the new partition API, and added unit tests validating the introspection behavior.
2026-08-20 15:29:53 +02:00
Eric Coissac 640f29725b Optimize sparse bit matrix packing to avoid intermediate copies
Refactor `pack_sparse_bit_matrix` to conditionally open either a packed or columnar dense representation based on file existence. This eliminates the previous unconditional pre-conversion step that forced a full packed copy. Update cleanup logic and add tests to verify direct transposition, proper file handling, and idempotency for both formats.
2026-08-20 14:38:20 +02:00
Eric Coissac c2e0533fa9 Refactor bit matrix operations and optimize distance computations
Consolidate row extraction and parallel column reduction into reusable generic utilities within the pairwise module. Replace manual iteration in builder and view methods with iterator-based zipped loops and deferred overflow processing to improve memory access patterns. Optimize distance functions by switching to bulk byte passes for branch-free SIMD vectorization, adding a merge-like helper to correct masked overflow values without secondary allocations. Add a comprehensive test validating the optimized distance paths against naive references across multiple thresholds and overflow scenarios.
2026-08-20 14:35:22 +02:00
Eric Coissac 9abee87af3 refactor: delegate distance computations and simplify matrix logic
Consolidate bit matrix operations by delegating pairwise metric calculations to IntSliceView and col_view implementations. Introduce chunked_presence_count in colgroup to handle threshold-based accumulation efficiently. Add a direct point lookup method to the sparse matrix representation to eliminate buffer allocations, and simplify match arms in persistent accessors accordingly. All public APIs and behavioral contracts remain unchanged.
2026-08-20 14:27:32 +02:00
Eric Coissac 19f9954050 refactor: centralize nonzero slot iteration logic across views
Extract duplicated traversal logic into a shared `NonzeroSlotsView` trait and `nonzero_triples` helper. Update slice view iterators to consistently yield `(position, value)` tuples and use a temporary buffer with explicit permutation mapping for sorted access. Delegate manual slot sorting and column iteration to the new shared helper, eliminating per-format duplication and eager collection while preserving existing public API signatures.
2026-08-20 14:20:24 +02:00
Eric Coissac 82374deca5 refactor: simplify sub_matrix with iterator-driven column population
Refactored `sub_matrix` to delegate column population to `fill_sub_matrix`, replacing manual buffer allocation and explicit permutation loops with direct value assignment via `enumerate_slots_values`. This eliminates intermediate allocations, reduces pipeline overhead, and simplifies control flow while preserving slot ordering and permutation semantics.
2026-08-20 14:17:05 +02:00
Eric Coissac ac38aa759b refactor: rename batch retrieval methods and update common_traits
Renames `get_batch`, `fill_batch`, and `fill_batch_sorted` to `collect_slots_values`, `fill_slots_values`, and `fill_slots_values_sorted` to align with updated `common_traits 0.13` APIs. Introduces optimized batch retrieval that sorts input indices for sequential mmap access before reordering outputs to match the original query order. Updates test suites to reflect the new method signatures without altering validation logic or coverage.
2026-08-20 14:15:28 +02:00
Eric Coissac 82ddeaddcd Introduce unified nonzero_iter API across matrix types
Replaces nested column-major point lookups with a batched iterator that delegates to format-native traversal strategies. The implementation enforces a single pass per matrix type, using row-major iteration for sparse formats and eager collection for packed/columnar layouts while preserving original slot ordering. Memory allocation is optimized by removing `n_cols`-wide buffers in favor of per-row buffering or lazy iteration. Correctness tests verify iterator output against dense baselines across all supported layouts, and architecture documentation is updated to reflect the new format-agnostic query pattern.
2026-08-20 14:07:09 +02:00
Eric Coissac 5a9d903e51 chore: update dependencies and adapt to updated crate APIs
Bumps core dependencies including ndarray, rand, hashbrown, niffler, ureq, sysinfo, indicatif, lru, and remove_dir_all. Adapts source code to accommodate breaking changes by migrating RNG initialization, adjusting HTTP response handling, and replacing the fs4 crate with standard library file locking. Adds a planning document for query benchmarking and sparse index regression tests.
2026-08-20 13:45:41 +02:00
Eric Coissac 32bcbd1465 chore(deps): update dependencies and clean up imports
Updated core libraries and ecosystem packages, including the full serde suite and regex dependencies. Restructured syn dependency resolution by pinning 2.0.117 while introducing a standalone 3.0.3 entry for derive macros. Downgraded windows-sys to 0.59.0 and replaced the anes dependency in criterion2 with bpaf and walkdir. Removed an unused IntSliceView import from test modules.
2026-08-20 13:25:35 +02:00
Eric Coissac dc3d82f8db Add resume API to persistent matrix builders for incremental appending
Introduce a `resume` method for persistent bit and int matrix builders to restore dimensions from persisted metadata. This enables incremental column appending across separate build sessions without requiring manual dimension tracking. Consolidate builder lifecycle management in the merge layer using a unified `MatrixBuilder` enum, simplify closure logic, and add instrumentation. Include tests verifying state preservation and data integrity across multiple resume cycles.
2026-08-20 12:36:47 +02:00
Eric Coissac 70b0527957 Add diagnostic logging and minor internal refactoring
Introduce runtime dimension tracking, matrix shape validation, and per-layer diagnostics across multiple modules to improve execution observability. Extract intermediate computation results into local variables, standardize collection patterns in partial methods, and replace naive iteration with optimized pairwise counting. All modifications are strictly additive or structural; public APIs, data models, and core logic remain unchanged.
2026-08-17 11:29:31 +02:00
Eric Coissac 11476bc557 Add sparse bit matrix support and enhance dimension logging
Introduce a `Sparse` variant for `PersistentBitMatrix` with full method dispatch and implementations of `ColumnWeights` and `BitPartials` traits. Refactor distance matrix computations in layered stores to eagerly collect results before reduction, enabling reliable dimension tracking. Add comprehensive debug logging across phylo commands and distance modules to report matrix shapes and flag out-of-bounds indices during CSV iteration.
2026-08-17 11:24:23 +02:00
Eric Coissac 7bac0f3850 large refactoring 2026-08-17 09:28:53 +02:00
Eric Coissac 3ba26b3dc1 feat: add sparse on-disk format for presence matrices
The index packing API now accepts a `sparse` parameter to generate `PersistentSparseBitMatrix` files alongside existing dense matrices. The sibling cache automatically detects this format via an `is_multi.prsb` marker file and routes queries identically to the dense variant. A new `--sparse` CLI flag exposes the option, with tests verifying end-to-end pipeline correctness and storage equivalence.
2026-08-16 21:51:02 +02:00
Eric Coissac 50f4820cb9 feat(obicompactvec): introduce sparse bit matrix with supporting primitives
Implements a compact, row-major sparse bit matrix backed by memory-mapped components, introducing EliasFano, PersistentFixedIntVec, and PersistentRankSelectBitVec primitives for efficient storage and decoding. Adds a BinaryMatrix trait to unify row-level operations across dense and sparse implementations. Corrects edge-case behaviors for zero-width bit storage and cardinality-0 rows. Delivers reduced on-disk size and faster random row access, with column reads remaining dense-only. Test suites and benchmarks are included but currently marked as ignored.
2026-08-16 21:43:09 +02:00
Eric Coissac 151493526c perf: add #[inline] attributes to obicompactvec methods
Adds compiler inlining hints to accessors, iterators, bitwise operations, and distance functions across multiple modules. This optimization aims to reduce call overhead for frequently invoked methods without modifying runtime behavior, API contracts, or data models.
2026-08-16 21:18:58 +02:00
Eric Coissac 693c18bfa7 introduce fast mode for optimized sibling presence checks
Centralize the layer count validation into PartitionCache and track it via a new fast_mode flag. Extend query tuples to include a pre-resolved destination layer index, enabling a fast-path batch lookup that bypasses per-layer probing when enabled. Refactor neighbor iteration and hit resolution to eliminate duplication and conditionally dispatch to the optimized path based on the cache state.
2026-08-16 21:12:05 +02:00
Eric Coissac fecfe84ea6 Move sibling annex implementation to obikphylo siblings module
Relocates `SiblingAnnex`, `FamilyMask`, and `SiblingAnnexBuilder` from `obicompactvec` to the local `obikphylo::siblings` module. Replaces the previous implementation with a memory-mapped version using `memmap2`, featuring a 2-byte-per-slot layout, explicit bitfield manipulation, and support for concurrent atomic writes. Updates all sibling module imports to local paths and adds the `memmap2` dependency to `obikphylo`.
2026-08-16 21:04:14 +02:00
Eric Coissac 32d6720f50 Fix batch enumeration offsets and refactor sibling annex construction
Shifts sibling annex construction from slot-indexed enumeration to iteration-order traversal by correcting cumulative k-mer offset tracking in batch enumeration. Replaces coarse per-partition parallelism with chunked work distribution to prevent thread starvation on skewed partitions. Decouples custom progress messages from ETA updates to eliminate display clobbering during high-frequency callbacks. Adds regression tests validating batch offset correctness, partial batch handling, and iterator-order consistency across layer builds.
2026-08-16 20:51:19 +02:00
Eric Coissac b1f54b7d2f Add cache-optimized batch retrieval and sub-matrix methods
Introduces batch retrieval and sub-matrix extraction methods across vector, view, reader, and matrix types. These implementations optimize cache locality by sorting requested indices for sequential memory access before applying an inverse permutation to restore original order. Includes allocation-free variants that populate caller-provided buffers. Updates architecture documentation to define sibling annex persistence in iteration order and clarify pipeline separation.
2026-08-16 14:01:35 +02:00
Eric Coissac 0de078fdf1 Store minorant flag in family mask to avoid costly k-mer reconstruction
Transition the minorant flag from a derived value to a stored field within the family mask, resolving a performance regression where on-the-fly reconstruction consumed significant query time. This change introduces O(1) k-mer reconstruction APIs, shifts minorant computation to the index build phase, and enables direct annex-based statistics. Supporting updates include adopting shared ownership for partition caches and refactoring batch processing pipelines.
2026-08-16 13:50:44 +02:00
Eric Coissac 79346c0c86 Add modular data structures, parallel pipelines, and system profiling
Establishes foundational infrastructure across multiple crates by introducing unified persistent bit matrix storage with columnar, packed, and implicit variants, alongside De Bruijn graph node encoding and unitig iteration logic. Adds a macro-driven parallel pipeline scheduler featuring NUMA-aware runners, bounded channels, and memory budgets to enforce concurrency limits. Implements streaming nucleotide parsers with pooled page buffers for FASTA, FASTQ, and Genbank formats, complemented by system resource monitoring, progress tracking, and stage profiling utilities. Collectively, these changes provide the core data models, execution frameworks, and I/O pipelines required for downstream k-mer indexing and analysis workloads.
2026-08-14 14:20:08 +02:00
Eric Coissac 1a470eab9e Refactor k-mer sibling tracking to compact bitmask and on-demand counts
Replaces the explicit `SiblingInfo` struct and 3-bit minorant flags with a derived 4-bit presence mask (`FamilyMask`) that tracks observed bases per family. This eliminates redundant file I/O overhead by introducing a `PartitionCache` for batch lookups, simplifies serialization, and updates all downstream builders, stats computation, and tests to operate on the new bitmask representation. Adjusts CLI output to report deduplicated family sizes instead of histograms, ignores generated CSV files, and updates documentation to reflect the fixed canonical reference and new theory.
2026-08-10 17:53:52 +02:00
Eric Coissac ea914bb536 feat: implement per-k-mer sibling counts and central neighbor generation
Introduce the siblingannex module in obicompactvec to store per-slot minorant flags and sibling counts in a memory-mapped annex file. Add a scatter-gather pipeline in obikindex to compute these values across index layers and write them to .psib files. Implement central_canonical_neighbors in obikseq for generating strand-aware k-mer variants around the middle base. Expose rolling statistics in obiskbuilder and update dependency graphs accordingly.
2026-08-10 15:01:59 +02:00
Eric Coissac 2610a4af79 feat: add Mash distance metric and rolling entropy support
Implement the Mash distance metric across the CLI, index, and compact vector traits. This includes adding a `Mash` variant to the `DistanceMetric` enum and `MetricArg` CLI argument, implementing the conversion from Jaccard distances using the standard mutation-rate estimator formula, and updating documentation with supported metrics and algorithmic references. Additionally, add an `entropy` method to rolling statistics for computing order-specific entropy.
2026-07-09 11:40:48 +02:00
Eric Coissac 040eff140c refactor(query): optimize mmap locality with column-major matrix fetch
Refactor the query pipeline into a two-stage MPHF hit-detection pass followed by a column-major matrix fetch to improve cache efficiency. Introduce a QueryHit enum for event-driven callbacks, decoupling hit detection from data population. Add scan/fetch metrics to QueryStats, update Phase 4 architecture docs, and align tests with the new callback signature.
2026-07-07 18:44:09 +02:00
Eric Coissac cd2f2f9417 fix: validate packed matrix columns before repacking
Release / create-release (push) Successful in 2m27s
Release / build-linux-x86_64 (push) Successful in 8m15s
Release / build-macos-arm64 (push) Failing after 30s
CI / build (pull_request) Successful in 3m17s
Add header parsing helpers to extract column counts without memory mapping. Update packing functions to verify existing files match current metadata, preventing stale or widened-column artifacts. Extract inline tests in obilayeredmap to an external module and add comprehensive aggregation tests. Bump obikmer to 1.1.35 and clean up repository configuration.
2026-07-03 17:20:22 +02:00
Eric Coissac 2b37e8aac4 fix(bitmatrix): explicitly compute diagonal entries for self-similarity
Release / create-release (push) Successful in 2m26s
Release / build-linux-x86_64 (push) Successful in 8m13s
Release / build-macos-arm64 (push) Failing after 30s
CI / build (pull_request) Successful in 3m21s
The pairwise matrix functions now explicitly calculate and overwrite diagonal entries using `f(i,i)`, replacing previous implicit symmetric mirroring or default values. Documentation has been updated to clarify that diagonals represent self-comparison weights, ensuring accurate self-similarity calculations. Additionally, the obikmer crate version has been bumped to 1.1.34.
2026-07-03 13:04:40 +02:00
Eric Coissac c694e1f2b0 feat: add benchmark pipeline, expose APIs, and enforce strict paths
Introduces a Make-based orchestration for simulating, indexing, merging, filtering, and verifying k-mer counts and presence. Exposes internal builder and iterator APIs publicly, enforces mandatory leading slashes for predicate patterns, registers the `obitaxonomy` crate, and updates tooling configurations alongside documentation.
2026-06-22 10:18:33 +02:00
Eric Coissac 280ca1f5a3 feat: add optimized new_ones constructor for all-ones bit vectors
Introduces `new_ones` and `add_col_ones` methods to directly initialize all-ones bit vectors and matrix columns. This replaces redundant initialization sequences that created zero-filled structures and applied bitwise NOT, with a single pass that writes contiguous 0xFF bytes to disk. The change eliminates inversion overhead, streamlines test setup, and improves performance for filter mask intersection logic while preserving identical semantics.
2026-06-22 10:00:01 +02:00
Eric Coissac 9abb2db92f refactor: replace explicit bit-setting loops with optimized bulk operations
Refactor bitmatrix, colgroup, and layer modules to replace manual iteration with concise `or_where` predicates and bulk inversion calls. This simplifies the codebase and leverages optimized internal implementations for improved performance.
2026-06-22 09:56:41 +02:00
Eric Coissac 7c1efa9cbb feat: add vectorized column filters and optimize partitioner iteration
Adds `FilterMask` and conditional bitwise methods (`*_where`) to `obicompactvec` for composable column-based slot filtering. Extends `obikpartitionner` with a `MatrixGroupOps` trait and `column_mask_expr` method to express aggregate constraints as vectorized masks. Refactors matrix builder management into a unified `Builders` enum and introduces `try_compute_combined_mask`, enabling O(1) slot checks and skipping unnecessary row reads during partitioning and rebuilding passes.
2026-06-22 09:54:51 +02:00
Eric Coissac 4c4524766c feat(matrix): add partial group reductions and column persistence
Expands MatrixGroupOps with partial_group_min/max helpers for bitwise reductions and introduces add_col_from methods to persist external vectors as matrix columns. Refactors column aggregation in the partitioner to leverage these group operations directly, replacing iterative row processing with simplified builder lifecycle management and explicit metadata serialization.
2026-06-22 09:49:04 +02:00
Eric Coissac f91c5a3f79 refactor(obicompactvec): unify bit and int vector slice views
Refactors column and matrix access to use unified `BitSliceView` and `IntSliceView` abstractions, replacing legacy `PackedCol`/`IntColView` types. Introduces `BitSlice`/`IntSlice` traits for zero-copy, trait-based bitwise and arithmetic operations across persistent and temporary vector types. Removes deprecated in-memory `MemoryBitVec` and `MemoryIntVec` implementations and their tests, while updating dependent crates to use the new view-based API and `BitSliceMut` trait.
2026-06-17 23:51:32 +02:00
Eric Coissac fb4962c4fe refactor: replace in-memory vectors with temp-file-backed storage
Introduces `TempCompactIntVec` and `TempBitVec` as temporary, file-backed intermediates to replace eager in-memory vectors, enabling OS-level paging under memory pressure. Updates the `MatrixGroupOps` trait to return `io::Result` types, allowing proper error propagation and supporting chunked accumulation for large column groups. Includes builder patterns with `.freeze()` finalization, automatic `TempDir` cleanup on drop, and necessary test updates to handle the new fallible signatures. Also fixes `Cargo.toml` section ordering.
2026-06-17 23:36:15 +02:00
Eric Coissac 1d38d87ff9 Add column group operations and mask_with trait
Introduce the `ColGroup` struct and `MatrixGroupOps` trait to manage named subsets of column indices and perform additive aggregations (count, sum, any). Implement these operations for `PersistentBitMatrix` and `PersistentCompactIntMatrix`, applying size-optimized branches for presence counts and direct accumulation for small groups. Additionally, add a `mask_with` trait method that efficiently zero-sets elements based on a mask, optimized for sparse masks with O(n_zeros) complexity. Include comprehensive tests covering overflow handling, slot masking, and result additivity across partitioned data.
2026-06-17 23:28:52 +02:00
Eric Coissac 93559c3294 feat: introduce unified column view types for bit and int matrices
This commit introduces `BitColView` and `IntColView` to abstract over Columnar and Packed storage formats, implementing `BitSlice` and `IntSlice` for uniform column access. It adds `col_view()` accessors to `PersistentBitMatrix` and `PackedCompactIntMatrix`, explicitly panicking on implicit variants. The new types are publicly re-exported, and unit tests are added to validate per-element retrieval, aggregation methods, and parity with the original columnar representation.
2026-06-17 23:24:11 +02:00
Eric Coissac 7ed7b26039 perf: optimize vec arithmetic and add overflow tests
Refactor `cmp_scalar`, `min`, `max`, `add`, and `diff` to operate directly on the primary byte array, deferring overflow slot resolution to a secondary pass. This eliminates HashMap lookups in the hot path and enables SIMD vectorization. Add six unit tests to validate correct promotion and demotion between storage slots when values cross the 255 threshold.
2026-06-17 23:18:18 +02:00
Eric Coissac 26de90f18d feat: add iteration and aggregation to compact int vec
Implemented `sum()`, `count_nonzero()`, and `iter()` to complete the numeric vector interface. The builder now computes aggregate values across memory-mapped regions and overflow entries, while the reader delegates these operations to its inherent methods. The iterator provides zero-copy access to underlying `u32` elements.
2026-06-17 23:15:56 +02:00
Eric Coissac 497d250d8a refactor: replace byte-level bit iteration with 64-bit words
Refactor `BitIter` to process `u64` chunks using word-aligned shifts instead of byte-level operations. Introduce a dedicated `MemoryBitIter` for `MemoryBitVec`, updating its `iter()` and `IntoIterator` implementations accordingly. Hide `MemoryBitIter` from the public API to narrow the crate's interface, while leveraging explicit alignment guarantees for safer and more efficient bit extraction.
2026-06-17 23:11:12 +02:00
Eric Coissac aa98e82875 refactor: introduce PackedIntCol view and use iterators
Centralizes overflow handling and improves modularity by replacing manual mmap indexing and row loops with composable iterator patterns. This change leverages Rust's iterator traits for efficient, idiomatic column traversal while encapsulating data access in a dedicated view struct.
2026-06-17 23:09:18 +02:00