Push zunrplorkwkt #70

2026-08-28T23:14:56Z

coissac commented

2026-08-28 23:14:56 +00:00

No description provided.

coissac added 93 commits 2026-08-28 23:14:57 +00:00

chore(deps): bump obikmer dependency to 1.2.2 308d2b9f92

Update Cargo.lock to track the patch release of obikmer. This modification only adjusts the version in the lockfile and introduces no behavioral or API changes.

Add resume API to persistent matrix builders for incremental appending dc3d82f8db

Introduce a `resume` method for persistent bit and int matrix builders to restore dimensions from persisted metadata. This enables incremental column appending across separate build sessions without requiring manual dimension tracking. Consolidate builder lifecycle management in the merge layer using a unified `MatrixBuilder` enum, simplify closure logic, and add instrumentation. Include tests verifying state preservation and data integrity across multiple resume cycles.

Consolidate compare_sparse as example and clean up project artifacts dbd8af376c

Restructure the project by moving the standalone compare_sparse utility into an example directory, removing Sankoff parameter configurations and benchmark scripts, updating version control ignores, and expanding the test suite with diagnostic checks and performance benchmarks.

chore: remove src/profile.json.gz 2dba217482

Removes the compressed JSON archive containing profile configuration and data model definitions. Build processes and runtime loaders referencing this path will require updates to prevent missing file errors.

chore: add editor and memory directory patterns to .gitignore 69747dcb53

Remove precomputed data assets, configurations, and documentation 3b65319529

This change removes precomputed model parameters, phylogenetic tree datasets, k-mer spectrum data, and compressed profile archives. It also deletes runtime logs and detailed pipeline documentation. The repository ignore list is updated to exclude the sandbox directory.

refine k-mer index architecture documentation and remove obsolete spec 0da725ffe9

Introduces raw mapping and iteration APIs that bypass membership checks, clarifies variant-specific storage layouts and auto-detection logic, and documents optimized batch access patterns with caller-provided buffers. Removes the outdated obicompactvector_reflexion.md specification to consolidate architectural details into current implementation docs.

chore(deps): update dependencies and clean up imports 32bcbd1465

Updated core libraries and ecosystem packages, including the full serde suite and regex dependencies. Restructured syn dependency resolution by pinning 2.0.117 while introducing a standalone 3.0.3 entry for derive macros. Downgraded windows-sys to 0.59.0 and replaced the anes dependency in criterion2 with bpaf and walkdir. Removed an unused IntSliceView import from test modules.

chore: update dependencies and adapt to updated crate APIs 5a9d903e51

Bumps core dependencies including ndarray, rand, hashbrown, niffler, ureq, sysinfo, indicatif, lru, and remove_dir_all. Adapts source code to accommodate breaking changes by migrating RNG initialization, adjusting HTTP response handling, and replacing the fs4 crate with standard library file locking. Adds a planning document for query benchmarking and sparse index regression tests.

Add benchmark pipeline for dense and sparse query testing 89ea077456

Introduces a complete query benchmark track to evaluate performance and verify consistency between dense and sparse index formats. Adds scripts to simulate fixed-size paired-end reads, pack a sparse presence index, execute queries in both modes, and capture wall time and RSS metrics. Includes a verification step that compares outputs by read ID to ensure content identity across parallel processing. Updates build configuration, documentation, and ignore patterns to support the new pipeline for two microbial specimens.

add some doc about optimisation for query a4eb20e67e

Introduce unified nonzero_iter API across matrix types 82ddeaddcd

Replaces nested column-major point lookups with a batched iterator that delegates to format-native traversal strategies. The implementation enforces a single pass per matrix type, using row-major iteration for sparse formats and eager collection for packed/columnar layouts while preserving original slot ordering. Memory allocation is optimized by removing `n_cols`-wide buffers in favor of per-row buffering or lazy iteration. Correctness tests verify iterator output against dense baselines across all supported layouts, and architecture documentation is updated to reflect the new format-agnostic query pattern.

refactor: rename batch retrieval methods and update common_traits ac38aa759b

Renames `get_batch`, `fill_batch`, and `fill_batch_sorted` to `collect_slots_values`, `fill_slots_values`, and `fill_slots_values_sorted` to align with updated `common_traits 0.13` APIs. Introduces optimized batch retrieval that sorts input indices for sequential mmap access before reordering outputs to match the original query order. Updates test suites to reflect the new method signatures without altering validation logic or coverage.

refactor: simplify sub_matrix with iterator-driven column population 82374deca5

Refactored `sub_matrix` to delegate column population to `fill_sub_matrix`, replacing manual buffer allocation and explicit permutation loops with direct value assignment via `enumerate_slots_values`. This eliminates intermediate allocations, reduces pipeline overhead, and simplifies control flow while preserving slot ordering and permutation semantics.

refactor: centralize nonzero slot iteration logic across views 19f9954050

Extract duplicated traversal logic into a shared `NonzeroSlotsView` trait and `nonzero_triples` helper. Update slice view iterators to consistently yield `(position, value)` tuples and use a temporary buffer with explicit permutation mapping for sorted access. Delegate manual slot sorting and column iteration to the new shared helper, eliminating per-format duplication and eager collection while preserving existing public API signatures.

refactor: delegate distance computations and simplify matrix logic 9abee87af3

Consolidate bit matrix operations by delegating pairwise metric calculations to IntSliceView and col_view implementations. Introduce chunked_presence_count in colgroup to handle threshold-based accumulation efficiently. Add a direct point lookup method to the sparse matrix representation to eliminate buffer allocations, and simplify match arms in persistent accessors accordingly. All public APIs and behavioral contracts remain unchanged.

Refactor bit matrix operations and optimize distance computations c2e0533fa9

Consolidate row extraction and parallel column reduction into reusable generic utilities within the pairwise module. Replace manual iteration in builder and view methods with iterator-based zipped loops and deferred overflow processing to improve memory access patterns. Optimize distance functions by switching to bulk byte passes for branch-free SIMD vectorization, adding a merge-like helper to correct masked overflow values without secondary allocations. Add a comprehensive test validating the optimized distance paths against naive references across multiple thresholds and overflow scenarios.

Optimize sparse bit matrix packing to avoid intermediate copies 640f29725b

Refactor `pack_sparse_bit_matrix` to conditionally open either a packed or columnar dense representation based on file existence. This eliminates the previous unconditional pre-conversion step that forced a full packed copy. Update cleanup logic and add tests to verify direct transposition, proper file handling, and idempotency for both formats.

centralize layer path construction and data loading 5a91817488

Replace manual directory formatting and inline matrix opening with the newly introduced `layer_dir` and `open_data` helpers from the `obilayeredmap` crate. This standardizes filesystem access, encapsulates layer naming conventions, and simplifies error handling without altering computational behavior or test suites.

Replace manual layer path construction with obilayeredmap::layer_dir 7eaa8c2016

Standardizes layer directory path resolution across obikindex, obikpartitionner, and obikphylo by replacing inline string formatting and join calls with the centralized obilayeredmap::layer_dir utility. This refactoring removes redundant path construction logic while preserving existing iteration bounds, control flow, and public APIs.

Centralize partition directory resolution and add caching design spec f7ebc7a1ab

Introduce a design specification outlining performance bottlenecks in layer data access and an agreed-upon implementation direction for caching. Refactor the codebase to centralize index and layer directory path resolution within the partition object, replacing manual string joining and external helper functions with dedicated accessor methods.

Centralize partition metadata access and add layer introspection APIs 76cbd3a886

Replaced scattered direct metadata loading with centralized instance methods on `KmerPartition` to guarantee consistent error mapping and legacy recovery. Introduced `StorageKind`, `LayerContent`, and `EvidenceKind` enums alongside lightweight disk-probe methods that inspect file presence without opening heavy data structures. Updated callers across the index, partitioner, and phylo modules to use the new partition API, and added unit tests validating the introspection behavior.

rename KmerPartition to KmerPartitions and update Mat enum f9ef6b8391

Rename the KmerPartition type to KmerPartitions across obikindex, obikpartitionner, and obikphylo/siblings to reflect an updated data model. Update the Mat enum in siblings/cache.rs to add a SparsePresence variant and simplify opening logic by delegating sparse versus dense detection to PersistentBitMatrix. Apply consistent code formatting, import reordering, and multi-line refactoring throughout the affected modules.

refactor: rename obikpartitionner crate to obikpartition 9379bbaad8

Update Cargo manifests, dependency paths, and Rust imports across the workspace to reflect the `obikpartitionner` to `obikpartition` rename. Synchronize architecture and implementation documentation with the new module structure. Fix minor syntax issues in test assertions to ensure compilation compatibility. No behavioral or API changes are introduced.

chore: make COUNTS_DIR constant pub(crate) f64ac14f0e

Adjusts the visibility of the COUNTS_DIR constant to pub(crate), enabling internal crate-wide access without altering runtime behavior or data structures.

Rename obikpartition crate to obikpartitionner b5ec0122d0

Update all dependent crates and internal imports to reference the renamed module. Expose configuration parameters and genome metadata through new public accessor methods in the index API, and update documentation comments to align with the new package name.

refactor: merge KmerPartitions into KmerIndex and rename obikpartition 6c860f120f

Consolidates partition logic, metadata storage, and layer management directly into KmerIndex. Renames obikpartitionner to obikpartition, retaining only PartitionRouter for superkmer routing. Removes intermediate .partition() accessors in favor of direct methods on the index and updates PartitionCache::build to accept &KmerIndex directly. Derives n_partitions from config.n_bits and consolidates k-mer/minimizer sizes into IndexMeta.config. Fixes a regression where PartitionRouter::open incorrectly defaulted to closed.

Rename Layer<D> to TypedLayer and introduce heterogeneous Layer enum 1c54e60c9a

Renames `Layer<D>` to `TypedLayer<D>` to establish a distinct typed abstraction. Introduces a new heterogeneous `Layer` enum that unifies count and presence storage with runtime dispatch, delegating operations to the underlying typed variants. Updates cache, index, and phylo consumers to align with the renamed type and new extension traits, preserving full test suite stability while preparing the foundation for multi-partition caching.

Introduce lifecycle-aware Layer::Empty variant and update callers c4b69e1af5

The `Layer` enum is transformed into a lifecycle-aware state machine with an `Empty` variant representing an unconstructed directory. Read and query operations now explicitly panic when invoked on this state, enforcing explicit progression through `create()` before use. Iterator methods are updated to handle the new variant exhaustively, and module visibility constants are adjusted to support the refactored structure.

feat: introduce obikpartition crate for kmer partition layer lookup 164e879585

Introduces the `obikpartition` crate containing the `KmerPartition` struct to manage open layers in sequential order. The `open` constructor eagerly initializes layers under a specified directory, while the `find` method returns the index of the first layer containing a given k-mer. Dependencies are strictly scoped to `obilayeredmap` and `obikseq`, with unit tests validating layer ingestion and single-kmer lookup behavior. Batch lookups and further migration are deferred to a subsequent phase.

Refactor indexing pipeline and partition path resolution 616cb76af3

Invert dependencies by moving partition path primitives to a dedicated module and updating the index crate accordingly. Reshape the partition router to accept a mutable index reference, enabling chainable configuration and resolving lifetime issues with explicit drops. Shift orchestration logic from the index crate to the CLI, replacing monolithic scatter calls with discrete dereplication and counting steps. Introduce a generic progress callback API to decouple rate calculation from UI rendering, and correct file I/O paths to route layer-0 artifacts under the partition index directory.

Extract dereplication logic into obikderep crate and centralize paths abc51c2add

Move partition dereplication logic into a dedicated `obikderep` crate that implements a two-phase hash-split/merge strategy with Rayon for parallel processing. Centralize superkmer file path construction in `obilayeredmap` and update dependent crates to use the new pipeline and shared path helpers. Adjust test suites to explicitly invoke the dereplication step.

feat: enable index resumption and enforce directory creation 5c1584967f

The command now supports reopening existing indexes instead of failing when the output file exists. Control flow branches between opening an existing index and constructing a new one, moving configuration setup exclusively to the creation path. Directory existence is enforced upfront with proper I/O error propagation. The --force flag retains its original semantics by removing the target directory before proceeding with a fresh build.

refactor: rename obikindex to obikindexer 96b6517541

Extract k-mer counting logic into a dedicated counter module 878b63566f

Decouple k-mer counting from the partitioner by introducing a new `Counter` struct. The module exposes a fluent builder API with optional partial file retention, executes partition processing in parallel via Rayon with memory-aware chunk sizing, and integrates thread-safe progress callbacks. Update all callers to use the new counter, simplify test pipelines by removing serialization overhead, and clarify algorithm separation in module documentation.

chore: remove CLAUDE.md and update .gitignore 0299b608e2

Removes the file from version control and adds an ignore rule to prevent tracking.

Rename index methods to hash and expose kmer iteration APIs 346095b9eb

The layer API has been updated to rename `index` and `index_batch` to `hash` and `hash_batch`, aligning with raw MPHF slot hashing behavior. New public methods have been added to directly expose kmer hashing and iteration, enforcing strict state validation on empty layers. All other modifications are consistent code formatting adjustments without functional changes.

Extract indexing stage into LayerBuilder with NUMA-aware scheduling 4b7b3c3c1a

Introduce a dedicated LayerBuilder struct to orchestrate parallel layer 0 construction across partitions. The fluent API supports configurable abundance thresholds and intermediate artifact retention. Parallel execution is delegated to a NUMA-aware PartitionRunner scheduler, while progress reporting and stage timing are shifted to the command layer. A new mark_indexed method cleanly separates state tracking from orchestration by generating a completion sentinel.

Rename Layer to KmerLayer across obikindex and obikphylo 31bb324752

This commit renames the `Layer` type to `KmerLayer` throughout the `obikindex` and `obikphylo` crates. All imports, struct fields, function signatures, pattern matches, and instantiation calls have been updated accordingly. The change is a purely structural refactor that tightens type constraints without altering runtime behavior or data models.

refactor: extract index construction state tracking into extension trait da3aa5a2cb

Moves pipeline state bookkeeping, including sentinel file marking and spectrum persistence, into the algorithms' run and close methods. Introduces a crate-private extension trait to satisfy Rust's orphan rule while implementing construction-only operations on KmerIndex. Updates helper function visibility for cross-crate access and removes the now-empty index_layer module.

Split index lifecycle into IndexBuilder trait for maintenance operations 02dbdd11aa

Extracts directory bookkeeping and construction methods into a new IndexBuilder trait, separating maintenance operations from scientific computation logic. Migrates relevant methods to the new trait, updates module imports across obikindex, and narrows the crate's public API surface. Adds an end-to-end CLI smoke test to verify multi-index merge workflows.

refactor: migrate index metadata to on-disk JSON with fallible access 8d6ba6546b

Migrate index state tracking from filesystem sentinel files to an on-disk JSON schema within `index.meta`. The `IndexMeta` struct is now wrapped in an `Arc` with internal locking, exposing only fallible methods for genome and state access. In-memory mutation capabilities have been removed, requiring callers to handle I/O errors explicitly and pass immutable references to downstream components like `PartitionRouter`. Public sentinel constants have been removed from exports.

Introduce obikalgorithm crate to unify pipeline algorithms 00ba968628

Define a shared Algorithm trait with an associated Output type and a parameterless run(&mut self) method. Refactor PartitionRouter, Dereplicator, Counter, and LayerBuilder to implement the trait, standardizing receivers to &mut self and moving configuration and progress callbacks to dedicated builder setters. Decouple error handling using a generic boxed error type and update workspace dependencies accordingly.

Simplify MphfLayer::open by removing explicit IndexMode parameter c9d10d55c7

The MphfLayer::open signature now accepts only a directory path. Mode configuration is resolved internally via disk-based auto-detection using LayerEvidence, eliminating the need for explicit arguments from callers. All internal query methods and external call sites have been updated accordingly, while TypedLayer::open temporarily retains the unused parameter for future cleanup.

Extract index modules into specialized workspace subcrates fc4464a0ef

This commit partitions the obikindex crate into multiple focused subcrates (obikfilter, obikmerge, obikquery, obikrebuild, obikselect, obikstats, obikdump, and obikidxcache) to reduce coupling and clarify module boundaries. It standardizes error handling across the workspace using OKIError and OKIResult, updates index APIs to support lazy, disk-backed partition access, and migrates NUMA system utilities to a new obisys crate. All modifications are structural, focusing on dependency graph expansion, import path updates, and API surface reorganization without altering core runtime behavior.

refactor: shift indexing algorithms to layer-centric partition APIs 2419a6c21d

Replaces manual filesystem path handling and parallel iteration with dedicated internal utilities. Introduces `par_over_layer0` and `layer0()` to abstract partition layer access, enabling partition-driven initialization and centralized progress tracking. Removes disabled or internal methods (`rewrite_config`, `open_data`) no longer part of the active interface. Updates error propagation and metadata persistence to align with the new layer-centric workflow.

Add directory-level locking and introduce layered index cache 7183e3adb4

Introduces directory-level locking to prevent concurrent index operations from corrupting shared directories, along with explicit APIs for acquiring, probing, and releasing locks. Restructures the index cache crate to use a layered store architecture that eagerly initializes metadata and provides fast hierarchical lookups. Updates dependent modules, test suites, and CLI commands to align with the refactored API surface, and adds an end-to-end smoke test for validation.

feat: Add superkmer CLI subcommand to obikmer2 1261aeba86

Introduces a new `superkmer` command that implements a multi-stage pipeline for reading nucleotide pages and constructing superkmers with configurable parameters. The implementation uses partition-aware scatter writing via `obifastwrite` to distribute output batches across workers, and adds the necessary local dependencies to the project manifest.

Start of a huge refactoring 5048f43eea

Add unitig streaming iterators and estimate CLI subcommand fb31a35c76

Introduce `iter_unitigs` methods across the index cache, content layer, MphfLayer, and typed layer to stream whole reconstructed sequences directly from underlying storage without decomposing into k-mers. Add a corresponding low-level streaming iterator in obiskio for thread-safe, lazy reads from memory-mapped files. Include a new CLI subcommand to compute and display approximate false-positive rates based on provided indexing parameters.

feat: add annotate CLI command for applying genome metadata via CSV fba9c65b1a

Introduces the `annotate` subcommand to apply genome metadata from an external CSV file to a pre-built k-mer index. The command supports configurable field separators, ID columns, and null markers, while providing a `--dump` option to export current index metadata as sorted CSV. Supporting changes include minor internal refactoring in `obikindex` to use object-level directory accessors and updates to the `csv` dependency.

feat: add col_weights API and refactor obikstats to use Algorithm trait 23812d1af8

Added a `col_weights` method to index layers for computing per-genome column sums or presence k-mer counts. Refactored `obikstats` to implement the `Algorithm` trait with a two-phase `new`/`run` model, replacing manual layer resolution with `IndexCache` for eager file I/O. Simplified per-genome counting logic and updated public exports and dependencies accordingly.

Add sorted_slots parameter to nonzero_triples and batch build_from_dense bb380d0c7d

Extended `nonzero_triples` with a boolean flag to conditionally skip sorting when input slots are already ordered. Updated call sites to pass appropriate flags, enabling a single-pass columnar traversal in the sparse matrix builder. Refactored `build_from_dense` into a batched processing pipeline that reduces repeated file access and improves sequential read performance. Added a diagnostic example to validate the new builder against existing dense matrices.

Add configurable sparsity flag to index finalization and merge CLI cbf6893f38

Extends `KmerIndex::finalize_indexed` to accept a `sparse` boolean parameter, enabling control over whether output presence matrices are packed in a compact sparse format or stored densely. The merge command now exposes a `--dense` flag to invert this setting, defaulting to the existing sparse representation. Call sites in select and rebuild modules pass `false` to preserve current behavior. A benchmark lock file is also added for dependency tracking.

refactor(obikselect): implement Algorithm trait for selection 5dc16b0127

Shifts the selection logic from a direct method into a dedicated `Select` struct implementing the `Algorithm` trait. Introduces a two-phase initialization pattern with explicit builder methods and closure-based progress callbacks. Updates the crate's dependency graph to include `obikalgorithm`, refactors layer processing with unified matrix dispatch, and implements dual directory routing for partition execution.

Add select0 method and expand Elias-Fano query API ad0b7173a3

Update PersistentSparseBitMatrix documentation for clarity. Introduce select0 to the rank/select module, and add next_geq, contains, successor, and predecessor methods to the Elias-Fano implementation. Include naive reference implementations and unit tests to validate correctness and boundary conditions for all new APIs. All changes are strictly additive.

refactor: consolidate mmap file operations into shared helper functions 881b1532b5

Extracts file creation, opening, and memory mapping logic into a new `mmap_file` module. This replaces repetitive unsafe operations across persistent data structures with standardized helper functions, ensuring consistent header validation, error handling, and data layout while keeping public APIs unchanged.

refactor: extract partition iteration and unify group selection 16ade823d6

Move partition iteration logic to obikdump, introducing a FilteredPartitionIter trait over IndexCache for batch-oriented scanning with configurable data retrieval and early termination. Consolidate ingroup and outgroup index storage in GroupQuorumFilter into a unified Selection struct driven by predicate matching. Update dependency manifests to include obikidxcache, rayon, and obikentropy, and remove the deprecated dump_layer module while adjusting public API re-exports.

refactor: relocate partition iterator and expose graph builder API 294f132a0a

Move the partition iterator implementation from obikdump to obikfilter, updating imports and exposing the type publicly. Add obikidxcache as a local dependency for obikfilter. In obikindexer, expose the new build_layer_from_kmers function to enable shared graph-building logic across pipelines without code duplication.

feat: add sparse compact integer matrix implementation and utilities b7a8b5e6cf

Introduces `PersistentSparseCompactIntMatrix`, a row-major sparse integer matrix combining a bitmatrix support layer with dual value streams for singleton and multi partitions. Adds builder and packing utilities to convert dense count matrices into the new on-disk format, featuring idempotency guards and crash-safe persistence workflows. Exposes core accessors, iterators, and column weight calculations while adjusting internal visibility for companion modules. Includes unit tests and diagnostic examples to validate equivalence with dense representations and measure compaction metrics.

Add unified matrix builder abstraction and k-mer filtering CLI 354e6f9bf1

Introduces a unified MatrixBuilder abstraction for persistent bit and integer matrices, replacing custom dispatch enums and boolean flags across consumers. Refactors index, merge, and select layers to adopt explicit merge modes and centralized lifecycle management. Adds a complete k-mer filtering implementation with partition-level processing, progress tracking, and a new CLI subcommand supporting configurable predicates and thresholds.

Add obikselect crate for k-mer selection and aggregation 737966899f

Introduces a new `obikselect` module to handle k-mer selection and column aggregation. Adds a `select` CLI command in `obikmer2` that supports group predicates, aggregate operators, and output column filtering. Implements operator parsing with case-insensitive matching and default rules, along with a centralized module for resolving group specifications from metadata or genome filters. Updates dependencies to include the new crate and its prerequisites.

Add dump subcommand and introduce reusable group filtering arguments 768fa0183d

Introduces a new `dump` CLI command to export index k-mers as a CSV table. Adds a reusable `GroupFilterArgs` struct for ingroup/outgroup metadata-predicate quorum filtering with configurable count and fraction thresholds. Refactors the filter command to use explicit flags and a `GenomeSelector`-based pipeline, improving error handling and predicate application order. Exposes `IndexDump` from the `obikdump` crate and updates local dependencies accordingly.

Add unitig extraction command to obikdump and obikmer2 96dfb5300b

Introduces a new unitig extraction feature that reads k-mers from an index, filters them by group metadata, and partitions the computation using rayon. The implementation constructs per-partition de Bruijn graphs, merges them, computes node degrees, and writes the resulting sequences as FASTA. A corresponding CLI command is added to obikmer2 to expose this functionality with configurable filtering and progress reporting.

Implement pack CLI command for matrix packing and layer compaction 904d85f33b

Adds the `pack` subcommand to `obikmer2` for persisting presence and count matrices in sparse or dense formats. Introduces the `compact_layer` module in `obikrebuild` to merge multiple partition layers into a single layer in-place via atomic operations. Updates dependencies across `obikdump`, `obikmer2`, and `obikrebuild` to resolve new local crate references. Shifts the architecture from multi-layer accumulation to in-place compaction, removing legacy rebuild modules.

Add sparse storage variant to PersistentCompactIntMatrix b0890d1781

Introduce a new `Sparse` format alongside existing `Columnar` and `Packed` variants, enabling optimized row-major pairwise counting for distance and similarity metrics via the `CountPartials` trait. Update storage detection priorities, extend matrix dispatch logic to sparse backends, and correct diagonal/off-diagonal formulas in bit matrix partial computations. Expand layer APIs with format-agnostic `nonzero_iter`, update usage documentation for the `--sparse` flag, and add comprehensive tests verifying roundtrip integrity and metric equivalence against dense implementations.

refactor: unify iterator construction in nonzero_iter 10a0909687

Change the return type of `nonzero_iter` from an opaque `impl Iterator` to a `Box<dyn Iterator>`. This enables direct return coercion in the `Count` layer branch, removing an explicit `Box::new()` wrapper and aligning it with the existing `Presence` implementation. The modification shifts iterator construction to runtime dynamic dispatch while preserving the public API contract and iteration semantics.

refactor: Shift KmerIndex ownership to Arc for thread-safe sharing bd7729b095

Replaces lifetime-bound references with runtime reference counting across multiple crates. This enables safe concurrent access across parallel workers without explicit cloning or manual lifetime management. Introduces the `query` and `utils` CLI commands in obikmer2, along with supporting modules for batch processing, sparse indexing, sliding-window findere logic, and output formatting. Updates dependency manifests and aligns test suites with the new ownership model.

feat: add convert command for in-place index evidence modification 4ea3cd32ba

Introduce a new CLI command that modifies existing indices to support exact, approximate, or hybrid evidence modes. This change enables the reindex module in obikrebuild, exposing an IndexReindex trait for evidence persistence and layer processing. It also refactors obikindexer to publicly expose an IndexBuilder trait for post-build evidence construction, while adding a set_evidence API in obikindex to safely update configuration fields without altering core parameters.

refactor: restructure obikphylo module layout and adjust visibility 6bdc9354d3

Relocate cardcomp into the siblings directory and update internal import paths to reflect the new hierarchy. Adjust the crate's public API surface by updating module declarations and re-exports. This is a purely structural change with no modifications to logic, data models, or runtime behavior.

Rename PersistentCompactIntMatrix to PersistentIntMatrix 4b6005962e

Update all matrix type references, imports, and instantiations across
the codebase to use the new PersistentIntMatrix name. The change also
includes standardizing code formatting, such as multi-line statements
and import ordering, without altering any behavioral logic or public
API contracts.

feat: add phylogenetic analysis CLI command and lazy distance traits 635fc830d1

Introduce the `obikphylo` crate to support genome-vs-genome distance matrix computation and phylogenetic tree inference via Neighbor-Joining and UPGMA algorithms. Extend the `obikmer2` CLI with a new `phylo` command that exposes configurable metrics, presence thresholds, and CSV/Newick output options. Refactor `obikindex` to expose trait-based partial aggregation for efficient distance metric finalization without full matrix materialization. Update dependency graphs and remove obsolete storage modules.

refactor: decouple tree construction from Newick serialization 9dee6dcd08

Refactor phylogenetic algorithms (`neighbor_joining`, `upgma`) to return an explicit `Tree` struct instead of a serialized Newick string. This change makes serialization an explicit step for downstream consumers via the new public `Tree::to_newick()` method, decoupling tree construction from output formatting. The `siblings` module has also been moved to `siblings_old`.

Introduce concurrency-bounded parallel processing for index cache layers d084396aba

Implement an `obipipeline::Throttle` with an RAII guard to acquire and release concurrency slots. Expose new bounded parallel methods on `IndexCache` to process cached layers with a configurable cap. Refactor downstream aggregation logic to use single-pass parallel map-reduce instead of manual collect-map-reduce sequences, enforcing a maximum of 8 concurrent layer scans to bound memory usage.

Implement --sibling-annex flag and k-mer family annex builder 920120d20e

Introduces a new `--sibling-annex` CLI flag that triggers the construction of a sibling-count/minorant annex for multi-genome indices. The implementation adds a `FamilyMask` data structure with memory-mapped I/O, enabling lock-free concurrent updates via atomic bitwise operations. Batched processing improves cache locality and parallelism, while helper functions derive family presence and minorant flags dynamically. The feature is exposed through an `IndexCache` extension trait, protected by an exclusive directory lock to prevent index corruption during construction.

Rename SiblingAnnexBuildExt to SiblingExt and add sibling iterators bbb58a698f

Refactors the `SiblingAnnexBuildExt` trait to `SiblingExt` across all imports, implementations, and documentation. Introduces new modules for streaming and batched iteration over sibling layers, along with a two-phase pipeline for resolving minorant families.

Add per-family Shannon entropy calculation and reporting 5a0b71d105

Introduces an entropy.pent persisted format and algorithms to compute per-family Shannon entropy across genomic partitions. Refactors IndexCache to use Arc-based layer sharing for efficient iteration and restructures the siblings module with extension traits for annex generation. Adds a --shannon CLI flag to export partition-level entropy metrics to CSV.

feat: add phylogenetic sampling, masking, and plain FASTA writer beb2951c20

Introduces configurable phylogenetic sampling and masking controls via new CLI flags (`--subsample`, `--entropy`, `--exclude-genome`, etc.). Adds a complete SNP pseudo-alignment pipeline featuring entropy-biased Gaussian sampling, post-hoc state masking, and proportional per-layer filtering. Extends the FASTA writer with a `write_plain_record` API for bare-header output without JSON annotations.

Add sibling family size distribution statistics and CLI flags 89629e118d

Introduces `--sibling-stats` and `--sibling-hist` flags to compute and export family-size distributions from pre-built annexes. The new algorithms module implements parallel and sequential scanning routines, while the CLI layer handles CSV export with deduplication and standardized error handling.

Refactor sampling and reduction to use batched SurvivingFamily data e015362ce6

Replace per-family callbacks with a per-layer batch callback that accumulates surviving families into an Arc<Vec<SurvivingFamily>>. Update the alignment reduction routine to iterate over this batched structure, enabling zero-copy sharing for parallel operations and structuring the code for future extensibility.

Add single-pass pairwise reduction pipeline for SNP alignment 9043868228

Replaces the previous two-pass design with a single-pass reduction that computes per-genome-pair substitution counts, cardinality and base-composition tallies, and transition cost matrices. Introduces internal APIs to derive downstream n×n matrices from reduced data and applies configurable filtering based on SNP ratios and exclusion masks. Adds a fused pipeline entry point aggregating alignment and distance outputs for downstream phylogenetic commands. Adjusts visibility of existing reduction functions to restrict external access while permitting sibling module invocation.

feat: add Sankoff cost-matrix calibration CLI and serialization 1536217ac1

Introduce the --sankoff flag to generate subsampled calibration bundles and pairwise cost matrices. Expose internal sibling algorithm types as public to support external consumption. Implement YAML, CSV, and FASTA output serialization using serde, gated by a configurable substitution saturation ceiling and requiring the --subsample flag.

Add phylogenetic export support for TNT, PhyG, and IQ-TREE 0b40d2d0da

Extends the phylogenetic pipeline with exporters for TNT, PhyG, and IQ-TREE that generate executable scripts, cost matrices, and recoded alignments. Adds internal helpers for state indexing, floating-point matrix scaling, and Floyd-Warshall metric closure to satisfy external tool constraints. Adjusts visibility modifiers for sibling iterators and entropy annex structs, and updates design documentation for evolutionary distance metrics.

Implement SNP distance models with gamma correction and PHYLIP output 4f34a646c5

Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.

Add family overlap computation and CLI options for phylo analysis e101f629e6

Extends the entropy annex pipeline to optionally compute family overlap data in a single pass, introducing a memory-mapped binary format and an in-memory accumulator for efficient storage. Adds --min-shared-family to auto-exclude genomes below a threshold during SNP-family distance computation, and --family-overlap to export a shared family matrix as CSV. Centralizes upper-triangle indexing into a shared helper function to eliminate duplication.

Rename CLI to obikmer2, add phylo commands, and unify index caching 95fa0c93b2

Restructure the workspace and rename the CLI application to obikmer2. Replace direct KmerIndex usage across all commands with Arc-wrapped IndexCache to enable shared ownership. Introduce builder patterns for algorithmic operations and integrate explicit progress tracking. Add new Phylo, NameTree, and Convert subcommands with expanded CLI flags. Consolidate module structure, update dependency specifications, and remove legacy directories.

Update documentation for CLI command renames and new commands caf59b5658

Renames the reindex command to convert across all usage guides and navigation menus. Adds documentation for the newly introduced name-tree command. Updates flag references, such as replacing --metric with --distance in pack and phylo modules, and adjusts MkDocs configuration to match the revised structure. All changes are strictly limited to documentation and configuration files.

feat: add --force-copy flag to copy files instead of hard-linking 7363370065

Introduces the `--force-copy` CLI argument to bypass hard-linking during selection. When enabled, unchanged kmer-identity files are copied instead of hard-linked to ensure output independence from the source filesystem. The default behavior remains unchanged, continuing to use hard-links with automatic fallback to copying on failure.

Remove in-place flag, optimize presence counting, and handle NUMA panics 93fe838f97

The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.

implement hybrid hard-link copy strategy for directory bootstrap 6b0c0867cc

Introduces recursive directory replication functions that default to hard-linking for space efficiency, with automatic fallback to full file copying on cross-filesystem failures. Enforces mandatory real copies for presence and counts subdirectories to guarantee independent data matrices before merging, while preserving the existing separation between immutable index files and mutable matrix files.

docs: document partition runner deadlock fix and select CLI updates 579cfd8752

Update documentation to reflect architectural fixes in the NUMA partition runner and implementation changes in the merge and select modules. The partition runner now wraps task execution in `catch_unwind`, emitting panic events that count toward completion thresholds to prevent controller deadlocks, while re-raising the first payload on return. CLI documentation for select has been updated to mandate output directories, introduce default hard-linking with automatic fallback, and document presence aggregation optimizations. Known issues such as self-merge crashes and sparse-packed index gaps are now documented as failing fast due to these improvements.

feat: add batched int group stats API and expand benchmark variants 54e4600120

Introduces a `batch_int_group_stats` API for computing presence counts, sums, minimums, and maximums across sparse and dense matrix representations. The selection layer now utilizes this batched approach to optimize aggregation semantics for boolean and numeric operations. Additionally, reorganizes the benchmarking infrastructure to support querying across presence and count index variants in both dense and sparse formats, including new packing scripts and updated statistics aggregation.

refactor(benchmark): consolidate generated artifacts under run/ directory

ci.yml / build (pull_request) Successful in 3m49s

Details

52f26809af

Restructure the benchmark pipeline to direct all simulated data, indices, statistics, and query outputs into a unified `run/` directory. Update Makefile targets, shell scripts, and Python utilities to resolve paths relative to this new base. Adjust documentation and dependency tracking to match the revised layout, and remove outdated temporary artifacts.

coissac merged commit 7e28ca4853 into main

2026-08-28 23:15:38 +00:00

coissac deleted branch push-zunrplorkwkt

2026-08-28 23:15:39 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: OBIKmers/obikmer#70