Renames `Layer<D>` to `TypedLayer<D>` to establish a distinct typed abstraction. Introduces a new heterogeneous `Layer` enum that unifies count and presence storage with runtime dispatch, delegating operations to the underlying typed variants. Updates cache, index, and phylo consumers to align with the renamed type and new extension traits, preserving full test suite stability while preparing the foundation for multi-partition caching.
Consolidates partition logic, metadata storage, and layer management directly into KmerIndex. Renames obikpartitionner to obikpartition, retaining only PartitionRouter for superkmer routing. Removes intermediate .partition() accessors in favor of direct methods on the index and updates PartitionCache::build to accept &KmerIndex directly. Derives n_partitions from config.n_bits and consolidates k-mer/minimizer sizes into IndexMeta.config. Fixes a regression where PartitionRouter::open incorrectly defaulted to closed.
Update all dependent crates and internal imports to reference the renamed module. Expose configuration parameters and genome metadata through new public accessor methods in the index API, and update documentation comments to align with the new package name.
Update Cargo manifests, dependency paths, and Rust imports across the workspace to reflect the `obikpartitionner` to `obikpartition` rename. Synchronize architecture and implementation documentation with the new module structure. Fix minor syntax issues in test assertions to ensure compilation compatibility. No behavioral or API changes are introduced.
Rename the KmerPartition type to KmerPartitions across obikindex, obikpartitionner, and obikphylo/siblings to reflect an updated data model. Update the Mat enum in siblings/cache.rs to add a SparsePresence variant and simplify opening logic by delegating sparse versus dense detection to PersistentBitMatrix. Apply consistent code formatting, import reordering, and multi-line refactoring throughout the affected modules.
Replaced scattered direct metadata loading with centralized instance methods on `KmerPartition` to guarantee consistent error mapping and legacy recovery. Introduced `StorageKind`, `LayerContent`, and `EvidenceKind` enums alongside lightweight disk-probe methods that inspect file presence without opening heavy data structures. Updated callers across the index, partitioner, and phylo modules to use the new partition API, and added unit tests validating the introspection behavior.
Introduce a design specification outlining performance bottlenecks in layer data access and an agreed-upon implementation direction for caching. Refactor the codebase to centralize index and layer directory path resolution within the partition object, replacing manual string joining and external helper functions with dedicated accessor methods.
Standardizes layer directory path resolution across obikindex, obikpartitionner, and obikphylo by replacing inline string formatting and join calls with the centralized obilayeredmap::layer_dir utility. This refactoring removes redundant path construction logic while preserving existing iteration bounds, control flow, and public APIs.
Bumps core dependencies including ndarray, rand, hashbrown, niffler, ureq, sysinfo, indicatif, lru, and remove_dir_all. Adapts source code to accommodate breaking changes by migrating RNG initialization, adjusting HTTP response handling, and replacing the fs4 crate with standard library file locking. Adds a planning document for query benchmarking and sparse index regression tests.
Restructure the project by moving the standalone compare_sparse utility into an example directory, removing Sankoff parameter configurations and benchmark scripts, updating version control ignores, and expanding the test suite with diagnostic checks and performance benchmarks.
Explicitly sets `default-features = false` for the `obikindex` dependency to ensure no default features are activated during dependency resolution and compilation.
Adapts sibling k-mer tests to the updated canonical API by replacing string-based construction with direct byte conversion. Removes intermediate reverse complement steps and introduces a helper function to compute reverse complement strings directly from canonical k-mers, preserving existing test output while streamlining setup.
Adds a new binary that validates bit-level consistency between dense and sparse K-mer index representations by sampling slots across partitions and reporting discrepancies. Refactors sibling cache matrix initialization into a centralized factory method to simplify control flow and standardize error handling. Introduces diagnostic configurations, benchmark scripts, and an ignored test case to support k-mer resolution analysis.
Introduces --iqtree-min-freq (default 0.001) to treat low-frequency nucleotide states as missing data during IQ-TREE alignment generation when --free-loss is active. This triggers a recoding pass that folds rare states into the missing symbol, followed by non-informative site removal and alphabet recomputation to maintain output consistency. The change also adds Sankoff model configuration files and updates related tests and documentation.
Replaces premature exponentiation-based row normalization with log-sum-exp arithmetic to prevent tiny probabilities from collapsing to exactly zero. This eliminates spurious infinite costs for valid but rare transitions while preserving correct IEEE 754 semantics for genuinely unobserved pairs. Adds explicit guards against NaN in degenerate rows and includes a regression test verifying finite costs for probabilities as low as 1e-200.
Shift monomorphism filtering from the entropy scan layer to a lightweight, annex-only pre-pass. By replacing `Selection::All` with a pre-filtered subset, expensive per-genome resolution is strictly limited to non-monomorphic families. This avoids processing ~98% of minorants that are known to be monomorphic, while preserving positional read speedups for subsequent runs. The change is an internal performance refinement with no public API modifications.
The index packing API now accepts a `sparse` parameter to generate `PersistentSparseBitMatrix` files alongside existing dense matrices. The sibling cache automatically detects this format via an `is_multi.prsb` marker file and routes queries identically to the dense variant. A new `--sparse` CLI flag exposes the option, with tests verifying end-to-end pipeline correctness and storage equivalence.
Implements a compact, row-major sparse bit matrix backed by memory-mapped components, introducing EliasFano, PersistentFixedIntVec, and PersistentRankSelectBitVec primitives for efficient storage and decoding. Adds a BinaryMatrix trait to unify row-level operations across dense and sparse implementations. Corrects edge-case behaviors for zero-width bit storage and cardinality-0 rows. Delivers reduced on-disk size and faster random row access, with column reads remaining dense-only. Test suites and benchmarks are included but currently marked as ignored.
Introduces `--subsample N` and `--shannon` CLI flags to cap retained variable families via proportional reservoir sampling and compute per-family Shannon entropy. Updates the family scanning API to support explicit selection filtering with early-exit optimization, resolving an indexing drift issue in monomorphic layers. Streams entropy metrics for 15-state and 4-nucleotide spaces directly to CSV while maintaining parallel processing across sibling layers.
Centralize the layer count validation into PartitionCache and track it via a new fast_mode flag. Extend query tuples to include a pre-resolved destination layer index, enabling a fast-path batch lookup that bypasses per-layer probing when enabled. Refactor neighbor iteration and hit resolution to eliminate duplication and conditionally dispatch to the optimized path based on the cache state.
Shift the sibling annex construction pipeline from an in-memory atomic mask to a memory-mapped file backend. This enables lock-free concurrent writes directly into the mapped region, streamlining the two-phase write process to accumulate bits atomically before finalization. Adjusted the cache lookup to return the specific matching layer index rather than a boolean flag, and added tests to verify correct layer tracking and cross-partition resolution in merged indexes.
Relocates `SiblingAnnex`, `FamilyMask`, and `SiblingAnnexBuilder` from `obicompactvec` to the local `obikphylo::siblings` module. Replaces the previous implementation with a memory-mapped version using `memmap2`, featuring a 2-byte-per-slot layout, explicit bitfield manipulation, and support for concurrent atomic writes. Updates all sibling module imports to local paths and adds the `memmap2` dependency to `obikphylo`.
Instruments the phylo command pipeline with structured execution timing, wrapping major computational blocks with stage hooks and printing aggregated metrics upon completion. Additionally, parallelizes sibling counting logic using Rayon to process independent layer directories concurrently, preserving identical functionality and public API contracts.
Shifts sibling annex construction from slot-indexed enumeration to iteration-order traversal by correcting cumulative k-mer offset tracking in batch enumeration. Replaces coarse per-partition parallelism with chunked work distribution to prevent thread starvation on skewed partitions. Decouples custom progress messages from ETA updates to eliminate display clobbering during high-frequency callbacks. Adds regression tests validating batch offset correctness, partial batch handling, and iterator-order consistency across layer builds.
Relocate the `siblings` and `cardcomp` modules from `obikindex` to a dedicated `obikphylo` workspace member. Convert inherent methods on `KmerIndex` into extension traits, update import paths across `obikmer`, and add supporting accessor methods to `obikseq` and `obilayeredmap`. This restructuring reduces the public API surface of `obikindex` while organizing phylogenetic iteration, caching, and distance calculation logic under a dedicated crate.
Eric Coissac