Commit Graph
398 Commits
Author SHA1 Message Date
Eric Coissac f9edefd135 Refactor query system for performance and memory reduction
This change introduces dynamic chunk sizing based on input size estimates, ensuring memory constraints are respected while allowing for smaller chunks that yield significant speed improvements. Additionally, index structures were optimized, resulting in drastic memory footprint reduction and faster query execution times.
2026-10-15 12:21:38 +02:00
Eric Coissac 025c40c0db Refactor chunking logic to enforce 128 k-mer limit
This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
2026-10-15 12:20:36 +02:00
Eric Coissac 8468319a63 Refactor unitig detection and graph traversal logic
Introduce new hashing mechanisms and support for directional unitig checks. Fix fragmentation issues and improve performance metrics through optimized graph construction and new benchmarking utilities.
2026-10-15 12:19:24 +02:00
Eric Coissac d32d86193f Refactor index metadata loading to use buffered I/O
This change replaces unbuffered file reads with buffered I/O operations to improve performance during index metadata loading. This includes implementing new methods for reading index metadata files and wrapping file handles in `BufReader` to optimize subsequent reads.
2026-10-15 12:17:51 +02:00
Eric Coissac 093d81c359 Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
2026-10-15 12:17:09 +02:00
Eric Coissac 09c80468b2 Refactor k-mer storage to use flat arrays
This change replaces the HashMap-based k-mer storage mechanism with a compact, flat array structure for `PartitionKmers`. This significantly reduces memory usage per partition and improves query execution time.

It also updates the batch processing logic to handle the new data structure, introduces detailed memory footprint calculations, and updates relevant query layer APIs. Test coverage has been expanded to validate correct partitioning and k-mer ordering.
2026-10-15 12:16:21 +02:00
Eric Coissac fe38424682 Implement batch lookup and chunked data management
Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline.

This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
2026-10-15 12:14:51 +02:00
Eric Coissac 8b87078478 Implement MPHF migration tool and sparse matrix permutation logic
Refactors sparse matrix handling to implement row permutation logic and introduces a standalone tool for migrating indexes from MPHF 1.x to 2.x formats.
2026-10-15 12:12:51 +02:00
Eric Coissac d686d39506 Refactor indexing logic and update core dependencies
This commit introduces significant updates to the indexing system, including the introduction of zero-copy types, modifications to the `Mphf` type definition, and new hybrid search strategies. Core algorithms are updated to handle new hashing types and improved error handling during pointer hash construction. Dependency versions are also updated across the board, including new security features and system-level crates.
2026-10-15 12:07:08 +02:00
Eric Coissac 3bd7fd4f41 Implement memory mapping for file construction
Refactors the building process to utilize memory mapping (`memmap2`) for file creation and introduces the necessary infrastructure for publishing built files atomically. This includes expanding the public API surface and cleaning up deprecated architectural components.
2026-10-15 12:02:33 +02:00
Eric Coissac c226744405 Refactor sampling algorithms and coverage metrics
Optimize family scanning and sampling plan calculation logic. Introduce new per-genome richness metrics and redefine the sampling fraction $r$ for improved analysis.
2026-10-15 12:01:15 +02:00
Eric Coissac cb5fc202b6 Refactor sampling logic and introduce diagnostic outputs
This commit introduces new features for sampling, including detailed `SamplePlan` generation, Shannon entropy summaries, and site statistics reporting. It formalizes conditional distance matrix computation and clarifies the logic for family weighting and site selection.
2026-10-15 12:00:44 +02:00
Eric Coissac f29dcbcfb8 Implement advanced sampling and family annex logic
Release / create-release (push) Successful in 2m35s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 2m3s
ci.yml / build (pull_request) Successful in 4m8s
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data.

This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
2026-10-13 08:57:18 +02:00
Eric Coissac 0d12494f1a Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
2026-10-13 08:54:21 +02:00
Eric Coissac fdfc1eb3dd Refactor siblings module and enhance sampling logic
Introduces filtering for scan families, handles genome exclusion arguments, refactors the `siblings` module architecture, and clarifies invariants for statistical calculations.
2026-10-13 08:45:26 +02:00
Eric Coissac 996ebaf977 Refactor algorithms to use configuration flags
Introduce boolean flags for `AlignmentReducer` and `PairwiseReducer` to control behavior. Update masking logic and introduce the `SurvivingFamily` structure for explicit data handling.
2026-10-13 08:44:19 +02:00
Eric Coissac cd06fa2ce3 Implement masking logic for pairwise reduction
Introduces masking based on `free_loss` and `no_ambiguity` flags within pairwise algorithms. Updates reducer initialization and adds tests to verify cardinality calculation.
2026-10-13 08:43:29 +02:00
Eric Coissac 1702addbf9 Refactor algorithms for unified sampling and reduction
Introduces a unified sampling trunk, new reducer traits, and refined logic for SNP distance calculation and session handling. Enforces strict parameter sharing across all algorithms to ensure consistent sampling.
2026-10-13 08:43:06 +02:00
Eric Coissac 0d1ecac233 Refactor k-mer indexing, enhance session features, and update documentation
This commit introduces significant refinements across the codebase. Core indexing logic is updated with hard constraints on k-mer length, statistical modeling clarifications, and the implementation of the Karlin & Ost theory for chance-collision statistics. Session management features are enhanced with robust memoization capabilities, atomic file operations, per-key locking, and memory mapping support. Additionally, documentation is updated, including a new citation style, and new mathematical notebook support is added.
2026-10-13 08:41:42 +02:00
Eric Coissac 2693a6af09 Implement session persistence and checkpointing
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m21s
Release / build-macos-arm64 (push) Successful in 2m1s
ci.yml / build (pull_request) Successful in 3m45s
Introduces session persistence via the `--session` option, allowing sampled sets to be restored. Implements chunked dumps and shared checkpointing for alignment and tally artifacts, ensuring state restoration upon interruption. Refines sampling logic and introduces serialization mechanisms for state management.
2026-09-12 07:48:51 +02:00
Eric Coissac 7861e886e7 Introduce session persistence and artifact caching
This change implements a robust session management system allowing users to persist distance calculation samples and parameters across invocations. It includes concurrency-safe directory locking, deterministic parameter encoding, artifact integrity checks, and caching mechanisms to skip resampling when valid data is available.
2026-09-12 07:46:55 +02:00
Eric Coissac a8bcf5ffac Refactor distance matrix calculation logic for consistency
Ensures that combined distance matrix calculations use the exact same site selection by retaining sampled data within Sankoff bundles. This involves refactoring post-sampling logic into shared functions, implementing `SankoffBundle` to reuse internal tally data, and adding validation to guarantee consistent site selection across all distance calculation paths.
2026-09-12 07:45:32 +02:00
Eric Coissac 020b391636 Implement gamma shape correction for SNP distance calculation
Introduces support for rate heterogeneity via a Poisson-Gamma mixture model. This includes new command-line options (`--gamma-shape`, `--gamma-shape auto`) to enable automatic estimation of the shape parameter $\alpha$ based on substitution counts across partitions. The correction is applied to the distance metric, with logic to disable the correction if variance checks fail.
2026-09-12 07:44:39 +02:00
Eric Coissac c0e1968654 Feat: Introduce resource profiling and monitoring APIs
Release / create-release (push) Successful in 2m27s
ci.yml / build (pull_request) Successful in 3m52s
Release / build-linux-x86_64 (push) Successful in 7m50s
Release / build-macos-arm64 (push) Successful in 2m0s
This change introduces a comprehensive profiling system, allowing for periodic resource usage monitoring (CPU, I/O, memory) via a background thread. It also exposes new public APIs for budget management, locking, NUMA partitioning, and progress tracking.
2026-09-11 09:50:09 +02:00
Eric Coissac 98dba1802d Implement NUMA topology detection and CPU budget capping
Release / create-release (push) Successful in 2m30s
ci.yml / build (pull_request) Successful in 3m50s
Release / build-linux-x86_64 (push) Successful in 8m23s
Release / build-macos-arm64 (push) Successful in 1m57s
This change introduces logic to detect NUMA topology, calculate per-node worker limits based on available CPU cores, and cap resource allocation to ensure worker counts respect the physical capacity of each node.
2026-09-11 07:22:35 +02:00
Eric Coissac b66a488277 Introduce CPU budget and dynamic thread limiting
Release / create-release (push) Successful in 3m4s
ci.yml / build (pull_request) Successful in 4m17s
Release / build-linux-x86_64 (push) Successful in 8m11s
Release / build-macos-arm64 (push) Successful in 2m2s
This change implements mechanisms to enforce resource limits based on the available CPU budget, including setting a hard CPU ceiling and calculating effective thread counts for CLI, query, and superkmer operations.
2026-09-10 17:08:32 +02:00
Eric Coissac 8bb1d52514 disable default features for obisys dependency
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 7m57s
Release / build-macos-arm64 (push) Successful in 1m53s
ci.yml / build (pull_request) Successful in 3m42s
Update obisys dependency configuration across multiple crates to explicitly set default-features = false. This restricts compilation to only explicitly enabled feature flags, affecting build resolution and potentially compile-time paths or binary size. Also updates the obikmer package version to 1.3.1.
2026-08-29 08:13:49 +02:00
Eric Coissac 4d55258ada chore: bump obikmer to 1.3.0 and disable obisys default features
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m22s
Release / build-macos-arm64 (push) Failing after 1m15s
ci.yml / build (pull_request) Successful in 3m33s
Updates package version and explicitly disables default features for the local obisys dependency.
2026-08-29 07:46:56 +02:00
Eric Coissac 52f26809af refactor(benchmark): consolidate generated artifacts under run/ directory
ci.yml / build (pull_request) Successful in 3m49s
Restructure the benchmark pipeline to direct all simulated data, indices, statistics, and query outputs into a unified `run/` directory. Update Makefile targets, shell scripts, and Python utilities to resolve paths relative to this new base. Adjust documentation and dependency tracking to match the revised layout, and remove outdated temporary artifacts.
2026-08-29 00:19:31 +02:00
Eric Coissac 54e4600120 feat: add batched int group stats API and expand benchmark variants
Introduces a `batch_int_group_stats` API for computing presence counts, sums, minimums, and maximums across sparse and dense matrix representations. The selection layer now utilizes this batched approach to optimize aggregation semantics for boolean and numeric operations. Additionally, reorganizes the benchmarking infrastructure to support querying across presence and count index variants in both dense and sparse formats, including new packing scripts and updated statistics aggregation.
2026-08-28 23:37:16 +02:00
Eric Coissac 6b0c0867cc implement hybrid hard-link copy strategy for directory bootstrap
Introduces recursive directory replication functions that default to hard-linking for space efficiency, with automatic fallback to full file copying on cross-filesystem failures. Enforces mandatory real copies for presence and counts subdirectories to guarantee independent data matrices before merging, while preserving the existing separation between immutable index files and mutable matrix files.
2026-08-28 23:18:34 +02:00
Eric Coissac 93fe838f97 Remove in-place flag, optimize presence counting, and handle NUMA panics
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
2026-08-28 23:16:37 +02:00
Eric Coissac 7363370065 feat: add --force-copy flag to copy files instead of hard-linking
Introduces the `--force-copy` CLI argument to bypass hard-linking during selection. When enabled, unchanged kmer-identity files are copied instead of hard-linked to ensure output independence from the source filesystem. The default behavior remains unchanged, continuing to use hard-links with automatic fallback to copying on failure.
2026-08-28 23:08:42 +02:00
Eric Coissac 95fa0c93b2 Rename CLI to obikmer2, add phylo commands, and unify index caching
Restructure the workspace and rename the CLI application to obikmer2. Replace direct KmerIndex usage across all commands with Arc-wrapped IndexCache to enable shared ownership. Introduce builder patterns for algorithmic operations and integrate explicit progress tracking. Add new Phylo, NameTree, and Convert subcommands with expanded CLI flags. Consolidate module structure, update dependency specifications, and remove legacy directories.
2026-08-28 22:34:46 +02:00
Eric Coissac e101f629e6 Add family overlap computation and CLI options for phylo analysis
Extends the entropy annex pipeline to optionally compute family overlap data in a single pass, introducing a memory-mapped binary format and an in-memory accumulator for efficient storage. Adds --min-shared-family to auto-exclude genomes below a threshold during SNP-family distance computation, and --family-overlap to export a shared family matrix as CSV. Centralizes upper-triangle indexing into a shared helper function to eliminate duplication.
2026-08-28 22:03:21 +02:00
Eric Coissac 4f34a646c5 Implement SNP distance models with gamma correction and PHYLIP output
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
2026-08-28 21:54:19 +02:00
Eric Coissac 0b40d2d0da Add phylogenetic export support for TNT, PhyG, and IQ-TREE
Extends the phylogenetic pipeline with exporters for TNT, PhyG, and IQ-TREE that generate executable scripts, cost matrices, and recoded alignments. Adds internal helpers for state indexing, floating-point matrix scaling, and Floyd-Warshall metric closure to satisfy external tool constraints. Adjusts visibility modifiers for sibling iterators and entropy annex structs, and updates design documentation for evolutionary distance metrics.
2026-08-28 21:32:32 +02:00
Eric Coissac 1536217ac1 feat: add Sankoff cost-matrix calibration CLI and serialization
Introduce the --sankoff flag to generate subsampled calibration bundles and pairwise cost matrices. Expose internal sibling algorithm types as public to support external consumption. Implement YAML, CSV, and FASTA output serialization using serde, gated by a configurable substitution saturation ceiling and requiring the --subsample flag.
2026-08-28 21:19:20 +02:00
Eric Coissac 9043868228 Add single-pass pairwise reduction pipeline for SNP alignment
Replaces the previous two-pass design with a single-pass reduction that computes per-genome-pair substitution counts, cardinality and base-composition tallies, and transition cost matrices. Introduces internal APIs to derive downstream n×n matrices from reduced data and applies configurable filtering based on SNP ratios and exclusion masks. Adds a fused pipeline entry point aggregating alignment and distance outputs for downstream phylogenetic commands. Adjusts visibility of existing reduction functions to restrict external access while permitting sibling module invocation.
2026-08-28 21:10:16 +02:00
Eric Coissac e015362ce6 Refactor sampling and reduction to use batched SurvivingFamily data
Replace per-family callbacks with a per-layer batch callback that accumulates surviving families into an Arc<Vec<SurvivingFamily>>. Update the alignment reduction routine to iterate over this batched structure, enabling zero-copy sharing for parallel operations and structuring the code for future extensibility.
2026-08-28 21:02:35 +02:00
Eric Coissac 89629e118d Add sibling family size distribution statistics and CLI flags
Introduces `--sibling-stats` and `--sibling-hist` flags to compute and export family-size distributions from pre-built annexes. The new algorithms module implements parallel and sequential scanning routines, while the CLI layer handles CSV export with deduplication and standardized error handling.
2026-08-28 20:59:07 +02:00
Eric Coissac beb2951c20 feat: add phylogenetic sampling, masking, and plain FASTA writer
Introduces configurable phylogenetic sampling and masking controls via new CLI flags (`--subsample`, `--entropy`, `--exclude-genome`, etc.). Adds a complete SNP pseudo-alignment pipeline featuring entropy-biased Gaussian sampling, post-hoc state masking, and proportional per-layer filtering. Extends the FASTA writer with a `write_plain_record` API for bare-header output without JSON annotations.
2026-08-28 20:54:51 +02:00
Eric Coissac 5a0b71d105 Add per-family Shannon entropy calculation and reporting
Introduces an entropy.pent persisted format and algorithms to compute per-family Shannon entropy across genomic partitions. Refactors IndexCache to use Arc-based layer sharing for efficient iteration and restructures the siblings module with extension traits for annex generation. Adds a --shannon CLI flag to export partition-level entropy metrics to CSV.
2026-08-28 20:45:47 +02:00
Eric Coissac bbb58a698f Rename SiblingAnnexBuildExt to SiblingExt and add sibling iterators
Refactors the `SiblingAnnexBuildExt` trait to `SiblingExt` across all imports, implementations, and documentation. Introduces new modules for streaming and batched iteration over sibling layers, along with a two-phase pipeline for resolving minorant families.
2026-08-28 20:36:17 +02:00
Eric Coissac 920120d20e Implement --sibling-annex flag and k-mer family annex builder
Introduces a new `--sibling-annex` CLI flag that triggers the construction of a sibling-count/minorant annex for multi-genome indices. The implementation adds a `FamilyMask` data structure with memory-mapped I/O, enabling lock-free concurrent updates via atomic bitwise operations. Batched processing improves cache locality and parallelism, while helper functions derive family presence and minorant flags dynamically. The feature is exposed through an `IndexCache` extension trait, protected by an exclusive directory lock to prevent index corruption during construction.
2026-08-28 20:29:01 +02:00
Eric Coissac d084396aba Introduce concurrency-bounded parallel processing for index cache layers
Implement an `obipipeline::Throttle` with an RAII guard to acquire and release concurrency slots. Expose new bounded parallel methods on `IndexCache` to process cached layers with a configurable cap. Refactor downstream aggregation logic to use single-pass parallel map-reduce instead of manual collect-map-reduce sequences, enforcing a maximum of 8 concurrent layer scans to bound memory usage.
2026-08-28 20:20:12 +02:00
Eric Coissac 9dee6dcd08 refactor: decouple tree construction from Newick serialization
Refactor phylogenetic algorithms (`neighbor_joining`, `upgma`) to return an explicit `Tree` struct instead of a serialized Newick string. This change makes serialization an explicit step for downstream consumers via the new public `Tree::to_newick()` method, decoupling tree construction from output formatting. The `siblings` module has also been moved to `siblings_old`.
2026-08-28 20:15:23 +02:00
Eric Coissac 635fc830d1 feat: add phylogenetic analysis CLI command and lazy distance traits
Introduce the `obikphylo` crate to support genome-vs-genome distance matrix computation and phylogenetic tree inference via Neighbor-Joining and UPGMA algorithms. Extend the `obikmer2` CLI with a new `phylo` command that exposes configurable metrics, presence thresholds, and CSV/Newick output options. Refactor `obikindex` to expose trait-based partial aggregation for efficient distance metric finalization without full matrix materialization. Update dependency graphs and remove obsolete storage modules.
2026-08-28 20:11:13 +02:00
Eric Coissac 4b6005962e Rename PersistentCompactIntMatrix to PersistentIntMatrix
Update all matrix type references, imports, and instantiations across
the codebase to use the new PersistentIntMatrix name. The change also
includes standardizing code formatting, such as multi-line statements
and import ordering, without altering any behavioral logic or public
API contracts.
2026-08-28 19:59:59 +02:00
Eric Coissac 6bdc9354d3 refactor: restructure obikphylo module layout and adjust visibility
Relocate cardcomp into the siblings directory and update internal import paths to reflect the new hierarchy. Adjust the crate's public API surface by updating module declarations and re-exports. This is a purely structural change with no modifications to logic, data models, or runtime behavior.
2026-08-28 19:46:09 +02:00