Commit Graph
509 Commits
Author SHA1 Message Date
Eric Coissac f9edefd135 Refactor query system for performance and memory reduction
This change introduces dynamic chunk sizing based on input size estimates, ensuring memory constraints are respected while allowing for smaller chunks that yield significant speed improvements. Additionally, index structures were optimized, resulting in drastic memory footprint reduction and faster query execution times.
2026-10-15 12:21:38 +02:00
Eric Coissac 025c40c0db Refactor chunking logic to enforce 128 k-mer limit
This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
2026-10-15 12:20:36 +02:00
Eric Coissac 8468319a63 Refactor unitig detection and graph traversal logic
Introduce new hashing mechanisms and support for directional unitig checks. Fix fragmentation issues and improve performance metrics through optimized graph construction and new benchmarking utilities.
2026-10-15 12:19:24 +02:00
Eric Coissac d32d86193f Refactor index metadata loading to use buffered I/O
This change replaces unbuffered file reads with buffered I/O operations to improve performance during index metadata loading. This includes implementing new methods for reading index metadata files and wrapping file handles in `BufReader` to optimize subsequent reads.
2026-10-15 12:17:51 +02:00
Eric Coissac 093d81c359 Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
2026-10-15 12:17:09 +02:00
Eric Coissac 09c80468b2 Refactor k-mer storage to use flat arrays
This change replaces the HashMap-based k-mer storage mechanism with a compact, flat array structure for `PartitionKmers`. This significantly reduces memory usage per partition and improves query execution time.

It also updates the batch processing logic to handle the new data structure, introduces detailed memory footprint calculations, and updates relevant query layer APIs. Test coverage has been expanded to validate correct partitioning and k-mer ordering.
2026-10-15 12:16:21 +02:00
Eric Coissac fe38424682 Implement batch lookup and chunked data management
Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline.

This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
2026-10-15 12:14:51 +02:00
Eric Coissac 8b87078478 Implement MPHF migration tool and sparse matrix permutation logic
Refactors sparse matrix handling to implement row permutation logic and introduces a standalone tool for migrating indexes from MPHF 1.x to 2.x formats.
2026-10-15 12:12:51 +02:00
Eric Coissac d686d39506 Refactor indexing logic and update core dependencies
This commit introduces significant updates to the indexing system, including the introduction of zero-copy types, modifications to the `Mphf` type definition, and new hybrid search strategies. Core algorithms are updated to handle new hashing types and improved error handling during pointer hash construction. Dependency versions are also updated across the board, including new security features and system-level crates.
2026-10-15 12:07:08 +02:00
Eric Coissac 3bd7fd4f41 Implement memory mapping for file construction
Refactors the building process to utilize memory mapping (`memmap2`) for file creation and introduces the necessary infrastructure for publishing built files atomically. This includes expanding the public API surface and cleaning up deprecated architectural components.
2026-10-15 12:02:33 +02:00
Eric Coissac c226744405 Refactor sampling algorithms and coverage metrics
Optimize family scanning and sampling plan calculation logic. Introduce new per-genome richness metrics and redefine the sampling fraction $r$ for improved analysis.
2026-10-15 12:01:15 +02:00
Eric Coissac 0e40e83fd1 Refactor probability formula notation
Update the mathematical formula for $P(X \geq r)$ to explicitly use summation notation and clearly format the probability expression.
2026-10-15 12:00:51 +02:00
Eric Coissac cb5fc202b6 Refactor sampling logic and introduce diagnostic outputs
This commit introduces new features for sampling, including detailed `SamplePlan` generation, Shannon entropy summaries, and site statistics reporting. It formalizes conditional distance matrix computation and clarifies the logic for family weighting and site selection.
2026-10-15 12:00:44 +02:00
coissac 2232628e58 Merge pull request 'Push kkkmrtvotrsu' (#77) from push-kkkmrtvotrsu into main
Reviewed-on: #77
2026-10-13 07:54:48 +00:00
Eric Coissac f29dcbcfb8 Implement advanced sampling and family annex logic
Release / create-release (push) Successful in 2m35s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 2m3s
ci.yml / build (pull_request) Successful in 4m8s
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data.

This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
v1.3.6
2026-10-13 08:57:18 +02:00
Eric Coissac 0d12494f1a Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
2026-10-13 08:54:21 +02:00
Eric Coissac 95bbc0b2ae Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
2026-10-13 08:52:37 +02:00
Eric Coissac fdfc1eb3dd Refactor siblings module and enhance sampling logic
Introduces filtering for scan families, handles genome exclusion arguments, refactors the `siblings` module architecture, and clarifies invariants for statistical calculations.
2026-10-13 08:45:26 +02:00
Eric Coissac 996ebaf977 Refactor algorithms to use configuration flags
Introduce boolean flags for `AlignmentReducer` and `PairwiseReducer` to control behavior. Update masking logic and introduce the `SurvivingFamily` structure for explicit data handling.
2026-10-13 08:44:19 +02:00
Eric Coissac cd06fa2ce3 Implement masking logic for pairwise reduction
Introduces masking based on `free_loss` and `no_ambiguity` flags within pairwise algorithms. Updates reducer initialization and adds tests to verify cardinality calculation.
2026-10-13 08:43:29 +02:00
Eric Coissac 1702addbf9 Refactor algorithms for unified sampling and reduction
Introduces a unified sampling trunk, new reducer traits, and refined logic for SNP distance calculation and session handling. Enforces strict parameter sharing across all algorithms to ensure consistent sampling.
2026-10-13 08:43:06 +02:00
Eric Coissac 0d1ecac233 Refactor k-mer indexing, enhance session features, and update documentation
This commit introduces significant refinements across the codebase. Core indexing logic is updated with hard constraints on k-mer length, statistical modeling clarifications, and the implementation of the Karlin & Ost theory for chance-collision statistics. Session management features are enhanced with robust memoization capabilities, atomic file operations, per-key locking, and memory mapping support. Additionally, documentation is updated, including a new citation style, and new mathematical notebook support is added.
2026-10-13 08:41:42 +02:00
Eric Coissac e715528d34 Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
2026-10-13 08:38:49 +02:00
Eric Coissac 9da0876ee6 Refactor indexing logic using hash-based minimizer selection.
Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
2026-10-13 08:34:51 +02:00
coissac 755cf85334 Merge pull request 'Push twnyxtspltux' (#76) from push-twnyxtspltux into main
Reviewed-on: #76
2026-09-12 05:57:25 +00:00
Eric Coissac 2693a6af09 Implement session persistence and checkpointing
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m21s
Release / build-macos-arm64 (push) Successful in 2m1s
ci.yml / build (pull_request) Successful in 3m45s
Introduces session persistence via the `--session` option, allowing sampled sets to be restored. Implements chunked dumps and shared checkpointing for alignment and tally artifacts, ensuring state restoration upon interruption. Refines sampling logic and introduces serialization mechanisms for state management.
v1.3.5
2026-09-12 07:48:51 +02:00
Eric Coissac 7861e886e7 Introduce session persistence and artifact caching
This change implements a robust session management system allowing users to persist distance calculation samples and parameters across invocations. It includes concurrency-safe directory locking, deterministic parameter encoding, artifact integrity checks, and caching mechanisms to skip resampling when valid data is available.
2026-09-12 07:46:55 +02:00
Eric Coissac a8bcf5ffac Refactor distance matrix calculation logic for consistency
Ensures that combined distance matrix calculations use the exact same site selection by retaining sampled data within Sankoff bundles. This involves refactoring post-sampling logic into shared functions, implementing `SankoffBundle` to reuse internal tally data, and adding validation to guarantee consistent site selection across all distance calculation paths.
2026-09-12 07:45:32 +02:00
Eric Coissac be838da256 Refine theoretical justification for alpha estimation.
Document limitations of various $\alpha$-estimation methods, clarifying that tree-based methods require a tree for accurate heterogeneity estimation and that `PartitionDispersion` avoids violating identifiability requirements.
2026-09-12 07:44:51 +02:00
Eric Coissac 020b391636 Implement gamma shape correction for SNP distance calculation
Introduces support for rate heterogeneity via a Poisson-Gamma mixture model. This includes new command-line options (`--gamma-shape`, `--gamma-shape auto`) to enable automatic estimation of the shape parameter $\alpha$ based on substitution counts across partitions. The correction is applied to the distance metric, with logic to disable the correction if variance checks fail.
2026-09-12 07:44:39 +02:00
coissac e846d35adb Merge pull request 'Feat: Introduce resource profiling and monitoring APIs' (#75) from push-sktolxuyskpp into main
Reviewed-on: #75
2026-09-11 07:51:03 +00:00
Eric Coissac c0e1968654 Feat: Introduce resource profiling and monitoring APIs
Release / create-release (push) Successful in 2m27s
ci.yml / build (pull_request) Successful in 3m52s
Release / build-linux-x86_64 (push) Successful in 7m50s
Release / build-macos-arm64 (push) Successful in 2m0s
This change introduces a comprehensive profiling system, allowing for periodic resource usage monitoring (CPU, I/O, memory) via a background thread. It also exposes new public APIs for budget management, locking, NUMA partitioning, and progress tracking.
v1.3.4
2026-09-11 09:50:09 +02:00
coissac e280b6d0a3 Merge pull request 'Implement NUMA topology detection and CPU budget capping' (#74) from push-tnkxqmwztnmm into main
Reviewed-on: #74
2026-09-11 05:23:42 +00:00
Eric Coissac 98dba1802d Implement NUMA topology detection and CPU budget capping
Release / create-release (push) Successful in 2m30s
ci.yml / build (pull_request) Successful in 3m50s
Release / build-linux-x86_64 (push) Successful in 8m23s
Release / build-macos-arm64 (push) Successful in 1m57s
This change introduces logic to detect NUMA topology, calculate per-node worker limits based on available CPU cores, and cap resource allocation to ensure worker counts respect the physical capacity of each node.
v1.3.3
2026-09-11 07:22:35 +02:00
coissac dd4285b269 Merge pull request 'Introduce CPU budget and dynamic thread limiting' (#73) from push-lztkokoxqpuw into main
Reviewed-on: #73
2026-09-10 15:10:19 +00:00
Eric Coissac b66a488277 Introduce CPU budget and dynamic thread limiting
Release / create-release (push) Successful in 3m4s
ci.yml / build (pull_request) Successful in 4m17s
Release / build-linux-x86_64 (push) Successful in 8m11s
Release / build-macos-arm64 (push) Successful in 2m2s
This change implements mechanisms to enforce resource limits based on the available CPU budget, including setting a hard CPU ceiling and calculating effective thread counts for CLI, query, and superkmer operations.
v1.3.2
2026-09-10 17:08:32 +02:00
Eric Coissac b954032faa chore: remove obsolete configuration and memory files
Deletes unused tool configurations, planning plans, and local memory files across multiple directories.
2026-08-31 09:09:40 +02:00
coissac 43d89d8f02 Merge pull request 'disable default features for obisys dependency' (#72) from push-wvkttrwzkqoo into main
Reviewed-on: #72
2026-08-29 06:18:11 +00:00
Eric Coissac 8bb1d52514 disable default features for obisys dependency
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 7m57s
Release / build-macos-arm64 (push) Successful in 1m53s
ci.yml / build (pull_request) Successful in 3m42s
Update obisys dependency configuration across multiple crates to explicitly set default-features = false. This restricts compilation to only explicitly enabled feature flags, affecting build resolution and potentially compile-time paths or binary size. Also updates the obikmer package version to 1.3.1.
v1.3.1
2026-08-29 08:13:49 +02:00
coissac 77d4429645 Merge pull request 'chore: bump obikmer to 1.3.0 and disable obisys default features' (#71) from push-sllyqzuztwkr into main
Reviewed-on: #71
2026-08-29 05:51:37 +00:00
Eric Coissac 4d55258ada chore: bump obikmer to 1.3.0 and disable obisys default features
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m22s
Release / build-macos-arm64 (push) Failing after 1m15s
ci.yml / build (pull_request) Successful in 3m33s
Updates package version and explicitly disables default features for the local obisys dependency.
v1.3.0
2026-08-29 07:46:56 +02:00
coissac 7e28ca4853 Merge pull request 'Push zunrplorkwkt' (#70) from push-zunrplorkwkt into main
Reviewed-on: #70
2026-08-28 23:15:37 +00:00
Eric Coissac 52f26809af refactor(benchmark): consolidate generated artifacts under run/ directory
ci.yml / build (pull_request) Successful in 3m49s
Restructure the benchmark pipeline to direct all simulated data, indices, statistics, and query outputs into a unified `run/` directory. Update Makefile targets, shell scripts, and Python utilities to resolve paths relative to this new base. Adjust documentation and dependency tracking to match the revised layout, and remove outdated temporary artifacts.
2026-08-29 00:19:31 +02:00
Eric Coissac 54e4600120 feat: add batched int group stats API and expand benchmark variants
Introduces a `batch_int_group_stats` API for computing presence counts, sums, minimums, and maximums across sparse and dense matrix representations. The selection layer now utilizes this batched approach to optimize aggregation semantics for boolean and numeric operations. Additionally, reorganizes the benchmarking infrastructure to support querying across presence and count index variants in both dense and sparse formats, including new packing scripts and updated statistics aggregation.
2026-08-28 23:37:16 +02:00
Eric Coissac 579cfd8752 docs: document partition runner deadlock fix and select CLI updates
Update documentation to reflect architectural fixes in the NUMA partition runner and implementation changes in the merge and select modules. The partition runner now wraps task execution in `catch_unwind`, emitting panic events that count toward completion thresholds to prevent controller deadlocks, while re-raising the first payload on return. CLI documentation for select has been updated to mandate output directories, introduce default hard-linking with automatic fallback, and document presence aggregation optimizations. Known issues such as self-merge crashes and sparse-packed index gaps are now documented as failing fast due to these improvements.
2026-08-28 23:25:46 +02:00
Eric Coissac 6b0c0867cc implement hybrid hard-link copy strategy for directory bootstrap
Introduces recursive directory replication functions that default to hard-linking for space efficiency, with automatic fallback to full file copying on cross-filesystem failures. Enforces mandatory real copies for presence and counts subdirectories to guarantee independent data matrices before merging, while preserving the existing separation between immutable index files and mutable matrix files.
2026-08-28 23:18:34 +02:00
Eric Coissac 93fe838f97 Remove in-place flag, optimize presence counting, and handle NUMA panics
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
2026-08-28 23:16:37 +02:00
Eric Coissac 7363370065 feat: add --force-copy flag to copy files instead of hard-linking
Introduces the `--force-copy` CLI argument to bypass hard-linking during selection. When enabled, unchanged kmer-identity files are copied instead of hard-linked to ensure output independence from the source filesystem. The default behavior remains unchanged, continuing to use hard-links with automatic fallback to copying on failure.
2026-08-28 23:08:42 +02:00
Eric Coissac caf59b5658 Update documentation for CLI command renames and new commands
Renames the reindex command to convert across all usage guides and navigation menus. Adds documentation for the newly introduced name-tree command. Updates flag references, such as replacing --metric with --distance in pack and phylo modules, and adjusts MkDocs configuration to match the revised structure. All changes are strictly limited to documentation and configuration files.
2026-08-28 23:06:05 +02:00
Eric Coissac 95fa0c93b2 Rename CLI to obikmer2, add phylo commands, and unify index caching
Restructure the workspace and rename the CLI application to obikmer2. Replace direct KmerIndex usage across all commands with Arc-wrapped IndexCache to enable shared ownership. Introduce builder patterns for algorithmic operations and integrate explicit progress tracking. Add new Phylo, NameTree, and Convert subcommands with expanded CLI flags. Consolidate module structure, update dependency specifications, and remove legacy directories.
2026-08-28 22:34:46 +02:00