Commit Graph
500 Commits
Author SHA1 Message Date
Eric Coissac 3bd7fd4f41 Implement memory mapping for file construction
Refactors the building process to utilize memory mapping (`memmap2`) for file creation and introduces the necessary infrastructure for publishing built files atomically. This includes expanding the public API surface and cleaning up deprecated architectural components.
2026-10-15 12:02:33 +02:00
Eric Coissac c226744405 Refactor sampling algorithms and coverage metrics
Optimize family scanning and sampling plan calculation logic. Introduce new per-genome richness metrics and redefine the sampling fraction $r$ for improved analysis.
2026-10-15 12:01:15 +02:00
Eric Coissac 0e40e83fd1 Refactor probability formula notation
Update the mathematical formula for $P(X \geq r)$ to explicitly use summation notation and clearly format the probability expression.
2026-10-15 12:00:51 +02:00
Eric Coissac cb5fc202b6 Refactor sampling logic and introduce diagnostic outputs
This commit introduces new features for sampling, including detailed `SamplePlan` generation, Shannon entropy summaries, and site statistics reporting. It formalizes conditional distance matrix computation and clarifies the logic for family weighting and site selection.
2026-10-15 12:00:44 +02:00
coissac 2232628e58 Merge pull request 'Push kkkmrtvotrsu' (#77) from push-kkkmrtvotrsu into main
Reviewed-on: #77
2026-10-13 07:54:48 +00:00
Eric Coissac f29dcbcfb8 Implement advanced sampling and family annex logic
Release / create-release (push) Successful in 2m35s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 2m3s
ci.yml / build (pull_request) Successful in 4m8s
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data.

This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
v1.3.6
2026-10-13 08:57:18 +02:00
Eric Coissac 0d12494f1a Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
2026-10-13 08:54:21 +02:00
Eric Coissac 95bbc0b2ae Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
2026-10-13 08:52:37 +02:00
Eric Coissac fdfc1eb3dd Refactor siblings module and enhance sampling logic
Introduces filtering for scan families, handles genome exclusion arguments, refactors the `siblings` module architecture, and clarifies invariants for statistical calculations.
2026-10-13 08:45:26 +02:00
Eric Coissac 996ebaf977 Refactor algorithms to use configuration flags
Introduce boolean flags for `AlignmentReducer` and `PairwiseReducer` to control behavior. Update masking logic and introduce the `SurvivingFamily` structure for explicit data handling.
2026-10-13 08:44:19 +02:00
Eric Coissac cd06fa2ce3 Implement masking logic for pairwise reduction
Introduces masking based on `free_loss` and `no_ambiguity` flags within pairwise algorithms. Updates reducer initialization and adds tests to verify cardinality calculation.
2026-10-13 08:43:29 +02:00
Eric Coissac 1702addbf9 Refactor algorithms for unified sampling and reduction
Introduces a unified sampling trunk, new reducer traits, and refined logic for SNP distance calculation and session handling. Enforces strict parameter sharing across all algorithms to ensure consistent sampling.
2026-10-13 08:43:06 +02:00
Eric Coissac 0d1ecac233 Refactor k-mer indexing, enhance session features, and update documentation
This commit introduces significant refinements across the codebase. Core indexing logic is updated with hard constraints on k-mer length, statistical modeling clarifications, and the implementation of the Karlin & Ost theory for chance-collision statistics. Session management features are enhanced with robust memoization capabilities, atomic file operations, per-key locking, and memory mapping support. Additionally, documentation is updated, including a new citation style, and new mathematical notebook support is added.
2026-10-13 08:41:42 +02:00
Eric Coissac e715528d34 Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
2026-10-13 08:38:49 +02:00
Eric Coissac 9da0876ee6 Refactor indexing logic using hash-based minimizer selection.
Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
2026-10-13 08:34:51 +02:00
coissac 755cf85334 Merge pull request 'Push twnyxtspltux' (#76) from push-twnyxtspltux into main
Reviewed-on: #76
2026-09-12 05:57:25 +00:00
Eric Coissac 2693a6af09 Implement session persistence and checkpointing
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m21s
Release / build-macos-arm64 (push) Successful in 2m1s
ci.yml / build (pull_request) Successful in 3m45s
Introduces session persistence via the `--session` option, allowing sampled sets to be restored. Implements chunked dumps and shared checkpointing for alignment and tally artifacts, ensuring state restoration upon interruption. Refines sampling logic and introduces serialization mechanisms for state management.
v1.3.5
2026-09-12 07:48:51 +02:00
Eric Coissac 7861e886e7 Introduce session persistence and artifact caching
This change implements a robust session management system allowing users to persist distance calculation samples and parameters across invocations. It includes concurrency-safe directory locking, deterministic parameter encoding, artifact integrity checks, and caching mechanisms to skip resampling when valid data is available.
2026-09-12 07:46:55 +02:00
Eric Coissac a8bcf5ffac Refactor distance matrix calculation logic for consistency
Ensures that combined distance matrix calculations use the exact same site selection by retaining sampled data within Sankoff bundles. This involves refactoring post-sampling logic into shared functions, implementing `SankoffBundle` to reuse internal tally data, and adding validation to guarantee consistent site selection across all distance calculation paths.
2026-09-12 07:45:32 +02:00
Eric Coissac be838da256 Refine theoretical justification for alpha estimation.
Document limitations of various $\alpha$-estimation methods, clarifying that tree-based methods require a tree for accurate heterogeneity estimation and that `PartitionDispersion` avoids violating identifiability requirements.
2026-09-12 07:44:51 +02:00
Eric Coissac 020b391636 Implement gamma shape correction for SNP distance calculation
Introduces support for rate heterogeneity via a Poisson-Gamma mixture model. This includes new command-line options (`--gamma-shape`, `--gamma-shape auto`) to enable automatic estimation of the shape parameter $\alpha$ based on substitution counts across partitions. The correction is applied to the distance metric, with logic to disable the correction if variance checks fail.
2026-09-12 07:44:39 +02:00
coissac e846d35adb Merge pull request 'Feat: Introduce resource profiling and monitoring APIs' (#75) from push-sktolxuyskpp into main
Reviewed-on: #75
2026-09-11 07:51:03 +00:00
Eric Coissac c0e1968654 Feat: Introduce resource profiling and monitoring APIs
Release / create-release (push) Successful in 2m27s
ci.yml / build (pull_request) Successful in 3m52s
Release / build-linux-x86_64 (push) Successful in 7m50s
Release / build-macos-arm64 (push) Successful in 2m0s
This change introduces a comprehensive profiling system, allowing for periodic resource usage monitoring (CPU, I/O, memory) via a background thread. It also exposes new public APIs for budget management, locking, NUMA partitioning, and progress tracking.
v1.3.4
2026-09-11 09:50:09 +02:00
coissac e280b6d0a3 Merge pull request 'Implement NUMA topology detection and CPU budget capping' (#74) from push-tnkxqmwztnmm into main
Reviewed-on: #74
2026-09-11 05:23:42 +00:00
Eric Coissac 98dba1802d Implement NUMA topology detection and CPU budget capping
Release / create-release (push) Successful in 2m30s
ci.yml / build (pull_request) Successful in 3m50s
Release / build-linux-x86_64 (push) Successful in 8m23s
Release / build-macos-arm64 (push) Successful in 1m57s
This change introduces logic to detect NUMA topology, calculate per-node worker limits based on available CPU cores, and cap resource allocation to ensure worker counts respect the physical capacity of each node.
v1.3.3
2026-09-11 07:22:35 +02:00
coissac dd4285b269 Merge pull request 'Introduce CPU budget and dynamic thread limiting' (#73) from push-lztkokoxqpuw into main
Reviewed-on: #73
2026-09-10 15:10:19 +00:00
Eric Coissac b66a488277 Introduce CPU budget and dynamic thread limiting
Release / create-release (push) Successful in 3m4s
ci.yml / build (pull_request) Successful in 4m17s
Release / build-linux-x86_64 (push) Successful in 8m11s
Release / build-macos-arm64 (push) Successful in 2m2s
This change implements mechanisms to enforce resource limits based on the available CPU budget, including setting a hard CPU ceiling and calculating effective thread counts for CLI, query, and superkmer operations.
v1.3.2
2026-09-10 17:08:32 +02:00
Eric Coissac b954032faa chore: remove obsolete configuration and memory files
Deletes unused tool configurations, planning plans, and local memory files across multiple directories.
2026-08-31 09:09:40 +02:00
coissac 43d89d8f02 Merge pull request 'disable default features for obisys dependency' (#72) from push-wvkttrwzkqoo into main
Reviewed-on: #72
2026-08-29 06:18:11 +00:00
Eric Coissac 8bb1d52514 disable default features for obisys dependency
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 7m57s
Release / build-macos-arm64 (push) Successful in 1m53s
ci.yml / build (pull_request) Successful in 3m42s
Update obisys dependency configuration across multiple crates to explicitly set default-features = false. This restricts compilation to only explicitly enabled feature flags, affecting build resolution and potentially compile-time paths or binary size. Also updates the obikmer package version to 1.3.1.
v1.3.1
2026-08-29 08:13:49 +02:00
coissac 77d4429645 Merge pull request 'chore: bump obikmer to 1.3.0 and disable obisys default features' (#71) from push-sllyqzuztwkr into main
Reviewed-on: #71
2026-08-29 05:51:37 +00:00
Eric Coissac 4d55258ada chore: bump obikmer to 1.3.0 and disable obisys default features
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m22s
Release / build-macos-arm64 (push) Failing after 1m15s
ci.yml / build (pull_request) Successful in 3m33s
Updates package version and explicitly disables default features for the local obisys dependency.
v1.3.0
2026-08-29 07:46:56 +02:00
coissac 7e28ca4853 Merge pull request 'Push zunrplorkwkt' (#70) from push-zunrplorkwkt into main
Reviewed-on: #70
2026-08-28 23:15:37 +00:00
Eric Coissac 52f26809af refactor(benchmark): consolidate generated artifacts under run/ directory
ci.yml / build (pull_request) Successful in 3m49s
Restructure the benchmark pipeline to direct all simulated data, indices, statistics, and query outputs into a unified `run/` directory. Update Makefile targets, shell scripts, and Python utilities to resolve paths relative to this new base. Adjust documentation and dependency tracking to match the revised layout, and remove outdated temporary artifacts.
2026-08-29 00:19:31 +02:00
Eric Coissac 54e4600120 feat: add batched int group stats API and expand benchmark variants
Introduces a `batch_int_group_stats` API for computing presence counts, sums, minimums, and maximums across sparse and dense matrix representations. The selection layer now utilizes this batched approach to optimize aggregation semantics for boolean and numeric operations. Additionally, reorganizes the benchmarking infrastructure to support querying across presence and count index variants in both dense and sparse formats, including new packing scripts and updated statistics aggregation.
2026-08-28 23:37:16 +02:00
Eric Coissac 579cfd8752 docs: document partition runner deadlock fix and select CLI updates
Update documentation to reflect architectural fixes in the NUMA partition runner and implementation changes in the merge and select modules. The partition runner now wraps task execution in `catch_unwind`, emitting panic events that count toward completion thresholds to prevent controller deadlocks, while re-raising the first payload on return. CLI documentation for select has been updated to mandate output directories, introduce default hard-linking with automatic fallback, and document presence aggregation optimizations. Known issues such as self-merge crashes and sparse-packed index gaps are now documented as failing fast due to these improvements.
2026-08-28 23:25:46 +02:00
Eric Coissac 6b0c0867cc implement hybrid hard-link copy strategy for directory bootstrap
Introduces recursive directory replication functions that default to hard-linking for space efficiency, with automatic fallback to full file copying on cross-filesystem failures. Enforces mandatory real copies for presence and counts subdirectories to guarantee independent data matrices before merging, while preserving the existing separation between immutable index files and mutable matrix files.
2026-08-28 23:18:34 +02:00
Eric Coissac 93fe838f97 Remove in-place flag, optimize presence counting, and handle NUMA panics
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
2026-08-28 23:16:37 +02:00
Eric Coissac 7363370065 feat: add --force-copy flag to copy files instead of hard-linking
Introduces the `--force-copy` CLI argument to bypass hard-linking during selection. When enabled, unchanged kmer-identity files are copied instead of hard-linked to ensure output independence from the source filesystem. The default behavior remains unchanged, continuing to use hard-links with automatic fallback to copying on failure.
2026-08-28 23:08:42 +02:00
Eric Coissac caf59b5658 Update documentation for CLI command renames and new commands
Renames the reindex command to convert across all usage guides and navigation menus. Adds documentation for the newly introduced name-tree command. Updates flag references, such as replacing --metric with --distance in pack and phylo modules, and adjusts MkDocs configuration to match the revised structure. All changes are strictly limited to documentation and configuration files.
2026-08-28 23:06:05 +02:00
Eric Coissac 95fa0c93b2 Rename CLI to obikmer2, add phylo commands, and unify index caching
Restructure the workspace and rename the CLI application to obikmer2. Replace direct KmerIndex usage across all commands with Arc-wrapped IndexCache to enable shared ownership. Introduce builder patterns for algorithmic operations and integrate explicit progress tracking. Add new Phylo, NameTree, and Convert subcommands with expanded CLI flags. Consolidate module structure, update dependency specifications, and remove legacy directories.
2026-08-28 22:34:46 +02:00
Eric Coissac e101f629e6 Add family overlap computation and CLI options for phylo analysis
Extends the entropy annex pipeline to optionally compute family overlap data in a single pass, introducing a memory-mapped binary format and an in-memory accumulator for efficient storage. Adds --min-shared-family to auto-exclude genomes below a threshold during SNP-family distance computation, and --family-overlap to export a shared family matrix as CSV. Centralizes upper-triangle indexing into a shared helper function to eliminate duplication.
2026-08-28 22:03:21 +02:00
Eric Coissac 4f34a646c5 Implement SNP distance models with gamma correction and PHYLIP output
Renames the CLI flag from --metric to --distance and introduces eight closed-form SNP distance models with optional Jin-Nei gamma correction. Integrates the ndarray crate for matrix operations and adds relaxed PHYLIP output formatting. Updates architecture and theory documentation to cover the new sparse matrix variants, algorithmic fixes, and distance metric implementations.
2026-08-28 21:54:19 +02:00
Eric Coissac 0b40d2d0da Add phylogenetic export support for TNT, PhyG, and IQ-TREE
Extends the phylogenetic pipeline with exporters for TNT, PhyG, and IQ-TREE that generate executable scripts, cost matrices, and recoded alignments. Adds internal helpers for state indexing, floating-point matrix scaling, and Floyd-Warshall metric closure to satisfy external tool constraints. Adjusts visibility modifiers for sibling iterators and entropy annex structs, and updates design documentation for evolutionary distance metrics.
2026-08-28 21:32:32 +02:00
Eric Coissac 1536217ac1 feat: add Sankoff cost-matrix calibration CLI and serialization
Introduce the --sankoff flag to generate subsampled calibration bundles and pairwise cost matrices. Expose internal sibling algorithm types as public to support external consumption. Implement YAML, CSV, and FASTA output serialization using serde, gated by a configurable substitution saturation ceiling and requiring the --subsample flag.
2026-08-28 21:19:20 +02:00
Eric Coissac 9043868228 Add single-pass pairwise reduction pipeline for SNP alignment
Replaces the previous two-pass design with a single-pass reduction that computes per-genome-pair substitution counts, cardinality and base-composition tallies, and transition cost matrices. Introduces internal APIs to derive downstream n×n matrices from reduced data and applies configurable filtering based on SNP ratios and exclusion masks. Adds a fused pipeline entry point aggregating alignment and distance outputs for downstream phylogenetic commands. Adjusts visibility of existing reduction functions to restrict external access while permitting sibling module invocation.
2026-08-28 21:10:16 +02:00
Eric Coissac e015362ce6 Refactor sampling and reduction to use batched SurvivingFamily data
Replace per-family callbacks with a per-layer batch callback that accumulates surviving families into an Arc<Vec<SurvivingFamily>>. Update the alignment reduction routine to iterate over this batched structure, enabling zero-copy sharing for parallel operations and structuring the code for future extensibility.
2026-08-28 21:02:35 +02:00
Eric Coissac 89629e118d Add sibling family size distribution statistics and CLI flags
Introduces `--sibling-stats` and `--sibling-hist` flags to compute and export family-size distributions from pre-built annexes. The new algorithms module implements parallel and sequential scanning routines, while the CLI layer handles CSV export with deduplication and standardized error handling.
2026-08-28 20:59:07 +02:00
Eric Coissac beb2951c20 feat: add phylogenetic sampling, masking, and plain FASTA writer
Introduces configurable phylogenetic sampling and masking controls via new CLI flags (`--subsample`, `--entropy`, `--exclude-genome`, etc.). Adds a complete SNP pseudo-alignment pipeline featuring entropy-biased Gaussian sampling, post-hoc state masking, and proportional per-layer filtering. Extends the FASTA writer with a `write_plain_record` API for bare-header output without JSON annotations.
2026-08-28 20:54:51 +02:00
Eric Coissac 5a0b71d105 Add per-family Shannon entropy calculation and reporting
Introduces an entropy.pent persisted format and algorithms to compute per-family Shannon entropy across genomic partitions. Refactors IndexCache to use Arc-based layer sharing for efficient iteration and restructures the siblings module with extension traits for annex generation. Adds a --shannon CLI flag to export partition-level entropy metrics to CSV.
2026-08-28 20:45:47 +02:00