Commit Graph
523 Commits
Author SHA1 Message Date
Eric Coissac 07bc5fc634 docs: update sparse packing strategy and benchmark scripts
ci.yml / build (pull_request) Successful in 4m53s
Release / create-release (push) Successful in 28s
Release / build-linux-x86_64 (push) Successful in 6m46s
Release / build-macos-arm64 (push) Successful in 4m8s
Update documentation to reflect that sparse packing now maps all columns of a layer at once and packs layers sequentially, reversing the previous choice to avoid materializing a packed copy. This change introduces an extra write/read cycle for the dense matrix, increasing pack benchmark time from 1.11s to 1.60s.

- Modify `pack_sparse_bit_matrix` in `src/obicompactvec/src/bitmatrix/sparse.rs` to always convert via the dense packed format, simplifying logic by removing conditional handling of existing files and ensuring a single mapping.
- Update `pack_sparse_compact_int_matrix` in `src/obicompactvec/src/sparse_intmatrix.rs` to exclusively handle conversion from the dense packed format (`matrix.pcmx`) to sparse, removing columnar format handling and simplifying file cleanup.
- Adjust `scripts/bench_merge_oracle.sh` to extract `PACK_WALL` metrics via `stage_field` for merge operations, updating grep patterns and output messages to report pack wall time.
- Revise `DevDocMD/implementation/merge_mmap_audit.md` and `merge_scaling.md` to document the new sequential packing strategy, noting that `N_part` scaling applies only to incremental merges and that `obisys::MemoryBudget` is no longer used by merge.
- Bump package version in `src/obikmer/Cargo.toml` from 1.4.3 to 1.4.4.
v1.4.4
2026-11-17 15:35:41 +01:00
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00
coissac b35127e4e3 Merge pull request 'Push smsmsoyssmnr' (#80) from push-smsmsoyssmnr into main
Reviewed-on: #80
2026-11-09 10:01:21 +00:00
Eric Coissac 0134c69cf4 docs: clarify VCF pseudo-alignment format and update obikmer dependency
ci.yml / build (pull_request) Successful in 3m40s
Release / create-release (push) Successful in 1m7s
Release / build-macos-arm64 (push) Successful in 1m52s
Release / build-linux-x86_64 (push) Successful in 6m41s
Update the VCF export documentation to reflect that `-` represents an overlapping deletion (extra allele `*`) and introduce the `<ABS>` symbolic allele for absent families. Add a new dedicated documentation file for the VCF pseudo-alignment format, remove obsolete descriptions from phylo.md, and fix benchmark scripts to handle errors and directory creation correctly.

- Update `obikmer` dependency from 1.4.1 to 1.4.2 in Cargo.toml and Cargo.lock
- Add `UserDocMD/formats/vcf.md` describing VCF pseudo-alignment structure, alleles, and genotypes
- Remove obsolete VCF descriptions from `UserDocMD/usage/phylo.md` and update ploidy/free-loss notes
- Update `DevDocMD/theory/evolutionary_distances.md` to clarify `-` as overlapping deletion and `<ABS>` for others
- Fix `benchmark/index_one_count.sh` to create index directory and check read command errors
- Fix `benchmark/index_one_presence.sh` to create index directory and check simulation errors
- Update `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` to use `<ABS>` for absent families instead of `*`
v1.4.2
2026-11-09 20:59:28 +11:00
Eric Coissac 5368c98875 refactor: replace no_ambiguity bool with AmbiguityPolicy enum
Replace the binary `no_ambiguity` flag with an `AmbiguityPolicy` enum (`Keep`, `Forbid`, `OnlyDiploid`) to support multiple ambiguity conventions. This change updates CLI arguments, internal logic, and persistent storage formats to distinguish between keeping all genomes, forbidding ambiguous ones, or restricting analysis to diploid sites only.

- Add `src/obikphylo/src/siblings/ambiguity.rs` defining the `AmbiguityPolicy` enum with variants `Keep`, `Forbid`, and `OnlyDiploid`, including methods for header flags and determining if a genome is dropped.
- Update `PhyloArgs` in `args.rs` to derive `AmbiquityPolicy` from CLI flags and construct `AnnexSelection` for sibling annexes.
- Modify `SamplingSpec` in `subsample.rs` to store `ambiguity: AmbiguityPolicy` instead of `no_ambiguity`, updating masking and scalar value logic accordingly.
- Update `SiblingExt` in `sibling_ext.rs` so `build_sibling_annex` accepts an `AnnexSelection` to build only requested derived annexes, and `family_overlap` uses the specific policy.
- Refactor `FamilyOverlap` in `family_overlap.rs` to store and use `AmbiguityPolicy`, replacing boolean checks with policy-based carrier determination.
- Update `FamilyScalarAnnex` in `family_scalar_annex.rs` to store `AmbiguityPolicy` in headers, enabling distinct file names for each policy variant.
- Modify `quartet_annex.rs` to generate distinct file names based on `AmbiguityPolicy`, ensuring no overlap between Keep, Forbid, and OnlyDiploid annexes.
- Update `family_annexes.rs` to use `AnnexSelection` for building entropy, quartet, and pair weight annexes under specific policies instead of a single boolean convention.
- Refactor algorithm modules (`entropy.rs`, `masking.rs`, `pair_weight.rs`, `quartet.rs`, `snp_distance.rs`) to accept `AmbiguityPolicy` arguments, replacing boolean checks with policy-specific logic (e.g., `OnlyDiploid` masking).
- Update documentation in `phylo.md` and `evolutionary_distances.md` to describe the new `--only-diploid` option, the behavior of `--sibling-annex` with specific policies, and the separate storage of entropies per convention.
2026-11-09 20:59:09 +11:00
Eric Coissac b55123a376 feat: add --only-diploid flag to filter for diploid genotypes only
Introduce a new `--only-diploid` option that restricts analysis to genomes with one or two family members (base or two-base IUPAC codes), treating others as missing data. This complements the existing `--no-ambiguity` behavior by providing a stricter filter for diploid-compatible sites, while ensuring conflict detection between the two flags.

- Add `only_diploid` boolean argument to `PhyloArgs` in `src/obikmer/src/cmd/phylo/args.rs`, documenting its effect on pseudo-alignment and its conflict with `--no-ambiguity`.
- Propagate `only_diploid` through `SamplingSpec` in `src/obikmer/src/cmd/phylo/mod.rs` and update the `run` function signature to accept the new argument.
- Implement masking logic in `src/obikphylo/src/siblings/algorithms/masking.rs`: modify `masked_state` to accept `only_diploid`, add `only_diploid_keeps_up_to_two_members` helper, and update `genome_mask` documentation.
- Update `sample_layer` in `src/obikphylo/src/siblings/algorithms/subsample.rs` to use `spec.only_diploid` for masking decisions and update `SamplingSpec` documentation.
- Adjust `tally_2genomes` in `src/obikphylo/src/siblings/algorithms/snp_distance.rs` to pass the new `only_diploid` parameter to `masked_state`.
- Update `DevDocMD/theory/evolutionary_distances.md` to describe the new `--only-diploid` option, the `masked_state` rule (`count_ones() > 2`), and clarify toy index behavior.
- Update `UserDocMD/usage/phylo.md` to document `--only-diploid`, `--quartet`, and `--pair-balance` behaviors, clarifying that diploid choice is a convention and explaining how three/four-allele genomes are treated as missing.
2026-11-09 20:58:21 +11:00
Eric Coissac b132f0847f feat: add VCF export and refine alignment output infrastructure
Introduce `--vcf` and `--fasta` options for phylo commands, allowing users to
export pseudo-alignment data as VCF or FASTA. Implement comprehensive VCF logic
including stable virtual coordinates, canonical ID generation, and flexible
genotype representation based on ambiguity settings. Refactor internal alignment
structures to track k-mer families and site loci, ensuring consistent data flow
from scanning through to output generation.

- Add `src/obikmer/src/cmd/phylo/alignment_out.rs` with `create` and
  `write_alignment_outputs` to handle file creation and dispatching of FASTA/VCF
  formats based on `PhyloArgs`.
- Add `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` implementing
  VCF-specific logic: `write_vcf`, `alignment`, `cell_alleles`, `majority_base`,
  and site construction via `site` using fixed $K=11$. Includes helpers for
  IUPAC state inversion, genotype ordering, and stable coordinate generation.
- Update `src/obikmer/src/cmd/phylo/args.rs` to add `fasta` and `vcf` flags
  and a new `alignment_output()` method that returns true if any output format
  is requested. Add `pseudo_alignment` field documentation.
- Modify `src/obikmer/src/cmd/phylo/mod.rs` to use `alignment_output()` for
  determining alignment needs, update error messages for `--n-sites`, and import
  `write_alignment_outputs` from the new module.
- Refactor `src/obikphylo/src/siblings/algorithms/alignment.rs` by adding
  `SiteLocus` struct (partition, layer, family_idx, family_id) and updating
  `SnpAlignment` and `AlignmentReducer` to track sites. Update `new`, `finish`,
  and `reduce` to populate site information from `SurvivingFamily`.
- Update `src/obikphylo/src/siblings/algorithms/iqtree.rs`: `alignment` now
  returns `SnpAlignment` with empty sites; update `drop_ascertainment_noninformative`
  and `recode_symbols_as_absent` to handle the new `sites` field.
- Update `src/obikphylo/src/siblings/algorithms/family_scan.rs`: `GeneratedBatch`
  now includes a `kmers` field; `scan_layer_families` accepts `CanonicalKmer`
  and passes it to closures instead of masks.
- Update `src/obikphylo/src/siblings/algorithms/subsample.rs`: `SurvivingFamily`
  gains a `kmer` field; `sample_layer` closure accepts and uses this k-mer.
- Update `src/obikphylo/src/siblings/algorithms/snp_distance.rs`: `tally_2genomes`
  initializes `SurvivingFamily` with explicit `CanonicalKmer` values.
- Update `src/obikphylo/src/siblings/algorithms/stats.rs`: `sibling_annex_stats`
  signature adds `_kmer` parameter.
- Update `src/obikphylo/src/siblings/algorithms/family_annexes.rs`: closures in
  `ensure_layer_family_annexes` and `ensure_layer_pair_weight_annexes` now accept
  `_kmer`.
- Update `src/obikphylo/src/siblings/algorithms/pairwise.rs`: `ambiguous_vs_single`
  initializes k-mer using `CanonicalKmer::from_raw_unchecked(0)`.
- Add `iupac_state` function to `src/obikphylo/src/siblings/algorithms/masking.rs`
  mapping DNA characters to binary codes.
- Update `src/obikphylo/src/siblings/algorithms/mod.rs` to export `SiteLocus`
  and include new `alignment_vcf` module.
- Add `family_id` and `write_family_id_ascii` methods to
  `src/obikseq/src/kmer.rs` for stable k-mer family identification.
- Add tests `family_id_partitions_kmers_into_families` and
  `family_id_ascii_masks_centre` to `src/obikseq/src/tests/kmer.rs`.
- Update `src/obikphylo/src/siblings/mod.rs` imports to include new algorithm
  exports.
2026-11-09 20:58:05 +11:00
coissac ee62b131f9 Merge pull request 'Push ltpzonnnkpqs' (#79) from push-ltpzonnnkpqs into main
Reviewed-on: #79
2026-10-16 14:43:50 +00:00
Eric Coissac 85296a2ef8 Implement directory locking with shared and exclusive modes
Release / create-release (push) Successful in 2m41s
Release / build-linux-x86_64 (push) Successful in 9m17s
Release / build-macos-arm64 (push) Successful in 2m2s
ci.yml / build (pull_request) Successful in 3m48s
Introduces a robust, OS-level directory locking mechanism using LockSlot for managing shared and exclusive access, supporting lock transitions and non-blocking probes. Also sets up the index structure with new builder and index modules.
v1.4.1
2026-10-16 16:02:00 +02:00
Eric Coissac 9ac3a37529 Implement explicit directory locking for index operations
Introduces explicit directory locking mechanisms (`DirLock`) to ensure exclusive access during index operations across command stages. This includes modifying index opening logic to acquire locks before proceeding, handling lock acquisition failures by exiting, and implementing safe conditional building of sibling annexes using locking to prevent concurrent corruption.
2026-10-16 16:00:49 +02:00
Eric Coissac 7c7d7615cb Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
2026-10-16 15:59:40 +02:00
coissac c6226dd40c Merge pull request 'Push wpvskkyozppn' (#78) from push-wpvskkyozppn into main
Reviewed-on: #78
2026-10-15 10:38:58 +00:00
Eric Coissac 8296e3544c Simplify Hybrid layer implementation and update dependencies
Release / create-release (push) Successful in 2m31s
Release / build-linux-x86_64 (push) Successful in 9m24s
Release / build-macos-arm64 (push) Successful in 2m5s
ci.yml / build (pull_request) Successful in 3m51s
Dropped fields related to Hybrid layers for simplification and removed the unused `ahash` dependency. Updated dependency versions for `obidebruinj`, `obikmer`, and related crates, and cleaned up temporary artifacts.
v1.4.0
2026-10-15 12:23:15 +02:00
Eric Coissac 12afbe819c Refactor MPHF layer logic and evidence handling
This commit implements core lookup functionality for MPHF layers, including `hash` and `find_strict` methods. It introduces support for different evidence types (Exact, Hybrid, Approximate) within layers, clarifying lookup behavior and improving partition calculation logic in the cache. Additionally, dead code is removed and dependencies are updated.
2026-10-15 12:22:45 +02:00
Eric Coissac f9edefd135 Refactor query system for performance and memory reduction
This change introduces dynamic chunk sizing based on input size estimates, ensuring memory constraints are respected while allowing for smaller chunks that yield significant speed improvements. Additionally, index structures were optimized, resulting in drastic memory footprint reduction and faster query execution times.
2026-10-15 12:21:38 +02:00
Eric Coissac 025c40c0db Refactor chunking logic to enforce 128 k-mer limit
This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
2026-10-15 12:20:36 +02:00
Eric Coissac 8468319a63 Refactor unitig detection and graph traversal logic
Introduce new hashing mechanisms and support for directional unitig checks. Fix fragmentation issues and improve performance metrics through optimized graph construction and new benchmarking utilities.
2026-10-15 12:19:24 +02:00
Eric Coissac d32d86193f Refactor index metadata loading to use buffered I/O
This change replaces unbuffered file reads with buffered I/O operations to improve performance during index metadata loading. This includes implementing new methods for reading index metadata files and wrapping file handles in `BufReader` to optimize subsequent reads.
2026-10-15 12:17:51 +02:00
Eric Coissac 093d81c359 Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
2026-10-15 12:17:09 +02:00
Eric Coissac 09c80468b2 Refactor k-mer storage to use flat arrays
This change replaces the HashMap-based k-mer storage mechanism with a compact, flat array structure for `PartitionKmers`. This significantly reduces memory usage per partition and improves query execution time.

It also updates the batch processing logic to handle the new data structure, introduces detailed memory footprint calculations, and updates relevant query layer APIs. Test coverage has been expanded to validate correct partitioning and k-mer ordering.
2026-10-15 12:16:21 +02:00
Eric Coissac fe38424682 Implement batch lookup and chunked data management
Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline.

This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
2026-10-15 12:14:51 +02:00
Eric Coissac 8b87078478 Implement MPHF migration tool and sparse matrix permutation logic
Refactors sparse matrix handling to implement row permutation logic and introduces a standalone tool for migrating indexes from MPHF 1.x to 2.x formats.
2026-10-15 12:12:51 +02:00
Eric Coissac d686d39506 Refactor indexing logic and update core dependencies
This commit introduces significant updates to the indexing system, including the introduction of zero-copy types, modifications to the `Mphf` type definition, and new hybrid search strategies. Core algorithms are updated to handle new hashing types and improved error handling during pointer hash construction. Dependency versions are also updated across the board, including new security features and system-level crates.
2026-10-15 12:07:08 +02:00
Eric Coissac 3bd7fd4f41 Implement memory mapping for file construction
Refactors the building process to utilize memory mapping (`memmap2`) for file creation and introduces the necessary infrastructure for publishing built files atomically. This includes expanding the public API surface and cleaning up deprecated architectural components.
2026-10-15 12:02:33 +02:00
Eric Coissac c226744405 Refactor sampling algorithms and coverage metrics
Optimize family scanning and sampling plan calculation logic. Introduce new per-genome richness metrics and redefine the sampling fraction $r$ for improved analysis.
2026-10-15 12:01:15 +02:00
Eric Coissac 0e40e83fd1 Refactor probability formula notation
Update the mathematical formula for $P(X \geq r)$ to explicitly use summation notation and clearly format the probability expression.
2026-10-15 12:00:51 +02:00
Eric Coissac cb5fc202b6 Refactor sampling logic and introduce diagnostic outputs
This commit introduces new features for sampling, including detailed `SamplePlan` generation, Shannon entropy summaries, and site statistics reporting. It formalizes conditional distance matrix computation and clarifies the logic for family weighting and site selection.
2026-10-15 12:00:44 +02:00
coissac 2232628e58 Merge pull request 'Push kkkmrtvotrsu' (#77) from push-kkkmrtvotrsu into main
Reviewed-on: #77
2026-10-13 07:54:48 +00:00
Eric Coissac f29dcbcfb8 Implement advanced sampling and family annex logic
Release / create-release (push) Successful in 2m35s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 2m3s
ci.yml / build (pull_request) Successful in 4m8s
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data.

This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
v1.3.6
2026-10-13 08:57:18 +02:00
Eric Coissac 0d12494f1a Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
2026-10-13 08:54:21 +02:00
Eric Coissac 95bbc0b2ae Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
2026-10-13 08:52:37 +02:00
Eric Coissac fdfc1eb3dd Refactor siblings module and enhance sampling logic
Introduces filtering for scan families, handles genome exclusion arguments, refactors the `siblings` module architecture, and clarifies invariants for statistical calculations.
2026-10-13 08:45:26 +02:00
Eric Coissac 996ebaf977 Refactor algorithms to use configuration flags
Introduce boolean flags for `AlignmentReducer` and `PairwiseReducer` to control behavior. Update masking logic and introduce the `SurvivingFamily` structure for explicit data handling.
2026-10-13 08:44:19 +02:00
Eric Coissac cd06fa2ce3 Implement masking logic for pairwise reduction
Introduces masking based on `free_loss` and `no_ambiguity` flags within pairwise algorithms. Updates reducer initialization and adds tests to verify cardinality calculation.
2026-10-13 08:43:29 +02:00
Eric Coissac 1702addbf9 Refactor algorithms for unified sampling and reduction
Introduces a unified sampling trunk, new reducer traits, and refined logic for SNP distance calculation and session handling. Enforces strict parameter sharing across all algorithms to ensure consistent sampling.
2026-10-13 08:43:06 +02:00
Eric Coissac 0d1ecac233 Refactor k-mer indexing, enhance session features, and update documentation
This commit introduces significant refinements across the codebase. Core indexing logic is updated with hard constraints on k-mer length, statistical modeling clarifications, and the implementation of the Karlin & Ost theory for chance-collision statistics. Session management features are enhanced with robust memoization capabilities, atomic file operations, per-key locking, and memory mapping support. Additionally, documentation is updated, including a new citation style, and new mathematical notebook support is added.
2026-10-13 08:41:42 +02:00
Eric Coissac e715528d34 Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
2026-10-13 08:38:49 +02:00
Eric Coissac 9da0876ee6 Refactor indexing logic using hash-based minimizer selection.
Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
2026-10-13 08:34:51 +02:00
coissac 755cf85334 Merge pull request 'Push twnyxtspltux' (#76) from push-twnyxtspltux into main
Reviewed-on: #76
2026-09-12 05:57:25 +00:00
Eric Coissac 2693a6af09 Implement session persistence and checkpointing
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m21s
Release / build-macos-arm64 (push) Successful in 2m1s
ci.yml / build (pull_request) Successful in 3m45s
Introduces session persistence via the `--session` option, allowing sampled sets to be restored. Implements chunked dumps and shared checkpointing for alignment and tally artifacts, ensuring state restoration upon interruption. Refines sampling logic and introduces serialization mechanisms for state management.
v1.3.5
2026-09-12 07:48:51 +02:00
Eric Coissac 7861e886e7 Introduce session persistence and artifact caching
This change implements a robust session management system allowing users to persist distance calculation samples and parameters across invocations. It includes concurrency-safe directory locking, deterministic parameter encoding, artifact integrity checks, and caching mechanisms to skip resampling when valid data is available.
2026-09-12 07:46:55 +02:00
Eric Coissac a8bcf5ffac Refactor distance matrix calculation logic for consistency
Ensures that combined distance matrix calculations use the exact same site selection by retaining sampled data within Sankoff bundles. This involves refactoring post-sampling logic into shared functions, implementing `SankoffBundle` to reuse internal tally data, and adding validation to guarantee consistent site selection across all distance calculation paths.
2026-09-12 07:45:32 +02:00
Eric Coissac be838da256 Refine theoretical justification for alpha estimation.
Document limitations of various $\alpha$-estimation methods, clarifying that tree-based methods require a tree for accurate heterogeneity estimation and that `PartitionDispersion` avoids violating identifiability requirements.
2026-09-12 07:44:51 +02:00
Eric Coissac 020b391636 Implement gamma shape correction for SNP distance calculation
Introduces support for rate heterogeneity via a Poisson-Gamma mixture model. This includes new command-line options (`--gamma-shape`, `--gamma-shape auto`) to enable automatic estimation of the shape parameter $\alpha$ based on substitution counts across partitions. The correction is applied to the distance metric, with logic to disable the correction if variance checks fail.
2026-09-12 07:44:39 +02:00
coissac e846d35adb Merge pull request 'Feat: Introduce resource profiling and monitoring APIs' (#75) from push-sktolxuyskpp into main
Reviewed-on: #75
2026-09-11 07:51:03 +00:00
Eric Coissac c0e1968654 Feat: Introduce resource profiling and monitoring APIs
Release / create-release (push) Successful in 2m27s
ci.yml / build (pull_request) Successful in 3m52s
Release / build-linux-x86_64 (push) Successful in 7m50s
Release / build-macos-arm64 (push) Successful in 2m0s
This change introduces a comprehensive profiling system, allowing for periodic resource usage monitoring (CPU, I/O, memory) via a background thread. It also exposes new public APIs for budget management, locking, NUMA partitioning, and progress tracking.
v1.3.4
2026-09-11 09:50:09 +02:00
coissac e280b6d0a3 Merge pull request 'Implement NUMA topology detection and CPU budget capping' (#74) from push-tnkxqmwztnmm into main
Reviewed-on: #74
2026-09-11 05:23:42 +00:00
Eric Coissac 98dba1802d Implement NUMA topology detection and CPU budget capping
Release / create-release (push) Successful in 2m30s
ci.yml / build (pull_request) Successful in 3m50s
Release / build-linux-x86_64 (push) Successful in 8m23s
Release / build-macos-arm64 (push) Successful in 1m57s
This change introduces logic to detect NUMA topology, calculate per-node worker limits based on available CPU cores, and cap resource allocation to ensure worker counts respect the physical capacity of each node.
v1.3.3
2026-09-11 07:22:35 +02:00
coissac dd4285b269 Merge pull request 'Introduce CPU budget and dynamic thread limiting' (#73) from push-lztkokoxqpuw into main
Reviewed-on: #73
2026-09-10 15:10:19 +00:00
Eric Coissac b66a488277 Introduce CPU budget and dynamic thread limiting
Release / create-release (push) Successful in 3m4s
ci.yml / build (pull_request) Successful in 4m17s
Release / build-linux-x86_64 (push) Successful in 8m11s
Release / build-macos-arm64 (push) Successful in 2m2s
This change implements mechanisms to enforce resource limits based on the available CPU budget, including setting a hard CPU ceiling and calculating effective thread counts for CLI, query, and superkmer operations.
v1.3.2
2026-09-10 17:08:32 +02:00