Commit Graph
28 Commits
Author SHA1 Message Date
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00
Eric Coissac 0134c69cf4 docs: clarify VCF pseudo-alignment format and update obikmer dependency
ci.yml / build (pull_request) Successful in 3m40s
Release / create-release (push) Successful in 1m7s
Release / build-macos-arm64 (push) Successful in 1m52s
Release / build-linux-x86_64 (push) Successful in 6m41s
Update the VCF export documentation to reflect that `-` represents an overlapping deletion (extra allele `*`) and introduce the `<ABS>` symbolic allele for absent families. Add a new dedicated documentation file for the VCF pseudo-alignment format, remove obsolete descriptions from phylo.md, and fix benchmark scripts to handle errors and directory creation correctly.

- Update `obikmer` dependency from 1.4.1 to 1.4.2 in Cargo.toml and Cargo.lock
- Add `UserDocMD/formats/vcf.md` describing VCF pseudo-alignment structure, alleles, and genotypes
- Remove obsolete VCF descriptions from `UserDocMD/usage/phylo.md` and update ploidy/free-loss notes
- Update `DevDocMD/theory/evolutionary_distances.md` to clarify `-` as overlapping deletion and `<ABS>` for others
- Fix `benchmark/index_one_count.sh` to create index directory and check read command errors
- Fix `benchmark/index_one_presence.sh` to create index directory and check simulation errors
- Update `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` to use `<ABS>` for absent families instead of `*`
2026-11-09 20:59:28 +11:00
Eric Coissac 5368c98875 refactor: replace no_ambiguity bool with AmbiguityPolicy enum
Replace the binary `no_ambiguity` flag with an `AmbiguityPolicy` enum (`Keep`, `Forbid`, `OnlyDiploid`) to support multiple ambiguity conventions. This change updates CLI arguments, internal logic, and persistent storage formats to distinguish between keeping all genomes, forbidding ambiguous ones, or restricting analysis to diploid sites only.

- Add `src/obikphylo/src/siblings/ambiguity.rs` defining the `AmbiguityPolicy` enum with variants `Keep`, `Forbid`, and `OnlyDiploid`, including methods for header flags and determining if a genome is dropped.
- Update `PhyloArgs` in `args.rs` to derive `AmbiquityPolicy` from CLI flags and construct `AnnexSelection` for sibling annexes.
- Modify `SamplingSpec` in `subsample.rs` to store `ambiguity: AmbiguityPolicy` instead of `no_ambiguity`, updating masking and scalar value logic accordingly.
- Update `SiblingExt` in `sibling_ext.rs` so `build_sibling_annex` accepts an `AnnexSelection` to build only requested derived annexes, and `family_overlap` uses the specific policy.
- Refactor `FamilyOverlap` in `family_overlap.rs` to store and use `AmbiguityPolicy`, replacing boolean checks with policy-based carrier determination.
- Update `FamilyScalarAnnex` in `family_scalar_annex.rs` to store `AmbiguityPolicy` in headers, enabling distinct file names for each policy variant.
- Modify `quartet_annex.rs` to generate distinct file names based on `AmbiguityPolicy`, ensuring no overlap between Keep, Forbid, and OnlyDiploid annexes.
- Update `family_annexes.rs` to use `AnnexSelection` for building entropy, quartet, and pair weight annexes under specific policies instead of a single boolean convention.
- Refactor algorithm modules (`entropy.rs`, `masking.rs`, `pair_weight.rs`, `quartet.rs`, `snp_distance.rs`) to accept `AmbiguityPolicy` arguments, replacing boolean checks with policy-specific logic (e.g., `OnlyDiploid` masking).
- Update documentation in `phylo.md` and `evolutionary_distances.md` to describe the new `--only-diploid` option, the behavior of `--sibling-annex` with specific policies, and the separate storage of entropies per convention.
2026-11-09 20:59:09 +11:00
Eric Coissac b55123a376 feat: add --only-diploid flag to filter for diploid genotypes only
Introduce a new `--only-diploid` option that restricts analysis to genomes with one or two family members (base or two-base IUPAC codes), treating others as missing data. This complements the existing `--no-ambiguity` behavior by providing a stricter filter for diploid-compatible sites, while ensuring conflict detection between the two flags.

- Add `only_diploid` boolean argument to `PhyloArgs` in `src/obikmer/src/cmd/phylo/args.rs`, documenting its effect on pseudo-alignment and its conflict with `--no-ambiguity`.
- Propagate `only_diploid` through `SamplingSpec` in `src/obikmer/src/cmd/phylo/mod.rs` and update the `run` function signature to accept the new argument.
- Implement masking logic in `src/obikphylo/src/siblings/algorithms/masking.rs`: modify `masked_state` to accept `only_diploid`, add `only_diploid_keeps_up_to_two_members` helper, and update `genome_mask` documentation.
- Update `sample_layer` in `src/obikphylo/src/siblings/algorithms/subsample.rs` to use `spec.only_diploid` for masking decisions and update `SamplingSpec` documentation.
- Adjust `tally_2genomes` in `src/obikphylo/src/siblings/algorithms/snp_distance.rs` to pass the new `only_diploid` parameter to `masked_state`.
- Update `DevDocMD/theory/evolutionary_distances.md` to describe the new `--only-diploid` option, the `masked_state` rule (`count_ones() > 2`), and clarify toy index behavior.
- Update `UserDocMD/usage/phylo.md` to document `--only-diploid`, `--quartet`, and `--pair-balance` behaviors, clarifying that diploid choice is a convention and explaining how three/four-allele genomes are treated as missing.
2026-11-09 20:58:21 +11:00
Eric Coissac b132f0847f feat: add VCF export and refine alignment output infrastructure
Introduce `--vcf` and `--fasta` options for phylo commands, allowing users to
export pseudo-alignment data as VCF or FASTA. Implement comprehensive VCF logic
including stable virtual coordinates, canonical ID generation, and flexible
genotype representation based on ambiguity settings. Refactor internal alignment
structures to track k-mer families and site loci, ensuring consistent data flow
from scanning through to output generation.

- Add `src/obikmer/src/cmd/phylo/alignment_out.rs` with `create` and
  `write_alignment_outputs` to handle file creation and dispatching of FASTA/VCF
  formats based on `PhyloArgs`.
- Add `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` implementing
  VCF-specific logic: `write_vcf`, `alignment`, `cell_alleles`, `majority_base`,
  and site construction via `site` using fixed $K=11$. Includes helpers for
  IUPAC state inversion, genotype ordering, and stable coordinate generation.
- Update `src/obikmer/src/cmd/phylo/args.rs` to add `fasta` and `vcf` flags
  and a new `alignment_output()` method that returns true if any output format
  is requested. Add `pseudo_alignment` field documentation.
- Modify `src/obikmer/src/cmd/phylo/mod.rs` to use `alignment_output()` for
  determining alignment needs, update error messages for `--n-sites`, and import
  `write_alignment_outputs` from the new module.
- Refactor `src/obikphylo/src/siblings/algorithms/alignment.rs` by adding
  `SiteLocus` struct (partition, layer, family_idx, family_id) and updating
  `SnpAlignment` and `AlignmentReducer` to track sites. Update `new`, `finish`,
  and `reduce` to populate site information from `SurvivingFamily`.
- Update `src/obikphylo/src/siblings/algorithms/iqtree.rs`: `alignment` now
  returns `SnpAlignment` with empty sites; update `drop_ascertainment_noninformative`
  and `recode_symbols_as_absent` to handle the new `sites` field.
- Update `src/obikphylo/src/siblings/algorithms/family_scan.rs`: `GeneratedBatch`
  now includes a `kmers` field; `scan_layer_families` accepts `CanonicalKmer`
  and passes it to closures instead of masks.
- Update `src/obikphylo/src/siblings/algorithms/subsample.rs`: `SurvivingFamily`
  gains a `kmer` field; `sample_layer` closure accepts and uses this k-mer.
- Update `src/obikphylo/src/siblings/algorithms/snp_distance.rs`: `tally_2genomes`
  initializes `SurvivingFamily` with explicit `CanonicalKmer` values.
- Update `src/obikphylo/src/siblings/algorithms/stats.rs`: `sibling_annex_stats`
  signature adds `_kmer` parameter.
- Update `src/obikphylo/src/siblings/algorithms/family_annexes.rs`: closures in
  `ensure_layer_family_annexes` and `ensure_layer_pair_weight_annexes` now accept
  `_kmer`.
- Update `src/obikphylo/src/siblings/algorithms/pairwise.rs`: `ambiguous_vs_single`
  initializes k-mer using `CanonicalKmer::from_raw_unchecked(0)`.
- Add `iupac_state` function to `src/obikphylo/src/siblings/algorithms/masking.rs`
  mapping DNA characters to binary codes.
- Update `src/obikphylo/src/siblings/algorithms/mod.rs` to export `SiteLocus`
  and include new `alignment_vcf` module.
- Add `family_id` and `write_family_id_ascii` methods to
  `src/obikseq/src/kmer.rs` for stable k-mer family identification.
- Add tests `family_id_partitions_kmers_into_families` and
  `family_id_ascii_masks_centre` to `src/obikseq/src/tests/kmer.rs`.
- Update `src/obikphylo/src/siblings/mod.rs` imports to include new algorithm
  exports.
2026-11-09 20:58:05 +11:00
Eric Coissac 7c7d7615cb Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
2026-10-16 15:59:40 +02:00
Eric Coissac 0e40e83fd1 Refactor probability formula notation
Update the mathematical formula for $P(X \geq r)$ to explicitly use summation notation and clearly format the probability expression.
2026-10-15 12:00:51 +02:00
Eric Coissac cb5fc202b6 Refactor sampling logic and introduce diagnostic outputs
This commit introduces new features for sampling, including detailed `SamplePlan` generation, Shannon entropy summaries, and site statistics reporting. It formalizes conditional distance matrix computation and clarifies the logic for family weighting and site selection.
2026-10-15 12:00:44 +02:00
Eric Coissac f29dcbcfb8 Implement advanced sampling and family annex logic
Release / create-release (push) Successful in 2m35s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 2m3s
ci.yml / build (pull_request) Successful in 4m8s
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data.

This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
2026-10-13 08:57:18 +02:00
Eric Coissac 0d12494f1a Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
2026-10-13 08:54:21 +02:00
Eric Coissac 0d1ecac233 Refactor k-mer indexing, enhance session features, and update documentation
This commit introduces significant refinements across the codebase. Core indexing logic is updated with hard constraints on k-mer length, statistical modeling clarifications, and the implementation of the Karlin & Ost theory for chance-collision statistics. Session management features are enhanced with robust memoization capabilities, atomic file operations, per-key locking, and memory mapping support. Additionally, documentation is updated, including a new citation style, and new mathematical notebook support is added.
2026-10-13 08:41:42 +02:00
Eric Coissac e715528d34 Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
2026-10-13 08:38:49 +02:00
Eric Coissac 9da0876ee6 Refactor indexing logic using hash-based minimizer selection.
Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
2026-10-13 08:34:51 +02:00
Eric Coissac 2693a6af09 Implement session persistence and checkpointing
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m21s
Release / build-macos-arm64 (push) Successful in 2m1s
ci.yml / build (pull_request) Successful in 3m45s
Introduces session persistence via the `--session` option, allowing sampled sets to be restored. Implements chunked dumps and shared checkpointing for alignment and tally artifacts, ensuring state restoration upon interruption. Refines sampling logic and introduces serialization mechanisms for state management.
2026-09-12 07:48:51 +02:00
Eric Coissac 7861e886e7 Introduce session persistence and artifact caching
This change implements a robust session management system allowing users to persist distance calculation samples and parameters across invocations. It includes concurrency-safe directory locking, deterministic parameter encoding, artifact integrity checks, and caching mechanisms to skip resampling when valid data is available.
2026-09-12 07:46:55 +02:00
Eric Coissac a8bcf5ffac Refactor distance matrix calculation logic for consistency
Ensures that combined distance matrix calculations use the exact same site selection by retaining sampled data within Sankoff bundles. This involves refactoring post-sampling logic into shared functions, implementing `SankoffBundle` to reuse internal tally data, and adding validation to guarantee consistent site selection across all distance calculation paths.
2026-09-12 07:45:32 +02:00
Eric Coissac 020b391636 Implement gamma shape correction for SNP distance calculation
Introduces support for rate heterogeneity via a Poisson-Gamma mixture model. This includes new command-line options (`--gamma-shape`, `--gamma-shape auto`) to enable automatic estimation of the shape parameter $\alpha$ based on substitution counts across partitions. The correction is applied to the distance metric, with logic to disable the correction if variance checks fail.
2026-09-12 07:44:39 +02:00
Eric Coissac 93fe838f97 Remove in-place flag, optimize presence counting, and handle NUMA panics
The `obikmer select` CLI no longer supports in-place index rewriting; the `--output` flag is now required, with benchmarks updated to use temporary directories for atomic replacement. Added `--dense` and `--force-copy` flags. Introduced `batch_presence_counts` to compute presence counts across multiple column groups in a single pass, eliminating redundant I/O. Refactored the aggregation pipeline to branch on layer content, applying the optimized batched counting for `Presence` layers. Enhanced the NUMA runner to catch worker panics, track them, and re-raise after thread join to prevent indefinite blocking.
2026-08-28 23:16:37 +02:00
Eric Coissac caf59b5658 Update documentation for CLI command renames and new commands
Renames the reindex command to convert across all usage guides and navigation menus. Adds documentation for the newly introduced name-tree command. Updates flag references, such as replacing --metric with --distance in pack and phylo modules, and adjusts MkDocs configuration to match the revised structure. All changes are strictly limited to documentation and configuration files.
2026-08-28 23:06:05 +02:00
Eric Coissac b0890d1781 Add sparse storage variant to PersistentCompactIntMatrix
Introduce a new `Sparse` format alongside existing `Columnar` and `Packed` variants, enabling optimized row-major pairwise counting for distance and similarity metrics via the `CountPartials` trait. Update storage detection priorities, extend matrix dispatch logic to sparse backends, and correct diagonal/off-diagonal formulas in bit matrix partial computations. Expand layer APIs with format-agnostic `nonzero_iter`, update usage documentation for the `--sparse` flag, and add comprehensive tests verifying roundtrip integrity and metric equivalence against dense implementations.
2026-08-28 19:06:39 +02:00
Eric Coissac 128db64564 feat(phylo): add --iqtree-min-freq to filter rare nucleotide states
Introduces --iqtree-min-freq (default 0.001) to treat low-frequency nucleotide states as missing data during IQ-TREE alignment generation when --free-loss is active. This triggers a recoding pass that folds rare states into the missing symbol, followed by non-informative site removal and alphabet recomputation to maintain output consistency. The change also adds Sankoff model configuration files and updates related tests and documentation.
2026-08-17 11:06:03 +02:00
Eric Coissac 9654201885 feat: introduce _iqtree_states.csv for compact symbol mapping
Generates a new CSV output that maps IQ-TREE's compact state symbols to canonical states alongside full-precision empirical frequencies. Updates documentation to clarify that state frequencies sum to 1.0 by design and documents conditional behavior under `--free-loss`. Includes unit tests verifying absent state exclusion, frequency summation, and CSV structure. Also restricts entropy annex resolution to non-monomorphic minorants to eliminate redundant per-genome checks.
2026-08-17 09:38:26 +02:00
Eric Coissac 7bac0f3850 large refactoring 2026-08-17 09:28:53 +02:00
Eric Coissac 3da501349b docs: clarify phylogenetic output files and tool usage
Restructure the output files section into categorized subsections with tables. Add explicit mappings between command-line options and generated files. Define CSV matrix conventions, clarify mathematical formulas for distance calculations, and document execution commands for external phylogenetic tools.
2026-08-16 13:17:51 +02:00
Eric Coissac eee71430a4 add name-tree command and fix --free-loss cost matrix
Introduce the obikmer name-tree subcommand to map numeric leaf labels in phylogenetic tree exports back to taxon names using a reference FASTA file. Correct the --free-loss flag behavior by removing cardinality transition costs from pairwise cost calculations, ensuring sibling gains and losses are priced identically to whole-family events. Update documentation, configuration parameters, and add reference phylogenetic data files.
2026-08-16 13:11:06 +02:00
Eric Coissac 8615da59a8 Add phylogenetic CLI options for family overlap and missing data
Introduces CLI flags for computing pairwise family overlap matrices and filtering genomes below a shared family threshold. Adds a free-loss mode that recodes locus non-detection states to missing data symbols in Sankoff-calibrated alignments, resolving ascertainment bias handling for IQ-TREE. Updates empirical transition parameters, removes the legacy model asset, and extends output writers for CSV diagnostics, FASTA pseudo-alignments, and Newick trees.
2026-08-16 11:54:43 +02:00
Eric Coissac dd889854cb rename distance subcommand to phylo
Rename the distance CLI subcommand to phylo across the codebase, documentation, and build configurations. Relocate source files from cmd/distance/ to a dedicated cmd/phylo/ module, update all internal routing references, and adjust benchmark scripts and Makefile targets to reflect the new command name.
2026-08-15 10:07:03 +02:00
Eric Coissac 6acafa7f2c docs: add obikmer user guide and MkDocs build configuration
Introduces a comprehensive documentation set covering theoretical foundations, CLI usage, installation, and system architecture. Adds MkDocs configuration and Makefile targets to generate, serve with live reload, and clean the documentation site. Includes citation styles and bibliography files for academic references.
2026-08-13 17:19:01 +02:00