Commit Graph
531 Commits
Author SHA1 Message Date
Eric Coissac 7f829ea557 docs: clarify merge re-merge behavior and input file handling
Update documentation to explain that merged indices can serve as sources if at least one source is unmerged, and that the resulting column order follows the base index first. Clarify that explicitly named files are always read regardless of extension, with format detection based on content.

- refactor: prevent I/O errors when re-merging packed outputs by restricting `choose_base` to valid destinations via new `can_be_destination` check
- fix: handle zero-layer partitions in `build_index_layer` and `remove_build_artifacts` to skip unnecessary operations
- feat: allow `PathIter` to treat unrecognized extensions as explicit files, ensuring they are not silently dropped during directory expansion
- test: add comprehensive tests for `PathIter` including extension recognition, explicit file handling, and stdin sentinel behavior
- test: update smoke tests to verify empty partitions, `.fna.gz` indexing, and refusal of merging only packed indexes
- chore: remove obsolete cargo build environment variables from `run.sh` script
2026-11-19 07:10:58 +01:00
Eric Coissac 07c0a54e04 feat: add compatibility testing scripts and update dependencies
Add `compat_ptrhash.sh` and `compat_ptrhash2.sh` to facilitate determinism control and incremental merge testing between obikmer binaries. Update `tikv-jemallocator` dependency version in `Cargo.toml` and refresh `Cargo.lock` with new crates and checksums.

- Add scripts/docker_debug/compat_ptrhash.sh for compatibility tests (T0-T4) comparing ptr_hash versions using Docker and md5sum
- Add scripts/docker_debug/compat_ptrhash2.sh for determinism control and incremental merge operations between binaries A and B
- Update src/obikmer/Cargo.toml to change tikv-jemallocator version from 0.7.0 to 0.5.4 under musl target cfg
- Update src/Cargo.lock by adding new crates (anes, block-pseudorand, criterion, plotters, etc.) and updating existing package versions/checksums
2026-11-19 07:10:37 +01:00
Eric Coissac af525cd268 feat: switch musl builds to jemalloc allocator for stability
Update Cargo.lock to remove unused dependencies and update tikv-jemallocator. Configure src/obikmer/Cargo.toml to use jemalloc for static musl builds, addressing fragmentation issues with the default multi-threaded allocator. Update main.rs to conditionally set Jemalloc as the global allocator when targeting musl.

- Remove unused dependencies from Cargo.lock: anes, block-pseudorand, chiapos-chacha8, criterion, criterion-plot, itertools (and its either dep), nanorand, hermit-abi, is-terminal, plotters ecosystem, tinytemplate deps, voracious_radix_sort rayon dep, and web-sys deps.
- Update tikv-jemallocator version from 0.5.4 to 0.7.0 in Cargo.lock.
- Add tikv-jemallocator = "0.7.0" to musl-specific dependencies in src/obikmer/Cargo.toml.
- Set Jemalloc as the global allocator in main.rs for static musl builds.
2026-11-19 07:10:20 +01:00
Eric Coissac d0668eb167 feat: add columnar sparse matrix support and windowed transposition
Introduce `sparse_columnar` and `sparse_intmatrix_columnar` modules to enable
building sparse matrices directly from columnar data without dense intermediates.
Add a `transpose` module with `RowBatch` and `transpose_windows` for efficient
windowed column-to-row transposition based on density checks.

- Add `src/obicompactvec/src/bitmatrix/sparse_columnar.rs` with `build_from_columnar`,
  `build_from_columnar_with`, and `read_bit_window` to build sparse bit matrices
  from columnar files using windowed reads.
- Add `src/obicompactvec/src/sparse_intmatrix_columnar.rs` with `build_from_columnar`,
  `build_from_columnar_with`, `invalid`, and `read_int_window` for integer matrix
  construction from PCIV columns, handling sentinel values and overflow.
- Add `src/obicompactvec/src/transpose.rs` containing `RowBatch<V>` struct and
  `transpose_windows` function for dynamic window sizing and memory-bounded
  column-to-row transposition.
- Update `src/obicompactvec/src/bitmatrix/sparse.rs` to use `packed_bit_matrix_n_cols`
  and `MatrixMeta` for validation, ensuring stale columnar data is transposed directly
  into a single `matrix.pbmx` file while removing old column files.
- Update `src/obicompactvec/src/sparse_intmatrix.rs` to use `packed_int_matrix_n_cols`
  for column count validation, making `pack_sparse_compact_int_matrix` idempotent and
  avoiding dense intermediates by transposing columnar data directly.
- Expose `packed_bit_matrix_n_cols` in `src/obicompactvec/src/bitmatrix/packed.rs` and
  make `packed_int_matrix_n_cols` public in `src/obicompactvec/src/intmatrix.rs`.
- Add `sparse_columnar` module to `src/obicompactvec/src/bitmatrix/mod.rs` and export
  `PersistentBitMatrixBuilder` and `batch_presence_counts`.
- Add `sparse_intmatrix_columnar` and `transpose` modules to `src/obicompactvec/src/lib.rs`.
- Add `transpose_columnar` test module in `src/obicompactvec/src/tests/mod.rs` with
  tests verifying columnar transpose matches dense route output.
2026-11-19 07:08:41 +01:00
Eric Coissac 4e98de4b77 feat: add BitSliceView::for_each_set_in and optimize build_from_dense
Introduce a new `views_scan` module containing `BitSliceView::for_each_set_in`, which iterates over set bits in a specified range using optimized word traversal. Update `build_from_dense` to use this method for batched, ordered column access, ensuring sorted row output. Add comprehensive tests for the new iteration logic and boundary handling.

- Add `src/obicompactvec/src/views_scan.rs` with `BitSliceView::for_each_set_in`, which iterates over a `Range<usize>` calling a closure for each set bit while optimizing sparse traversal via `word.trailing_zeros()`.
- Add `src/obicompactvec/src/tests/views_scan.rs` with tests `words`, `naive`, `scanned`, and helpers verifying that `for_each_set_in` matches naive iteration, handles empty/reversed ranges correctly, and rejects out-of-bounds ranges.
- Modify `build_from_dense` in `src/obicompactvec/src/bitmatrix/sparse.rs` to replace `nonzero_triples` with `dense.col_view(c).for_each_set_in`, ensuring column visits are batched and ordered for sorted row processing.
- Update module structure in `src/obicompactvec/src/lib.rs` and `src/obicompactvec/src/tests/mod.rs` to include the new `views_scan` module.
2026-11-19 07:08:07 +01:00
Eric Coissac 03eaa4544a feat: add pack optimization prompt, debug infrastructure, and monitoring scripts
Add a prompt template for optimizing the `pack` step and a suite of Docker-based
debug tools including build, run, and monitoring scripts. Enhance `pack_matrices`
with LFD partition sizing and detailed timing reports, and update `stop` with
additional system statistics.

- Add `prompts/pack_optimization.md` to guide optimization of the `pack` step,
  specifying context, performance measurements, and deliverable structures.
- Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets,
  `build.sh` for image creation, `run.sh` for parallel indexing and merging,
  `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and
  `gen_fasta_real.py` for real genome generation with `read_fna`.
- Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics,
  writing samples to CSV and summary reports.
- Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing
  memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize.
- Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and
  `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition
  sizing and clarify documentation regarding `stat()` proxy bytes.
- Modify `src/obisys/src/stage.rs`: update `stop` definition to include new
  variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and
  logic checking for swap usage.
2026-11-19 07:07:50 +01:00
Eric Coissac 24291a4484 refactor: remove unused dependencies and simplify logic
Remove obsolete dependencies across multiple crates to clean up the build graph and update `obikdump` to use `try_update`. Add dev-dependencies for tracing in `obiread` and clarify memoization logic in `obiksession`.

- Remove `obifastwrite`, `tracing`, `obidebruinj`, `obikentropy`, `obipipeline`, `obidex`, and `obiskbuilder` from `Cargo.lock`.
- Remove `obifastwrite` and `tracing` dependencies from `src/obidebruinj/Cargo.toml`.
- Update `dump` function in `src/obikdump/src/dump.rs` to use `try_update` instead of `fetch_update` for updating remaining counts.
- Remove `obitaxonomy`, `obidebruinj`, `obipipeline`, and `obikentropy` dependencies from `src/obikindex/Cargo.toml`.
- Remove `obiksession` dependency from `src/obikmer/Cargo.toml`.
- Remove `obiskbuilder` dependency from `src/obikphylo/Cargo.toml` and update `kodama` version to "0.3.0".
- Remove `bytes` dependency from `src/obikrope/Cargo.toml` and set `criterion2` version to "3" with `cargo_bench_support`.
- Add `tracing` and `tracing-subscriber` as dev-dependencies in `src/obiread/Cargo.toml`.
- Remove `lazy_static` dependency from `src/obiskbuilder/Cargo.toml` and add `obikseq` as a test-utils dev-dependency.
- Clarify `memo_key_from_debug` in `src/obiksession/src/memoize.rs` with comments explaining Debug-derived field usage.
2026-11-19 07:07:16 +01:00
coissac cbc344e4b4 Merge pull request 'Push txuyrtlmyoyl' (#81) from push-txuyrtlmyoyl into main
Reviewed-on: #81
2026-11-17 14:38:33 +00:00
Eric Coissac 07bc5fc634 docs: update sparse packing strategy and benchmark scripts
ci.yml / build (pull_request) Successful in 4m53s
Release / create-release (push) Successful in 28s
Release / build-linux-x86_64 (push) Successful in 6m46s
Release / build-macos-arm64 (push) Successful in 4m8s
Update documentation to reflect that sparse packing now maps all columns of a layer at once and packs layers sequentially, reversing the previous choice to avoid materializing a packed copy. This change introduces an extra write/read cycle for the dense matrix, increasing pack benchmark time from 1.11s to 1.60s.

- Modify `pack_sparse_bit_matrix` in `src/obicompactvec/src/bitmatrix/sparse.rs` to always convert via the dense packed format, simplifying logic by removing conditional handling of existing files and ensuring a single mapping.
- Update `pack_sparse_compact_int_matrix` in `src/obicompactvec/src/sparse_intmatrix.rs` to exclusively handle conversion from the dense packed format (`matrix.pcmx`) to sparse, removing columnar format handling and simplifying file cleanup.
- Adjust `scripts/bench_merge_oracle.sh` to extract `PACK_WALL` metrics via `stage_field` for merge operations, updating grep patterns and output messages to report pack wall time.
- Revise `DevDocMD/implementation/merge_mmap_audit.md` and `merge_scaling.md` to document the new sequential packing strategy, noting that `N_part` scaling applies only to incremental merges and that `obisys::MemoryBudget` is no longer used by merge.
- Bump package version in `src/obikmer/Cargo.toml` from 1.4.3 to 1.4.4.
v1.4.4
2026-11-17 15:35:41 +01:00
Eric Coissac 1f199c69db feat: implement in-memory column builders and refactor merge partitioning
Introduce `MemBitVecBuilder` and `MemCompactIntVecBuilder` to construct columns in RAM before writing, reducing the number of open file descriptors during merge operations. Refactor `partition_merge.rs` to use these new builders via `add_mem_col`, decoupling live memory mappings from source counts to prevent `SIGBUS` errors.

- Add `mem_builder` module with `MemBitVecBuilder` and `MemCompactIntVecBuilder` structs for in-memory column construction and PCIV file writing.
- Extend `MatrixBuilder` with `add_mem_col` and `n_cols` methods, updating `ColBuilder` enum to support `MemBit` and `MemInt` variants.
- Refactor `partition_merge.rs` to use `add_mem_col` instead of `add_col`, introducing `run_pass2` for batch processing and optimizing destination layer handling.
- Update `format.rs` with `pciv_index_layout`, `write_pciv_header`, and `write_pciv_tail` functions to manage PCIV index layout and header/tail sections.
- Modify `mmap_file.rs` to use `write_std_header` and introduce `write_built` for standardized file writing and flushing logic.
- Make `MAGIC` constant public in `bitvec.rs` and update documentation for memory mapping alignment and structure.
- Add test modules `tests/matrix_builder.rs` and `tests/mem_builder.rs` to verify column counts and byte-identical output of new builders against persistent versions.
- Update `mkdocs.yml` navigation to include "Merge mmap audit (2026-11)" documentation entry.
- Add implementation audit documentation in `DevDocMD/implementation/merge_mmap_audit.md` detailing memory mapping failures and refactoring plans.
- Add merge testing scripts `scripts/smoke_test_merge.sh` and `scripts/bench_merge_oracle.sh` for validation and benchmarking.
- Bump `obikmer` package version from 1.4.2 to 1.4.3 in `Cargo.toml`.
2026-11-17 15:16:41 +01:00
coissac b35127e4e3 Merge pull request 'Push smsmsoyssmnr' (#80) from push-smsmsoyssmnr into main
Reviewed-on: #80
2026-11-09 10:01:21 +00:00
Eric Coissac 0134c69cf4 docs: clarify VCF pseudo-alignment format and update obikmer dependency
ci.yml / build (pull_request) Successful in 3m40s
Release / create-release (push) Successful in 1m7s
Release / build-macos-arm64 (push) Successful in 1m52s
Release / build-linux-x86_64 (push) Successful in 6m41s
Update the VCF export documentation to reflect that `-` represents an overlapping deletion (extra allele `*`) and introduce the `<ABS>` symbolic allele for absent families. Add a new dedicated documentation file for the VCF pseudo-alignment format, remove obsolete descriptions from phylo.md, and fix benchmark scripts to handle errors and directory creation correctly.

- Update `obikmer` dependency from 1.4.1 to 1.4.2 in Cargo.toml and Cargo.lock
- Add `UserDocMD/formats/vcf.md` describing VCF pseudo-alignment structure, alleles, and genotypes
- Remove obsolete VCF descriptions from `UserDocMD/usage/phylo.md` and update ploidy/free-loss notes
- Update `DevDocMD/theory/evolutionary_distances.md` to clarify `-` as overlapping deletion and `<ABS>` for others
- Fix `benchmark/index_one_count.sh` to create index directory and check read command errors
- Fix `benchmark/index_one_presence.sh` to create index directory and check simulation errors
- Update `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` to use `<ABS>` for absent families instead of `*`
v1.4.2
2026-11-09 20:59:28 +11:00
Eric Coissac 5368c98875 refactor: replace no_ambiguity bool with AmbiguityPolicy enum
Replace the binary `no_ambiguity` flag with an `AmbiguityPolicy` enum (`Keep`, `Forbid`, `OnlyDiploid`) to support multiple ambiguity conventions. This change updates CLI arguments, internal logic, and persistent storage formats to distinguish between keeping all genomes, forbidding ambiguous ones, or restricting analysis to diploid sites only.

- Add `src/obikphylo/src/siblings/ambiguity.rs` defining the `AmbiguityPolicy` enum with variants `Keep`, `Forbid`, and `OnlyDiploid`, including methods for header flags and determining if a genome is dropped.
- Update `PhyloArgs` in `args.rs` to derive `AmbiquityPolicy` from CLI flags and construct `AnnexSelection` for sibling annexes.
- Modify `SamplingSpec` in `subsample.rs` to store `ambiguity: AmbiguityPolicy` instead of `no_ambiguity`, updating masking and scalar value logic accordingly.
- Update `SiblingExt` in `sibling_ext.rs` so `build_sibling_annex` accepts an `AnnexSelection` to build only requested derived annexes, and `family_overlap` uses the specific policy.
- Refactor `FamilyOverlap` in `family_overlap.rs` to store and use `AmbiguityPolicy`, replacing boolean checks with policy-based carrier determination.
- Update `FamilyScalarAnnex` in `family_scalar_annex.rs` to store `AmbiguityPolicy` in headers, enabling distinct file names for each policy variant.
- Modify `quartet_annex.rs` to generate distinct file names based on `AmbiguityPolicy`, ensuring no overlap between Keep, Forbid, and OnlyDiploid annexes.
- Update `family_annexes.rs` to use `AnnexSelection` for building entropy, quartet, and pair weight annexes under specific policies instead of a single boolean convention.
- Refactor algorithm modules (`entropy.rs`, `masking.rs`, `pair_weight.rs`, `quartet.rs`, `snp_distance.rs`) to accept `AmbiguityPolicy` arguments, replacing boolean checks with policy-specific logic (e.g., `OnlyDiploid` masking).
- Update documentation in `phylo.md` and `evolutionary_distances.md` to describe the new `--only-diploid` option, the behavior of `--sibling-annex` with specific policies, and the separate storage of entropies per convention.
2026-11-09 20:59:09 +11:00
Eric Coissac b55123a376 feat: add --only-diploid flag to filter for diploid genotypes only
Introduce a new `--only-diploid` option that restricts analysis to genomes with one or two family members (base or two-base IUPAC codes), treating others as missing data. This complements the existing `--no-ambiguity` behavior by providing a stricter filter for diploid-compatible sites, while ensuring conflict detection between the two flags.

- Add `only_diploid` boolean argument to `PhyloArgs` in `src/obikmer/src/cmd/phylo/args.rs`, documenting its effect on pseudo-alignment and its conflict with `--no-ambiguity`.
- Propagate `only_diploid` through `SamplingSpec` in `src/obikmer/src/cmd/phylo/mod.rs` and update the `run` function signature to accept the new argument.
- Implement masking logic in `src/obikphylo/src/siblings/algorithms/masking.rs`: modify `masked_state` to accept `only_diploid`, add `only_diploid_keeps_up_to_two_members` helper, and update `genome_mask` documentation.
- Update `sample_layer` in `src/obikphylo/src/siblings/algorithms/subsample.rs` to use `spec.only_diploid` for masking decisions and update `SamplingSpec` documentation.
- Adjust `tally_2genomes` in `src/obikphylo/src/siblings/algorithms/snp_distance.rs` to pass the new `only_diploid` parameter to `masked_state`.
- Update `DevDocMD/theory/evolutionary_distances.md` to describe the new `--only-diploid` option, the `masked_state` rule (`count_ones() > 2`), and clarify toy index behavior.
- Update `UserDocMD/usage/phylo.md` to document `--only-diploid`, `--quartet`, and `--pair-balance` behaviors, clarifying that diploid choice is a convention and explaining how three/four-allele genomes are treated as missing.
2026-11-09 20:58:21 +11:00
Eric Coissac b132f0847f feat: add VCF export and refine alignment output infrastructure
Introduce `--vcf` and `--fasta` options for phylo commands, allowing users to
export pseudo-alignment data as VCF or FASTA. Implement comprehensive VCF logic
including stable virtual coordinates, canonical ID generation, and flexible
genotype representation based on ambiguity settings. Refactor internal alignment
structures to track k-mer families and site loci, ensuring consistent data flow
from scanning through to output generation.

- Add `src/obikmer/src/cmd/phylo/alignment_out.rs` with `create` and
  `write_alignment_outputs` to handle file creation and dispatching of FASTA/VCF
  formats based on `PhyloArgs`.
- Add `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` implementing
  VCF-specific logic: `write_vcf`, `alignment`, `cell_alleles`, `majority_base`,
  and site construction via `site` using fixed $K=11$. Includes helpers for
  IUPAC state inversion, genotype ordering, and stable coordinate generation.
- Update `src/obikmer/src/cmd/phylo/args.rs` to add `fasta` and `vcf` flags
  and a new `alignment_output()` method that returns true if any output format
  is requested. Add `pseudo_alignment` field documentation.
- Modify `src/obikmer/src/cmd/phylo/mod.rs` to use `alignment_output()` for
  determining alignment needs, update error messages for `--n-sites`, and import
  `write_alignment_outputs` from the new module.
- Refactor `src/obikphylo/src/siblings/algorithms/alignment.rs` by adding
  `SiteLocus` struct (partition, layer, family_idx, family_id) and updating
  `SnpAlignment` and `AlignmentReducer` to track sites. Update `new`, `finish`,
  and `reduce` to populate site information from `SurvivingFamily`.
- Update `src/obikphylo/src/siblings/algorithms/iqtree.rs`: `alignment` now
  returns `SnpAlignment` with empty sites; update `drop_ascertainment_noninformative`
  and `recode_symbols_as_absent` to handle the new `sites` field.
- Update `src/obikphylo/src/siblings/algorithms/family_scan.rs`: `GeneratedBatch`
  now includes a `kmers` field; `scan_layer_families` accepts `CanonicalKmer`
  and passes it to closures instead of masks.
- Update `src/obikphylo/src/siblings/algorithms/subsample.rs`: `SurvivingFamily`
  gains a `kmer` field; `sample_layer` closure accepts and uses this k-mer.
- Update `src/obikphylo/src/siblings/algorithms/snp_distance.rs`: `tally_2genomes`
  initializes `SurvivingFamily` with explicit `CanonicalKmer` values.
- Update `src/obikphylo/src/siblings/algorithms/stats.rs`: `sibling_annex_stats`
  signature adds `_kmer` parameter.
- Update `src/obikphylo/src/siblings/algorithms/family_annexes.rs`: closures in
  `ensure_layer_family_annexes` and `ensure_layer_pair_weight_annexes` now accept
  `_kmer`.
- Update `src/obikphylo/src/siblings/algorithms/pairwise.rs`: `ambiguous_vs_single`
  initializes k-mer using `CanonicalKmer::from_raw_unchecked(0)`.
- Add `iupac_state` function to `src/obikphylo/src/siblings/algorithms/masking.rs`
  mapping DNA characters to binary codes.
- Update `src/obikphylo/src/siblings/algorithms/mod.rs` to export `SiteLocus`
  and include new `alignment_vcf` module.
- Add `family_id` and `write_family_id_ascii` methods to
  `src/obikseq/src/kmer.rs` for stable k-mer family identification.
- Add tests `family_id_partitions_kmers_into_families` and
  `family_id_ascii_masks_centre` to `src/obikseq/src/tests/kmer.rs`.
- Update `src/obikphylo/src/siblings/mod.rs` imports to include new algorithm
  exports.
2026-11-09 20:58:05 +11:00
coissac ee62b131f9 Merge pull request 'Push ltpzonnnkpqs' (#79) from push-ltpzonnnkpqs into main
Reviewed-on: #79
2026-10-16 14:43:50 +00:00
Eric Coissac 85296a2ef8 Implement directory locking with shared and exclusive modes
Release / create-release (push) Successful in 2m41s
Release / build-linux-x86_64 (push) Successful in 9m17s
Release / build-macos-arm64 (push) Successful in 2m2s
ci.yml / build (pull_request) Successful in 3m48s
Introduces a robust, OS-level directory locking mechanism using LockSlot for managing shared and exclusive access, supporting lock transitions and non-blocking probes. Also sets up the index structure with new builder and index modules.
v1.4.1
2026-10-16 16:02:00 +02:00
Eric Coissac 9ac3a37529 Implement explicit directory locking for index operations
Introduces explicit directory locking mechanisms (`DirLock`) to ensure exclusive access during index operations across command stages. This includes modifying index opening logic to acquire locks before proceeding, handling lock acquisition failures by exiting, and implementing safe conditional building of sibling annexes using locking to prevent concurrent corruption.
2026-10-16 16:00:49 +02:00
Eric Coissac 7c7d7615cb Implement sibling annex building and extension logic
This commit introduces the `SiblingBuilder` and `SiblingExt` traits to manage the creation, invalidation, and atomic building of sibling annexes. It also adds new functionality for sampling, statistics, and improved documentation for sibling-related operations.
2026-10-16 15:59:40 +02:00
coissac c6226dd40c Merge pull request 'Push wpvskkyozppn' (#78) from push-wpvskkyozppn into main
Reviewed-on: #78
2026-10-15 10:38:58 +00:00
Eric Coissac 8296e3544c Simplify Hybrid layer implementation and update dependencies
Release / create-release (push) Successful in 2m31s
Release / build-linux-x86_64 (push) Successful in 9m24s
Release / build-macos-arm64 (push) Successful in 2m5s
ci.yml / build (pull_request) Successful in 3m51s
Dropped fields related to Hybrid layers for simplification and removed the unused `ahash` dependency. Updated dependency versions for `obidebruinj`, `obikmer`, and related crates, and cleaned up temporary artifacts.
v1.4.0
2026-10-15 12:23:15 +02:00
Eric Coissac 12afbe819c Refactor MPHF layer logic and evidence handling
This commit implements core lookup functionality for MPHF layers, including `hash` and `find_strict` methods. It introduces support for different evidence types (Exact, Hybrid, Approximate) within layers, clarifying lookup behavior and improving partition calculation logic in the cache. Additionally, dead code is removed and dependencies are updated.
2026-10-15 12:22:45 +02:00
Eric Coissac f9edefd135 Refactor query system for performance and memory reduction
This change introduces dynamic chunk sizing based on input size estimates, ensuring memory constraints are respected while allowing for smaller chunks that yield significant speed improvements. Additionally, index structures were optimized, resulting in drastic memory footprint reduction and faster query execution times.
2026-10-15 12:21:38 +02:00
Eric Coissac 025c40c0db Refactor chunking logic to enforce 128 k-mer limit
This change reduces the maximum number of k-mers per chunk from 256 to 128. This modification requires updating the data model to handle sequences exceeding this limit by splitting them into overlapping chunks, ensuring no k-mers are lost. Extensive testing has been added to validate the new chunking logic and k-mer constraints.
2026-10-15 12:20:36 +02:00
Eric Coissac 8468319a63 Refactor unitig detection and graph traversal logic
Introduce new hashing mechanisms and support for directional unitig checks. Fix fragmentation issues and improve performance metrics through optimized graph construction and new benchmarking utilities.
2026-10-15 12:19:24 +02:00
Eric Coissac d32d86193f Refactor index metadata loading to use buffered I/O
This change replaces unbuffered file reads with buffered I/O operations to improve performance during index metadata loading. This includes implementing new methods for reading index metadata files and wrapping file handles in `BufReader` to optimize subsequent reads.
2026-10-15 12:17:51 +02:00
Eric Coissac 093d81c359 Introduce memory mapping for index files and sparse matrices
This change implements memory-mapped views for index readers and sparse matrices, enabling shared page-cache access. It also includes performance analysis confirming that the default chunk size cap negatively impacts parallelism for small inputs.
2026-10-15 12:17:09 +02:00
Eric Coissac 09c80468b2 Refactor k-mer storage to use flat arrays
This change replaces the HashMap-based k-mer storage mechanism with a compact, flat array structure for `PartitionKmers`. This significantly reduces memory usage per partition and improves query execution time.

It also updates the batch processing logic to handle the new data structure, introduces detailed memory footprint calculations, and updates relevant query layer APIs. Test coverage has been expanded to validate correct partitioning and k-mer ordering.
2026-10-15 12:16:21 +02:00
Eric Coissac fe38424682 Implement batch lookup and chunked data management
Introduces support for batched MPHF lookups, chunked data handling via `SlottedKmers`, and a comprehensive benchmarking suite to measure performance across the indexing and querying pipeline.

This includes new layer functionality for batch operations, chunking logic in the query layer, and scripts to measure wall time and resource usage for various commands. Documentation related to specific implementation details has also been cleaned up.
2026-10-15 12:14:51 +02:00
Eric Coissac 8b87078478 Implement MPHF migration tool and sparse matrix permutation logic
Refactors sparse matrix handling to implement row permutation logic and introduces a standalone tool for migrating indexes from MPHF 1.x to 2.x formats.
2026-10-15 12:12:51 +02:00
Eric Coissac d686d39506 Refactor indexing logic and update core dependencies
This commit introduces significant updates to the indexing system, including the introduction of zero-copy types, modifications to the `Mphf` type definition, and new hybrid search strategies. Core algorithms are updated to handle new hashing types and improved error handling during pointer hash construction. Dependency versions are also updated across the board, including new security features and system-level crates.
2026-10-15 12:07:08 +02:00
Eric Coissac 3bd7fd4f41 Implement memory mapping for file construction
Refactors the building process to utilize memory mapping (`memmap2`) for file creation and introduces the necessary infrastructure for publishing built files atomically. This includes expanding the public API surface and cleaning up deprecated architectural components.
2026-10-15 12:02:33 +02:00
Eric Coissac c226744405 Refactor sampling algorithms and coverage metrics
Optimize family scanning and sampling plan calculation logic. Introduce new per-genome richness metrics and redefine the sampling fraction $r$ for improved analysis.
2026-10-15 12:01:15 +02:00
Eric Coissac 0e40e83fd1 Refactor probability formula notation
Update the mathematical formula for $P(X \geq r)$ to explicitly use summation notation and clearly format the probability expression.
2026-10-15 12:00:51 +02:00
Eric Coissac cb5fc202b6 Refactor sampling logic and introduce diagnostic outputs
This commit introduces new features for sampling, including detailed `SamplePlan` generation, Shannon entropy summaries, and site statistics reporting. It formalizes conditional distance matrix computation and clarifies the logic for family weighting and site selection.
2026-10-15 12:00:44 +02:00
coissac 2232628e58 Merge pull request 'Push kkkmrtvotrsu' (#77) from push-kkkmrtvotrsu into main
Reviewed-on: #77
2026-10-13 07:54:48 +00:00
Eric Coissac f29dcbcfb8 Implement advanced sampling and family annex logic
Release / create-release (push) Successful in 2m35s
Release / build-linux-x86_64 (push) Successful in 8m19s
Release / build-macos-arm64 (push) Successful in 2m3s
ci.yml / build (pull_request) Successful in 4m8s
Introduces complex sampling mechanisms including quartet filtering, pair balance weighting, and entropy bias. Enforces the `--no-ambiguity` convention for family counting and updates annex storage for scalar data.

This includes new logic for calculating family weights based on overlap, quota-based layer determination, and filtering families based on quartet informativeness. Updates also include argument parsing for `--n-sites` and improved handling of family overlap files.
v1.3.6
2026-10-13 08:57:18 +02:00
Eric Coissac 0d12494f1a Refactor indexing, sampling, and entropy documentation
This commit introduces new theoretical frameworks for k-mer indexing, pair coverage sampling, and sibling partition unification. It also implements new algorithms for calculating family entropy and statistical tests for overlap analysis. Documentation is updated across the board to reflect these new features and theoretical concepts.
2026-10-13 08:54:21 +02:00
Eric Coissac 95bbc0b2ae Implement sparse matrix optimizations and update architecture documentation
This commit introduces significant functional improvements related to sparse matrix handling, including new storage layers, performance fixes for sparse operations, and bug corrections in matrix logic. Additionally, architectural documentation is cleaned up by removing obsolete entries and modifying structural text related to the Sibling Annex discussion.
2026-10-13 08:52:37 +02:00
Eric Coissac fdfc1eb3dd Refactor siblings module and enhance sampling logic
Introduces filtering for scan families, handles genome exclusion arguments, refactors the `siblings` module architecture, and clarifies invariants for statistical calculations.
2026-10-13 08:45:26 +02:00
Eric Coissac 996ebaf977 Refactor algorithms to use configuration flags
Introduce boolean flags for `AlignmentReducer` and `PairwiseReducer` to control behavior. Update masking logic and introduce the `SurvivingFamily` structure for explicit data handling.
2026-10-13 08:44:19 +02:00
Eric Coissac cd06fa2ce3 Implement masking logic for pairwise reduction
Introduces masking based on `free_loss` and `no_ambiguity` flags within pairwise algorithms. Updates reducer initialization and adds tests to verify cardinality calculation.
2026-10-13 08:43:29 +02:00
Eric Coissac 1702addbf9 Refactor algorithms for unified sampling and reduction
Introduces a unified sampling trunk, new reducer traits, and refined logic for SNP distance calculation and session handling. Enforces strict parameter sharing across all algorithms to ensure consistent sampling.
2026-10-13 08:43:06 +02:00
Eric Coissac 0d1ecac233 Refactor k-mer indexing, enhance session features, and update documentation
This commit introduces significant refinements across the codebase. Core indexing logic is updated with hard constraints on k-mer length, statistical modeling clarifications, and the implementation of the Karlin & Ost theory for chance-collision statistics. Session management features are enhanced with robust memoization capabilities, atomic file operations, per-key locking, and memory mapping support. Additionally, documentation is updated, including a new citation style, and new mathematical notebook support is added.
2026-10-13 08:41:42 +02:00
Eric Coissac e715528d34 Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
2026-10-13 08:38:49 +02:00
Eric Coissac 9da0876ee6 Refactor indexing logic using hash-based minimizer selection.
Updates indexing architecture to use hashing for minimizer distribution and introduces a new hash-based selection mechanism. Additionally, updates documentation references and implements a new script for converting MkDocs into a flat wiki structure.
2026-10-13 08:34:51 +02:00
coissac 755cf85334 Merge pull request 'Push twnyxtspltux' (#76) from push-twnyxtspltux into main
Reviewed-on: #76
2026-09-12 05:57:25 +00:00
Eric Coissac 2693a6af09 Implement session persistence and checkpointing
Release / create-release (push) Successful in 2m32s
Release / build-linux-x86_64 (push) Successful in 8m21s
Release / build-macos-arm64 (push) Successful in 2m1s
ci.yml / build (pull_request) Successful in 3m45s
Introduces session persistence via the `--session` option, allowing sampled sets to be restored. Implements chunked dumps and shared checkpointing for alignment and tally artifacts, ensuring state restoration upon interruption. Refines sampling logic and introduces serialization mechanisms for state management.
v1.3.5
2026-09-12 07:48:51 +02:00
Eric Coissac 7861e886e7 Introduce session persistence and artifact caching
This change implements a robust session management system allowing users to persist distance calculation samples and parameters across invocations. It includes concurrency-safe directory locking, deterministic parameter encoding, artifact integrity checks, and caching mechanisms to skip resampling when valid data is available.
2026-09-12 07:46:55 +02:00
Eric Coissac a8bcf5ffac Refactor distance matrix calculation logic for consistency
Ensures that combined distance matrix calculations use the exact same site selection by retaining sampled data within Sankoff bundles. This involves refactoring post-sampling logic into shared functions, implementing `SankoffBundle` to reuse internal tally data, and adding validation to guarantee consistent site selection across all distance calculation paths.
2026-09-12 07:45:32 +02:00