Push vpnykuuvnrnx #82
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5db7b440da |
refactor: treat absence as state unless --free-loss and update quartet annexes
Refactor the handling of genome absence to be a distinct state unless `--free-loss` is specified. This change introduces `QuartetConvention` to pair ambiguity policies with the free-loss flag, updating file naming conventions and logic across sampling, masking, and stats modules. - Update `--exclude-genome` description in `pair_coverage_sampling.md` to clarify absence as a state unless `--free-loss` is specified. - Move multi-member genome handling convention from `pair_coverage_sampling.md` to file name conventions. - Clarify persisted annex structure in `pair_coverage_sampling.md` as `PersistentBitVec` per non-monomorphic family with counting convention carried by file name. - Fix initial version's absence handling in `has_bipartition`, the annex, and `--sample-stats` to correctly count index 0 when absence is a state. - Replace entropy-only builder with `AnnexRequest` and `ensure_layer_family_annexes`. - Consolidate sampling parameters into a single `SamplingSpec`. - Update family constraint description in `sampling_design.md`. - Modify uniform draws and `--quartet` removal text in `sampling_design.md`. - Clarify state counting in tests for `--free-loss` option in `sampling_design.md`. - State that absence forms a pair without `--free-loss` but never with it in `sampling_design.md`. - Update `--quartet` option to require at least two distinct states, each carried by at least two genomes. - Define absence as a state unless `--free-loss` is given for family carrier definition. - Clarify `--pair-balance` favors families covering pairs with fewest shared families. - Modify logic for families fitting every tree topology equally well in `phylo.md`. - Change `--quartet` restriction logic to apply after pre-checks involving `--no-ambiguity` and `--exclude-genome`. - Update `obikmer` dependency version from 1.4.4 to 1.4.5 in `Cargo.lock` and `Cargo.toml`. - Modify `annex_selection` in `args/mod.rs` to derive `free_loss`, `entropy`, `quartet`, and `overlap` from `SamplingSpec`. - Update `SamplingArgs::quartet` documentation in `sampling.rs` to describe state determination based on `--free-loss`. - Add `free_loss` field to `AnnexSelection` struct in `family_annexes.rs`. - Change `AnnexRequest` to use `Vec<QuartetConvention>` instead of `Vec<AmbiguityPolicy>`. - Update `request` logic in `family_annexes.rs` to conditionally create `QuartetConvention` based on `self.quartet`. - Modify `dedup` signature in `family_annexes.rs` to accept iterator of type `T: Ord`. - Update `ensure_layer_family_annexes` in `family_annexes.rs` to use `missing_files` and `QuartetConvention`. - Add `missing_files` definition to `family_annexes.rs`. - Change `has_bipartition` in `masking.rs` to filter counts where count is `>= 2`. - Update `is_quartet_informative` in `masking.rs` to accept `free_loss` boolean parameter. - Modify `two_pairs_of_distinct_states_is_informative` and `singleton_variant_is_not_informative` in `masking.rs` to test both `free_loss` policies. - Add `under_free_loss_absence_is_missing_and_forms_no_pair` and `without_free_loss_absence_is_a_state_that_can_form_a_pair` definitions to `masking.rs`. - Update `no_ambiguity_drops_multi_member_genomes_from_the_count` in `masking.rs` to check `is_quartet_informative` under `free_loss=true`. - Add `the_annex_rule_and_the_exact_check_agree_on_the_same_family` definition to `masking.rs`. - Remove `absent_genomes_are_not_a_state` definition from `masking.rs`. - Update `restrict_to_quartet` in `quartet.rs` to accept `QuartetConvention` instead of `AmbiquityPolicy`. - Modify `add_site` in `sample_stats.rs` to handle state 0 as a state participating in bipartition. - Add `absence_as_a_state_makes_a_site_informative` definition to `sample_stats.rs`. - Add `quartet_convention` function to `SamplingSpec` in `subsample.rs`. - Update `sample_index` in `subsample.rs` to use `spec.quartet_convention()` for `AnnexRequest`. - Update `restrict_to_quartet` call in `subsample.rs` to use `spec.quartet_convention()`. - Add comment regarding `--no-ambiguity` counting convention in `helpers.rs`. - Import `QUARTET_ANNEX_FILE_NAMES` and `QuartetConvention` from `quartet_annex` in `mod.rs`. - Add `QuartetConvention` struct with `policy` and `free_loss` fields to `quartet_annex.rs`. - Modify `quartet_annex_file_name` in `quartet_annex.rs` to take `QuartetConvention`. - Add `the_conventions_never_share_a_file` function to `quartet_annex.rs`. - Remove `the_policies_never_share_a_file` definition from `quartet_annex.rs`. |
||
|
|
c93616b432 |
feat: add threshold-based Hamming distance and refine phylo documentation
Add `threshold_hamming_dist_matrix` to calculate k-mers present in exactly one genome using union minus intersection. Update `DistanceArgs` and `distance.rs` to support this metric on count indexes via `--presence-threshold`, removing the previous restriction that hamming was only for presence/absence indexes. - Add `threshold_hamming_dist_matrix` method to `CountPartials` in `src/obikmer/src/traits.rs` - Implement new test `threshold_hamming_counts_kmers_present_in_one_genome_only` in `src/obicompactvec/src/tests/intmatrix.rs` - Update `DistanceArgs` in `src/obikmer/src/cmd/phylo/args/distance.rs` to clarify threshold effects on Jaccard/Mash/Hamming - Modify `distance` function in `src/obikphylo/src/distance.rs` to support Hamming on count indexes and use pre-computed bit structures - Update `obicompactvec.md` documentation with new function details and partial result explanations - Refine `index.md` descriptions for `phylo`, `unitig`, and `select` options |
||
|
|
c4eacb3f72 |
feat(phylo): expand genome distance metrics and refactor CLI arguments
Refactor the `phylo` command to support explicit distance metrics (Jaccard, Hamming) and SNP-based pseudo-alignments. Replace implicit metric selection with explicit requirements, such as requiring `--n-sites` for SNP analysis. Add a new `shared-kmers` output option and update documentation for performance characteristics of sparse vs dense formats. - Update `phylo` CLI to support genome-vs-genome distances using k-mer content or SNPs, including options for NJ/UPGMA tree building and data export - Replace `reindex` command with `convert` for in-place index evidence conversion (exact/approx/hybrid) and add `--upgrade-index` to `utils` for missing layer metadata - Refactor `distance_to_compute` logic to remove implicit Jaccard implications for `--nj`/`--upgma` unless `--distance` is specified, and enforce `--n-sites` for `--pseudo-alignment`/`--sankoff` - Add `shared_kmers` method to `Metrics` trait in `src/obikphylo/src/distance.rs` and new `write_shared_kmers` function in `src/obikmer/src/cmd/phylo/shared.rs` for CSV export - Update `PackArgs::dense` documentation to clarify that dense format is faster for column-oriented distance matrices while sparse is better for single-row access - Remove `requires_counts` definition and update assertions in `values.rs` to reflect new distance computation signatures and metric requirements |
||
|
|
14433e80a5 |
refactor: restructure phylo CLI arguments into submodules
Refactor the `phylo` command's argument handling by splitting the monolithic `args.rs` file into specialized submodules (`distance`, `outputs`, `sampling`, `select`) and a central `mod.rs`. This improves modularity and separates concerns for distance metrics, sampling options, and output configurations. - Move `alignment_output`, `ambiguity`, and `annex_selection` from `args.rs` to `args/mod.rs` in the `PhyloArgs` struct. - Relocate distance parsing logic (`parse_gamma_shape`, `parse_n_sites`, `target`) and helper functions (`distance_to_compute`, `as_classic`, `as_snp`, etc.) from `args.rs` to `values.rs`. - Add new submodule files: `args/distance.rs` for `DistanceArgs`, `args/outputs.rs` for `AlignArgs` and `CostsArgs`, `args/sampling.rs` for `SamplingArgs`, and `args/select.rs` for `SelectArgs` and `AnnexArgs`. - Update `main.rs` to register the new `Phylo` command using `cmd::phylo::PhyloArgs`. - Refactor `mod.rs` (`run`) to use nested argument structures (e.g., `args.dist`, `args.sample`) instead of flat fields. - Update `alignment_out.rs` to reference the new nested paths (`args.align.fasta`, `args.sample.no_ambiguity`). |
||
|
|
7f829ea557 |
docs: clarify merge re-merge behavior and input file handling
Update documentation to explain that merged indices can serve as sources if at least one source is unmerged, and that the resulting column order follows the base index first. Clarify that explicitly named files are always read regardless of extension, with format detection based on content. - refactor: prevent I/O errors when re-merging packed outputs by restricting `choose_base` to valid destinations via new `can_be_destination` check - fix: handle zero-layer partitions in `build_index_layer` and `remove_build_artifacts` to skip unnecessary operations - feat: allow `PathIter` to treat unrecognized extensions as explicit files, ensuring they are not silently dropped during directory expansion - test: add comprehensive tests for `PathIter` including extension recognition, explicit file handling, and stdin sentinel behavior - test: update smoke tests to verify empty partitions, `.fna.gz` indexing, and refusal of merging only packed indexes - chore: remove obsolete cargo build environment variables from `run.sh` script |
||
|
|
07c0a54e04 |
feat: add compatibility testing scripts and update dependencies
Add `compat_ptrhash.sh` and `compat_ptrhash2.sh` to facilitate determinism control and incremental merge testing between obikmer binaries. Update `tikv-jemallocator` dependency version in `Cargo.toml` and refresh `Cargo.lock` with new crates and checksums. - Add scripts/docker_debug/compat_ptrhash.sh for compatibility tests (T0-T4) comparing ptr_hash versions using Docker and md5sum - Add scripts/docker_debug/compat_ptrhash2.sh for determinism control and incremental merge operations between binaries A and B - Update src/obikmer/Cargo.toml to change tikv-jemallocator version from 0.7.0 to 0.5.4 under musl target cfg - Update src/Cargo.lock by adding new crates (anes, block-pseudorand, criterion, plotters, etc.) and updating existing package versions/checksums |
||
|
|
af525cd268 |
feat: switch musl builds to jemalloc allocator for stability
Update Cargo.lock to remove unused dependencies and update tikv-jemallocator. Configure src/obikmer/Cargo.toml to use jemalloc for static musl builds, addressing fragmentation issues with the default multi-threaded allocator. Update main.rs to conditionally set Jemalloc as the global allocator when targeting musl. - Remove unused dependencies from Cargo.lock: anes, block-pseudorand, chiapos-chacha8, criterion, criterion-plot, itertools (and its either dep), nanorand, hermit-abi, is-terminal, plotters ecosystem, tinytemplate deps, voracious_radix_sort rayon dep, and web-sys deps. - Update tikv-jemallocator version from 0.5.4 to 0.7.0 in Cargo.lock. - Add tikv-jemallocator = "0.7.0" to musl-specific dependencies in src/obikmer/Cargo.toml. - Set Jemalloc as the global allocator in main.rs for static musl builds. |
||
|
|
d0668eb167 |
feat: add columnar sparse matrix support and windowed transposition
Introduce `sparse_columnar` and `sparse_intmatrix_columnar` modules to enable building sparse matrices directly from columnar data without dense intermediates. Add a `transpose` module with `RowBatch` and `transpose_windows` for efficient windowed column-to-row transposition based on density checks. - Add `src/obicompactvec/src/bitmatrix/sparse_columnar.rs` with `build_from_columnar`, `build_from_columnar_with`, and `read_bit_window` to build sparse bit matrices from columnar files using windowed reads. - Add `src/obicompactvec/src/sparse_intmatrix_columnar.rs` with `build_from_columnar`, `build_from_columnar_with`, `invalid`, and `read_int_window` for integer matrix construction from PCIV columns, handling sentinel values and overflow. - Add `src/obicompactvec/src/transpose.rs` containing `RowBatch<V>` struct and `transpose_windows` function for dynamic window sizing and memory-bounded column-to-row transposition. - Update `src/obicompactvec/src/bitmatrix/sparse.rs` to use `packed_bit_matrix_n_cols` and `MatrixMeta` for validation, ensuring stale columnar data is transposed directly into a single `matrix.pbmx` file while removing old column files. - Update `src/obicompactvec/src/sparse_intmatrix.rs` to use `packed_int_matrix_n_cols` for column count validation, making `pack_sparse_compact_int_matrix` idempotent and avoiding dense intermediates by transposing columnar data directly. - Expose `packed_bit_matrix_n_cols` in `src/obicompactvec/src/bitmatrix/packed.rs` and make `packed_int_matrix_n_cols` public in `src/obicompactvec/src/intmatrix.rs`. - Add `sparse_columnar` module to `src/obicompactvec/src/bitmatrix/mod.rs` and export `PersistentBitMatrixBuilder` and `batch_presence_counts`. - Add `sparse_intmatrix_columnar` and `transpose` modules to `src/obicompactvec/src/lib.rs`. - Add `transpose_columnar` test module in `src/obicompactvec/src/tests/mod.rs` with tests verifying columnar transpose matches dense route output. |
||
|
|
4e98de4b77 |
feat: add BitSliceView::for_each_set_in and optimize build_from_dense
Introduce a new `views_scan` module containing `BitSliceView::for_each_set_in`, which iterates over set bits in a specified range using optimized word traversal. Update `build_from_dense` to use this method for batched, ordered column access, ensuring sorted row output. Add comprehensive tests for the new iteration logic and boundary handling. - Add `src/obicompactvec/src/views_scan.rs` with `BitSliceView::for_each_set_in`, which iterates over a `Range<usize>` calling a closure for each set bit while optimizing sparse traversal via `word.trailing_zeros()`. - Add `src/obicompactvec/src/tests/views_scan.rs` with tests `words`, `naive`, `scanned`, and helpers verifying that `for_each_set_in` matches naive iteration, handles empty/reversed ranges correctly, and rejects out-of-bounds ranges. - Modify `build_from_dense` in `src/obicompactvec/src/bitmatrix/sparse.rs` to replace `nonzero_triples` with `dense.col_view(c).for_each_set_in`, ensuring column visits are batched and ordered for sorted row processing. - Update module structure in `src/obicompactvec/src/lib.rs` and `src/obicompactvec/src/tests/mod.rs` to include the new `views_scan` module. |
||
|
|
03eaa4544a |
feat: add pack optimization prompt, debug infrastructure, and monitoring scripts
Add a prompt template for optimizing the `pack` step and a suite of Docker-based debug tools including build, run, and monitoring scripts. Enhance `pack_matrices` with LFD partition sizing and detailed timing reports, and update `stop` with additional system statistics. - Add `prompts/pack_optimization.md` to guide optimization of the `pack` step, specifying context, performance measurements, and deliverable structures. - Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets, `build.sh` for image creation, `run.sh` for parallel indexing and merging, `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and `gen_fasta_real.py` for real genome generation with `read_fna`. - Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics, writing samples to CSV and summary reports. - Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize. - Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition sizing and clarify documentation regarding `stat()` proxy bytes. - Modify `src/obisys/src/stage.rs`: update `stop` definition to include new variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and logic checking for swap usage. |
||
|
|
24291a4484 |
refactor: remove unused dependencies and simplify logic
Remove obsolete dependencies across multiple crates to clean up the build graph and update `obikdump` to use `try_update`. Add dev-dependencies for tracing in `obiread` and clarify memoization logic in `obiksession`. - Remove `obifastwrite`, `tracing`, `obidebruinj`, `obikentropy`, `obipipeline`, `obidex`, and `obiskbuilder` from `Cargo.lock`. - Remove `obifastwrite` and `tracing` dependencies from `src/obidebruinj/Cargo.toml`. - Update `dump` function in `src/obikdump/src/dump.rs` to use `try_update` instead of `fetch_update` for updating remaining counts. - Remove `obitaxonomy`, `obidebruinj`, `obipipeline`, and `obikentropy` dependencies from `src/obikindex/Cargo.toml`. - Remove `obiksession` dependency from `src/obikmer/Cargo.toml`. - Remove `obiskbuilder` dependency from `src/obikphylo/Cargo.toml` and update `kodama` version to "0.3.0". - Remove `bytes` dependency from `src/obikrope/Cargo.toml` and set `criterion2` version to "3" with `cargo_bench_support`. - Add `tracing` and `tracing-subscriber` as dev-dependencies in `src/obiread/Cargo.toml`. - Remove `lazy_static` dependency from `src/obiskbuilder/Cargo.toml` and add `obikseq` as a test-utils dev-dependency. - Clarify `memo_key_from_debug` in `src/obiksession/src/memoize.rs` with comments explaining Debug-derived field usage. |