Push vpnykuuvnrnx #82

Merged
coissac merged 11 commits from push-vpnykuuvnrnx into main 2026-11-19 06:17:44 +00:00
Owner
No description provided.
coissac added 11 commits 2026-11-19 06:17:37 +00:00
Remove obsolete dependencies across multiple crates to clean up the build graph and update `obikdump` to use `try_update`. Add dev-dependencies for tracing in `obiread` and clarify memoization logic in `obiksession`.

- Remove `obifastwrite`, `tracing`, `obidebruinj`, `obikentropy`, `obipipeline`, `obidex`, and `obiskbuilder` from `Cargo.lock`.
- Remove `obifastwrite` and `tracing` dependencies from `src/obidebruinj/Cargo.toml`.
- Update `dump` function in `src/obikdump/src/dump.rs` to use `try_update` instead of `fetch_update` for updating remaining counts.
- Remove `obitaxonomy`, `obidebruinj`, `obipipeline`, and `obikentropy` dependencies from `src/obikindex/Cargo.toml`.
- Remove `obiksession` dependency from `src/obikmer/Cargo.toml`.
- Remove `obiskbuilder` dependency from `src/obikphylo/Cargo.toml` and update `kodama` version to "0.3.0".
- Remove `bytes` dependency from `src/obikrope/Cargo.toml` and set `criterion2` version to "3" with `cargo_bench_support`.
- Add `tracing` and `tracing-subscriber` as dev-dependencies in `src/obiread/Cargo.toml`.
- Remove `lazy_static` dependency from `src/obiskbuilder/Cargo.toml` and add `obikseq` as a test-utils dev-dependency.
- Clarify `memo_key_from_debug` in `src/obiksession/src/memoize.rs` with comments explaining Debug-derived field usage.
Add a prompt template for optimizing the `pack` step and a suite of Docker-based
debug tools including build, run, and monitoring scripts. Enhance `pack_matrices`
with LFD partition sizing and detailed timing reports, and update `stop` with
additional system statistics.

- Add `prompts/pack_optimization.md` to guide optimization of the `pack` step,
  specifying context, performance measurements, and deliverable structures.
- Add `scripts/docker_debug/` infrastructure: Dockerfile for musl targets,
  `build.sh` for image creation, `run.sh` for parallel indexing and merging,
  `monitor.sh` for PID tracking, `gen_fasta.py` for synthetic genomes, and
  `gen_fasta_real.py` for real genome generation with `read_fna`.
- Add `scripts/monitor_pack.sh` as a 1 Hz sampler for I/O and memory metrics,
  writing samples to CSV and summary reports.
- Add `scripts/watch_merge.sh` to monitor process resource usage, enforcing
  memory limits via SIGTERM/SIGKILL guards based on VmRSS and VmSize.
- Modify `src/obikindex/src/index/kmer_index.rs`: add `pending_matrix_bytes` and
  `report_pack_timings`; update `pack_matrices` to use LFD strategy for partition
  sizing and clarify documentation regarding `stat()` proxy bytes.
- Modify `src/obisys/src/stage.rs`: update `stop` definition to include new
  variables (`user_secs`, `sys_secs`, faults, context switches, I/O blocks) and
  logic checking for swap usage.
Introduce a new `views_scan` module containing `BitSliceView::for_each_set_in`, which iterates over set bits in a specified range using optimized word traversal. Update `build_from_dense` to use this method for batched, ordered column access, ensuring sorted row output. Add comprehensive tests for the new iteration logic and boundary handling.

- Add `src/obicompactvec/src/views_scan.rs` with `BitSliceView::for_each_set_in`, which iterates over a `Range<usize>` calling a closure for each set bit while optimizing sparse traversal via `word.trailing_zeros()`.
- Add `src/obicompactvec/src/tests/views_scan.rs` with tests `words`, `naive`, `scanned`, and helpers verifying that `for_each_set_in` matches naive iteration, handles empty/reversed ranges correctly, and rejects out-of-bounds ranges.
- Modify `build_from_dense` in `src/obicompactvec/src/bitmatrix/sparse.rs` to replace `nonzero_triples` with `dense.col_view(c).for_each_set_in`, ensuring column visits are batched and ordered for sorted row processing.
- Update module structure in `src/obicompactvec/src/lib.rs` and `src/obicompactvec/src/tests/mod.rs` to include the new `views_scan` module.
Introduce `sparse_columnar` and `sparse_intmatrix_columnar` modules to enable
building sparse matrices directly from columnar data without dense intermediates.
Add a `transpose` module with `RowBatch` and `transpose_windows` for efficient
windowed column-to-row transposition based on density checks.

- Add `src/obicompactvec/src/bitmatrix/sparse_columnar.rs` with `build_from_columnar`,
  `build_from_columnar_with`, and `read_bit_window` to build sparse bit matrices
  from columnar files using windowed reads.
- Add `src/obicompactvec/src/sparse_intmatrix_columnar.rs` with `build_from_columnar`,
  `build_from_columnar_with`, `invalid`, and `read_int_window` for integer matrix
  construction from PCIV columns, handling sentinel values and overflow.
- Add `src/obicompactvec/src/transpose.rs` containing `RowBatch<V>` struct and
  `transpose_windows` function for dynamic window sizing and memory-bounded
  column-to-row transposition.
- Update `src/obicompactvec/src/bitmatrix/sparse.rs` to use `packed_bit_matrix_n_cols`
  and `MatrixMeta` for validation, ensuring stale columnar data is transposed directly
  into a single `matrix.pbmx` file while removing old column files.
- Update `src/obicompactvec/src/sparse_intmatrix.rs` to use `packed_int_matrix_n_cols`
  for column count validation, making `pack_sparse_compact_int_matrix` idempotent and
  avoiding dense intermediates by transposing columnar data directly.
- Expose `packed_bit_matrix_n_cols` in `src/obicompactvec/src/bitmatrix/packed.rs` and
  make `packed_int_matrix_n_cols` public in `src/obicompactvec/src/intmatrix.rs`.
- Add `sparse_columnar` module to `src/obicompactvec/src/bitmatrix/mod.rs` and export
  `PersistentBitMatrixBuilder` and `batch_presence_counts`.
- Add `sparse_intmatrix_columnar` and `transpose` modules to `src/obicompactvec/src/lib.rs`.
- Add `transpose_columnar` test module in `src/obicompactvec/src/tests/mod.rs` with
  tests verifying columnar transpose matches dense route output.
Update Cargo.lock to remove unused dependencies and update tikv-jemallocator. Configure src/obikmer/Cargo.toml to use jemalloc for static musl builds, addressing fragmentation issues with the default multi-threaded allocator. Update main.rs to conditionally set Jemalloc as the global allocator when targeting musl.

- Remove unused dependencies from Cargo.lock: anes, block-pseudorand, chiapos-chacha8, criterion, criterion-plot, itertools (and its either dep), nanorand, hermit-abi, is-terminal, plotters ecosystem, tinytemplate deps, voracious_radix_sort rayon dep, and web-sys deps.
- Update tikv-jemallocator version from 0.5.4 to 0.7.0 in Cargo.lock.
- Add tikv-jemallocator = "0.7.0" to musl-specific dependencies in src/obikmer/Cargo.toml.
- Set Jemalloc as the global allocator in main.rs for static musl builds.
Add `compat_ptrhash.sh` and `compat_ptrhash2.sh` to facilitate determinism control and incremental merge testing between obikmer binaries. Update `tikv-jemallocator` dependency version in `Cargo.toml` and refresh `Cargo.lock` with new crates and checksums.

- Add scripts/docker_debug/compat_ptrhash.sh for compatibility tests (T0-T4) comparing ptr_hash versions using Docker and md5sum
- Add scripts/docker_debug/compat_ptrhash2.sh for determinism control and incremental merge operations between binaries A and B
- Update src/obikmer/Cargo.toml to change tikv-jemallocator version from 0.7.0 to 0.5.4 under musl target cfg
- Update src/Cargo.lock by adding new crates (anes, block-pseudorand, criterion, plotters, etc.) and updating existing package versions/checksums
Update documentation to explain that merged indices can serve as sources if at least one source is unmerged, and that the resulting column order follows the base index first. Clarify that explicitly named files are always read regardless of extension, with format detection based on content.

- refactor: prevent I/O errors when re-merging packed outputs by restricting `choose_base` to valid destinations via new `can_be_destination` check
- fix: handle zero-layer partitions in `build_index_layer` and `remove_build_artifacts` to skip unnecessary operations
- feat: allow `PathIter` to treat unrecognized extensions as explicit files, ensuring they are not silently dropped during directory expansion
- test: add comprehensive tests for `PathIter` including extension recognition, explicit file handling, and stdin sentinel behavior
- test: update smoke tests to verify empty partitions, `.fna.gz` indexing, and refusal of merging only packed indexes
- chore: remove obsolete cargo build environment variables from `run.sh` script
Refactor the `phylo` command's argument handling by splitting the monolithic
`args.rs` file into specialized submodules (`distance`, `outputs`, `sampling`,
`select`) and a central `mod.rs`. This improves modularity and separates
concerns for distance metrics, sampling options, and output configurations.

- Move `alignment_output`, `ambiguity`, and `annex_selection` from `args.rs` to
  `args/mod.rs` in the `PhyloArgs` struct.
- Relocate distance parsing logic (`parse_gamma_shape`, `parse_n_sites`, `target`)
  and helper functions (`distance_to_compute`, `as_classic`, `as_snp`, etc.) from
  `args.rs` to `values.rs`.
- Add new submodule files: `args/distance.rs` for `DistanceArgs`, `args/outputs.rs`
  for `AlignArgs` and `CostsArgs`, `args/sampling.rs` for `SamplingArgs`, and
  `args/select.rs` for `SelectArgs` and `AnnexArgs`.
- Update `main.rs` to register the new `Phylo` command using `cmd::phylo::PhyloArgs`.
- Refactor `mod.rs` (`run`) to use nested argument structures (e.g., `args.dist`,
  `args.sample`) instead of flat fields.
- Update `alignment_out.rs` to reference the new nested paths (`args.align.fasta`,
  `args.sample.no_ambiguity`).
Refactor the `phylo` command to support explicit distance metrics (Jaccard, Hamming) and SNP-based pseudo-alignments. Replace implicit metric selection with explicit requirements, such as requiring `--n-sites` for SNP analysis. Add a new `shared-kmers` output option and update documentation for performance characteristics of sparse vs dense formats.

- Update `phylo` CLI to support genome-vs-genome distances using k-mer content or SNPs, including options for NJ/UPGMA tree building and data export
- Replace `reindex` command with `convert` for in-place index evidence conversion (exact/approx/hybrid) and add `--upgrade-index` to `utils` for missing layer metadata
- Refactor `distance_to_compute` logic to remove implicit Jaccard implications for `--nj`/`--upgma` unless `--distance` is specified, and enforce `--n-sites` for `--pseudo-alignment`/`--sankoff`
- Add `shared_kmers` method to `Metrics` trait in `src/obikphylo/src/distance.rs` and new `write_shared_kmers` function in `src/obikmer/src/cmd/phylo/shared.rs` for CSV export
- Update `PackArgs::dense` documentation to clarify that dense format is faster for column-oriented distance matrices while sparse is better for single-row access
- Remove `requires_counts` definition and update assertions in `values.rs` to reflect new distance computation signatures and metric requirements
Add `threshold_hamming_dist_matrix` to calculate k-mers present in exactly one genome using union minus intersection. Update `DistanceArgs` and `distance.rs` to support this metric on count indexes via `--presence-threshold`, removing the previous restriction that hamming was only for presence/absence indexes.

- Add `threshold_hamming_dist_matrix` method to `CountPartials` in `src/obikmer/src/traits.rs`
- Implement new test `threshold_hamming_counts_kmers_present_in_one_genome_only` in `src/obicompactvec/src/tests/intmatrix.rs`
- Update `DistanceArgs` in `src/obikmer/src/cmd/phylo/args/distance.rs` to clarify threshold effects on Jaccard/Mash/Hamming
- Modify `distance` function in `src/obikphylo/src/distance.rs` to support Hamming on count indexes and use pre-computed bit structures
- Update `obicompactvec.md` documentation with new function details and partial result explanations
- Refine `index.md` descriptions for `phylo`, `unitig`, and `select` options
refactor: treat absence as state unless --free-loss and update quartet annexes
ci.yml / build (pull_request) Successful in 4m33s
Release / create-release (push) Successful in 1m56s
Release / build-linux-x86_64 (push) Successful in 11m18s
Release / build-macos-arm64 (push) Successful in 2m27s
5db7b440da
Refactor the handling of genome absence to be a distinct state unless `--free-loss` is specified. This change introduces `QuartetConvention` to pair ambiguity policies with the free-loss flag, updating file naming conventions and logic across sampling, masking, and stats modules.

- Update `--exclude-genome` description in `pair_coverage_sampling.md` to clarify absence as a state unless `--free-loss` is specified.
- Move multi-member genome handling convention from `pair_coverage_sampling.md` to file name conventions.
- Clarify persisted annex structure in `pair_coverage_sampling.md` as `PersistentBitVec` per non-monomorphic family with counting convention carried by file name.
- Fix initial version's absence handling in `has_bipartition`, the annex, and `--sample-stats` to correctly count index 0 when absence is a state.
- Replace entropy-only builder with `AnnexRequest` and `ensure_layer_family_annexes`.
- Consolidate sampling parameters into a single `SamplingSpec`.
- Update family constraint description in `sampling_design.md`.
- Modify uniform draws and `--quartet` removal text in `sampling_design.md`.
- Clarify state counting in tests for `--free-loss` option in `sampling_design.md`.
- State that absence forms a pair without `--free-loss` but never with it in `sampling_design.md`.
- Update `--quartet` option to require at least two distinct states, each carried by at least two genomes.
- Define absence as a state unless `--free-loss` is given for family carrier definition.
- Clarify `--pair-balance` favors families covering pairs with fewest shared families.
- Modify logic for families fitting every tree topology equally well in `phylo.md`.
- Change `--quartet` restriction logic to apply after pre-checks involving `--no-ambiguity` and `--exclude-genome`.
- Update `obikmer` dependency version from 1.4.4 to 1.4.5 in `Cargo.lock` and `Cargo.toml`.
- Modify `annex_selection` in `args/mod.rs` to derive `free_loss`, `entropy`, `quartet`, and `overlap` from `SamplingSpec`.
- Update `SamplingArgs::quartet` documentation in `sampling.rs` to describe state determination based on `--free-loss`.
- Add `free_loss` field to `AnnexSelection` struct in `family_annexes.rs`.
- Change `AnnexRequest` to use `Vec<QuartetConvention>` instead of `Vec<AmbiguityPolicy>`.
- Update `request` logic in `family_annexes.rs` to conditionally create `QuartetConvention` based on `self.quartet`.
- Modify `dedup` signature in `family_annexes.rs` to accept iterator of type `T: Ord`.
- Update `ensure_layer_family_annexes` in `family_annexes.rs` to use `missing_files` and `QuartetConvention`.
- Add `missing_files` definition to `family_annexes.rs`.
- Change `has_bipartition` in `masking.rs` to filter counts where count is `>= 2`.
- Update `is_quartet_informative` in `masking.rs` to accept `free_loss` boolean parameter.
- Modify `two_pairs_of_distinct_states_is_informative` and `singleton_variant_is_not_informative` in `masking.rs` to test both `free_loss` policies.
- Add `under_free_loss_absence_is_missing_and_forms_no_pair` and `without_free_loss_absence_is_a_state_that_can_form_a_pair` definitions to `masking.rs`.
- Update `no_ambiguity_drops_multi_member_genomes_from_the_count` in `masking.rs` to check `is_quartet_informative` under `free_loss=true`.
- Add `the_annex_rule_and_the_exact_check_agree_on_the_same_family` definition to `masking.rs`.
- Remove `absent_genomes_are_not_a_state` definition from `masking.rs`.
- Update `restrict_to_quartet` in `quartet.rs` to accept `QuartetConvention` instead of `AmbiquityPolicy`.
- Modify `add_site` in `sample_stats.rs` to handle state 0 as a state participating in bipartition.
- Add `absence_as_a_state_makes_a_site_informative` definition to `sample_stats.rs`.
- Add `quartet_convention` function to `SamplingSpec` in `subsample.rs`.
- Update `sample_index` in `subsample.rs` to use `spec.quartet_convention()` for `AnnexRequest`.
- Update `restrict_to_quartet` call in `subsample.rs` to use `spec.quartet_convention()`.
- Add comment regarding `--no-ambiguity` counting convention in `helpers.rs`.
- Import `QUARTET_ANNEX_FILE_NAMES` and `QuartetConvention` from `quartet_annex` in `mod.rs`.
- Add `QuartetConvention` struct with `policy` and `free_loss` fields to `quartet_annex.rs`.
- Modify `quartet_annex_file_name` in `quartet_annex.rs` to take `QuartetConvention`.
- Add `the_conventions_never_share_a_file` function to `quartet_annex.rs`.
- Remove `the_policies_never_share_a_file` definition from `quartet_annex.rs`.
coissac merged commit d78a6b4501 into main 2026-11-19 06:17:44 +00:00
coissac deleted branch push-vpnykuuvnrnx 2026-11-19 06:17:44 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: OBIKmers/obikmer#82