Push punmmmpkuymy #83

Merged
coissac merged 4 commits from push-punmmmpkuymy into main 2026-11-20 13:24:07 +00:00
Owner
No description provided.
coissac added 4 commits 2026-11-20 13:23:49 +00:00
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
Introduce `count_geq` across the codebase to return the number of k-mers present in columns with counts greater than or equal to a specified threshold. This unifies the interface for counting distinct k-mers and updates `partial_kmer_counts` to rely on `count_geq(1)`.

- Add `count_geq` method to `PersistentBitMatrix` in `src/obicompactvec/src/bitmatrix/persistent.rs`, delegating to `count_ones`.
- Implement `ColumnWeights` trait for `PersistentBitMatrix` and add `count_geq` to `PersistentSparseBitMatrix` in `src/obicompactvec/src/bitmatrix/sparse.rs`.
- Add `count_geq` methods to `IntMatrix` variants (`Columnar`, `Packed`, `Sparse`) and `PersistentIntMatrix` in `src/obicompactvec/src/intmatrix.rs`, removing `partial_kmer_counts`.
- Implement `count_geq` for `SparseCompactIntMatrix` in `src/obicompactvec/src/sparse_intmatrix.rs`, handling threshold 0 explicitly, and remove `partial_kmer_counts`.
- Add `count_geq` method to `Reader` struct in `src/obicompactvec/src/reader.rs` with documentation.
- Update `ColumnWeights` trait in `src/obicompactvec/src/traits.rs` to include `count_geq` and redefine `partial_kmer_counts` as `count_geq(1)`.
- Add `count_geq` method to `Views` struct in `src/obicompactvec/src/views.rs` for distance calculations.
- Add `count_geq` tests in `src/obicompactvec/src/tests/sparse_intmatrix.rs` verifying behavior against naive implementations.
- Add `count_geq` to `ContentLayer` in `src/obikindex/src/layer/content_layer.rs`, delegating based on layer type.
- Add `count_geq` to `TypedLayer` in `src/obikindex/src/layer/typed_layer.rs`, delegating to `obicompactvec::ColumnWeights`.
- Add `count_geq` method to distance struct in `src/obikphylo/src/distance.rs` using `self.reduce`.
Add a new `estimate --index` command to compare observed k-mer sharing against
a random-sequence model using a Poisson-binomial distribution. Introduce the
`row_presence_histogram` API to support this analysis across various matrix
types. Update documentation and dependencies accordingly.

- Add `src/obikstats/src/null_model.rs` implementing `SharingNullModel` and
  `SharingRow` for computing cumulative probabilities and false discovery rates
  based on a Poisson-binomial distribution.
- Add `src/obikstats/src/sharing.rs` defining `GenomeSharing` and `SharingSpectrum`
  to compute k-mer sharing spectra using parallel map-reduce.
- Add `src/obikmer/src/cmd/estimate/sharing.rs` with `run` and `write_csv` to
  execute sharing estimation and output CSV results containing observed, expected,
  and deviation metrics.
- Modify `src/obikmer/src/cmd/estimate/mod.rs` to support the new `--index`
  subcommand, adding `EstimateArgs` fields for index path, presence threshold,
  and output file.
- Add `src/obicompactvec/src/row_histogram.rs` implementing `row_presence_histogram`
  for `PersistentIntMatrix`, `PersistentBitMatrix`, and sparse matrices to count
  rows present in exactly $j$ columns.
- Update `src/obikindex/src/layer/content_layer.rs` and `src/obikindex/src/layer/typed_layer.rs`
  to expose the new `row_presence_histogram` method, dispatching based on layer type.
- Add tests in `src/obicompactvec/src/tests/bitmatrix.rs`, `intmatrix.rs`, and
  `sparse.rs` to verify `count_geq` and `row_presence_histogram` against naive
  implementations for various matrix structures.
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to document the Poisson-binomial
  approach, $D_r$ bounds, and FDR calculations, replacing previous binomial approximations.
- Update `UserDocMD/usage/estimate.md` to describe the two uses of `estimate`:
  parameter estimation and index comparison via `--index`.
- Add `wiki` target to `Makefile` for regenerating documentation from user docs.
- Bump `obikmer` package version from 1.4.5 to 1.4.6 in `Cargo.toml` and `Cargo.lock`.
feat: update indexing pipeline and bump obikmer version to 1.4.7
Release / create-release (push) Successful in 1m15s
Release / build-linux-x86_64 (push) Successful in 6m43s
Release / build-macos-arm64 (push) Successful in 1m57s
ci.yml / build (pull_request) Successful in 3m42s
1119af85ba
Refactor the indexing process to support parallel phases, resumable builds, and three verification modes (Exact, Approximate, Hybrid). Update the on-disk layout to manage evidence-specific files and state in `index.meta`, and refine layer merging logic. Bump the obikmer package version from 1.4.6 to 1.4.7.

- Modify indexing routing to use low bits of hash and enable parallel, independent phases for resumable builds
- Update counting phase to build provisional MPHF and temporary counts, deleted unless `--keep-intermediate` is set
- Combine quorum filtering with local assembly by filtering kmers during de Bruijn graph construction before unitig assembly
- Expand verification modes to Exact, Approximate, and Hybrid, allowing post-build conversion between them
- Update on-disk layout to include specific files for each evidence type and manage build state in `index.meta`
- Refine layer merging to create new layers only for genuinely new kmer content and determine destination evidence mode from sources
- Add `--sparse` option to packing logic for sparse representation support
- Bump obikmer package version from 1.4.6 to 1.4.7 in Cargo.toml
coissac merged commit d7ea6973e3 into main 2026-11-20 13:24:07 +00:00
coissac deleted branch push-punmmmpkuymy 2026-11-20 13:24:08 +00:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: OBIKmers/obikmer#83