Push punmmmpkuymy #83
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1119af85ba |
feat: update indexing pipeline and bump obikmer version to 1.4.7
Refactor the indexing process to support parallel phases, resumable builds, and three verification modes (Exact, Approximate, Hybrid). Update the on-disk layout to manage evidence-specific files and state in `index.meta`, and refine layer merging logic. Bump the obikmer package version from 1.4.6 to 1.4.7. - Modify indexing routing to use low bits of hash and enable parallel, independent phases for resumable builds - Update counting phase to build provisional MPHF and temporary counts, deleted unless `--keep-intermediate` is set - Combine quorum filtering with local assembly by filtering kmers during de Bruijn graph construction before unitig assembly - Expand verification modes to Exact, Approximate, and Hybrid, allowing post-build conversion between them - Update on-disk layout to include specific files for each evidence type and manage build state in `index.meta` - Refine layer merging to create new layers only for genuinely new kmer content and determine destination evidence mode from sources - Add `--sparse` option to packing logic for sparse representation support - Bump obikmer package version from 1.4.6 to 1.4.7 in Cargo.toml |
||
|
|
36c7247acb |
feat: add k-mer sharing estimation and Poisson-binomial null model
Add a new `estimate --index` command to compare observed k-mer sharing against a random-sequence model using a Poisson-binomial distribution. Introduce the `row_presence_histogram` API to support this analysis across various matrix types. Update documentation and dependencies accordingly. - Add `src/obikstats/src/null_model.rs` implementing `SharingNullModel` and `SharingRow` for computing cumulative probabilities and false discovery rates based on a Poisson-binomial distribution. - Add `src/obikstats/src/sharing.rs` defining `GenomeSharing` and `SharingSpectrum` to compute k-mer sharing spectra using parallel map-reduce. - Add `src/obikmer/src/cmd/estimate/sharing.rs` with `run` and `write_csv` to execute sharing estimation and output CSV results containing observed, expected, and deviation metrics. - Modify `src/obikmer/src/cmd/estimate/mod.rs` to support the new `--index` subcommand, adding `EstimateArgs` fields for index path, presence threshold, and output file. - Add `src/obicompactvec/src/row_histogram.rs` implementing `row_presence_histogram` for `PersistentIntMatrix`, `PersistentBitMatrix`, and sparse matrices to count rows present in exactly $j$ columns. - Update `src/obikindex/src/layer/content_layer.rs` and `src/obikindex/src/layer/typed_layer.rs` to expose the new `row_presence_histogram` method, dispatching based on layer type. - Add tests in `src/obicompactvec/src/tests/bitmatrix.rs`, `intmatrix.rs`, and `sparse.rs` to verify `count_geq` and `row_presence_histogram` against naive implementations for various matrix structures. - Update `UserDocMD/theory/kmer_indexing/kmers.md` to document the Poisson-binomial approach, $D_r$ bounds, and FDR calculations, replacing previous binomial approximations. - Update `UserDocMD/usage/estimate.md` to describe the two uses of `estimate`: parameter estimation and index comparison via `--index`. - Add `wiki` target to `Makefile` for regenerating documentation from user docs. - Bump `obikmer` package version from 1.4.5 to 1.4.6 in `Cargo.toml` and `Cargo.lock`. |
||
|
|
6d6d5e5e53 |
feat: add count_geq method to calculate column counts above a threshold
Introduce `count_geq` across the codebase to return the number of k-mers present in columns with counts greater than or equal to a specified threshold. This unifies the interface for counting distinct k-mers and updates `partial_kmer_counts` to rely on `count_geq(1)`. - Add `count_geq` method to `PersistentBitMatrix` in `src/obicompactvec/src/bitmatrix/persistent.rs`, delegating to `count_ones`. - Implement `ColumnWeights` trait for `PersistentBitMatrix` and add `count_geq` to `PersistentSparseBitMatrix` in `src/obicompactvec/src/bitmatrix/sparse.rs`. - Add `count_geq` methods to `IntMatrix` variants (`Columnar`, `Packed`, `Sparse`) and `PersistentIntMatrix` in `src/obicompactvec/src/intmatrix.rs`, removing `partial_kmer_counts`. - Implement `count_geq` for `SparseCompactIntMatrix` in `src/obicompactvec/src/sparse_intmatrix.rs`, handling threshold 0 explicitly, and remove `partial_kmer_counts`. - Add `count_geq` method to `Reader` struct in `src/obicompactvec/src/reader.rs` with documentation. - Update `ColumnWeights` trait in `src/obicompactvec/src/traits.rs` to include `count_geq` and redefine `partial_kmer_counts` as `count_geq(1)`. - Add `count_geq` method to `Views` struct in `src/obicompactvec/src/views.rs` for distance calculations. - Add `count_geq` tests in `src/obicompactvec/src/tests/sparse_intmatrix.rs` verifying behavior against naive implementations. - Add `count_geq` to `ContentLayer` in `src/obikindex/src/layer/content_layer.rs`, delegating based on layer type. - Add `count_geq` to `TypedLayer` in `src/obikindex/src/layer/typed_layer.rs`, delegating to `obicompactvec::ColumnWeights`. - Add `count_geq` method to distance struct in `src/obikphylo/src/distance.rs` using `self.reduce`. |
||
|
|
33d7a59dd9 |
docs: restructure documentation for kmer indexing and phylogeny theory
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models. - Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic. - Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation. - Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers. - Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details. - Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics. - Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture. - Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model. - Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections. - Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation. - Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design. - Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure. - Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations. - Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`. - Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013). - Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation. - Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance. - Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990). - Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section. - Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`. - Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`. |