docs: restructure documentation for kmer indexing and phylogeny theory
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models. - Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic. - Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation. - Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers. - Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details. - Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics. - Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture. - Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model. - Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections. - Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation. - Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design. - Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure. - Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations. - Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`. - Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013). - Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation. - Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance. - Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990). - Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section. - Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`. - Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
This commit is contained in:
1 parent
d78a6b4501
commit
33d7a59dd9
43 files changed
+25375
-1117
No files matched your search
@@ -12,7 +12,7 @@
|
||||
<link rel="prev" href="../../usage/predicates/">
|
||||
|
||||
|
||||
<link rel="next" href="../../architecture/">
|
||||
<link rel="next" href="../vcf/">
|
||||
|
||||
|
||||
|
||||
@@ -256,15 +256,26 @@
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Kmers and super-kmers
|
||||
Kmer indexing
|
||||
|
||||
|
||||
|
||||
@@ -272,11 +283,19 @@
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_1">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
Kmer indexing
|
||||
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
@@ -285,7 +304,7 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/encoding/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
@@ -313,14 +332,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Low-complexity kmer filter
|
||||
Kmers
|
||||
|
||||
|
||||
|
||||
@@ -341,7 +360,7 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
@@ -369,7 +388,35 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Super-kmers
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
@@ -390,6 +437,347 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Low-complexity kmer filter
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Phylogeny
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_2">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
Phylogeny
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Kmer-based
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_2_1">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
Kmer-based
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Whole k-mer set distances
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
SNP-based
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_2_2">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
SNP-based
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Central-position SNP model
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Gamma rate heterogeneity
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sankoff model
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sampling design
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1039,6 +1427,34 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../vcf/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
VCF pseudo-alignment
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1162,9 +1578,9 @@
|
||||
|
||||
<h1 id="index-construction-and-on-disk-layout">Index construction and on-disk layout</h1>
|
||||
<h2 id="construction-pipeline">Construction pipeline</h2>
|
||||
<p>Building an index (<a href="../../usage/index_command/"><code>index</code></a>) proceeds through a fixed sequence of phases, each operating independently per partition (see <a href="../../theory/indexing_architecture/">Partitioning and indexing architecture</a>):</p>
|
||||
<p>Building an index (<a href="../../usage/index_command/"><code>index</code></a>) proceeds through a fixed sequence of phases, each operating independently per partition (see <a href="../../theory/kmer_indexing/indexing_architecture/">Partitioning and indexing architecture</a>):</p>
|
||||
<ol>
|
||||
<li><strong>Scatter.</strong> A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see <a href="../../theory/entropy_filter/">Low-complexity kmer filter</a>), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by <code>hash(minimizer) mod n_partitions</code> into one file per partition.</li>
|
||||
<li><strong>Scatter.</strong> A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see <a href="../../theory/kmer_indexing/entropy_filter/">Low-complexity kmer filter</a>), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by <code>hash(minimizer) mod n_partitions</code> into one file per partition.</li>
|
||||
<li><strong>Dereplication.</strong> Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.</li>
|
||||
<li><strong>Exact counting.</strong> Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.</li>
|
||||
<li><strong>Quorum filtering.</strong> Kmers outside the <code>--min-abundance</code>/<code>--max-abundance</code> range are dropped, and super-kmers are recompacted around the surviving kmer set.</li>
|
||||
|
||||
Reference in new issue
Block a user