docs: restructure documentation for kmer indexing and phylogeny theory
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models. - Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic. - Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation. - Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers. - Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details. - Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics. - Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture. - Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model. - Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections. - Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation. - Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design. - Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure. - Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations. - Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`. - Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013). - Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation. - Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance. - Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990). - Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section. - Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`. - Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
This commit is contained in:
1 parent
d78a6b4501
commit
33d7a59dd9
43 files changed
+25375
-1117
No files matched your search
+482
-14
@@ -256,15 +256,26 @@
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Kmers and super-kmers
|
||||
Kmer indexing
|
||||
|
||||
|
||||
|
||||
@@ -272,11 +283,19 @@
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_1">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
Kmer indexing
|
||||
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
@@ -285,7 +304,7 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/encoding/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
@@ -313,14 +332,14 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/entropy_filter/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Low-complexity kmer filter
|
||||
Kmers
|
||||
|
||||
|
||||
|
||||
@@ -341,7 +360,7 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/minimizer_selection/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
@@ -369,7 +388,35 @@
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/indexing_architecture/" class="md-nav__link">
|
||||
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Super-kmers
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
@@ -390,6 +437,347 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Low-complexity kmer filter
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Phylogeny
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_2">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
Phylogeny
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Kmer-based
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_2_1">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
Kmer-based
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Whole k-mer set distances
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item md-nav__item--nested">
|
||||
|
||||
|
||||
|
||||
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
|
||||
|
||||
|
||||
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
SNP-based
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
</label>
|
||||
|
||||
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
|
||||
<label class="md-nav__title" for="__nav_3_2_2">
|
||||
<span class="md-nav__icon md-icon"></span>
|
||||
|
||||
|
||||
SNP-based
|
||||
|
||||
|
||||
</label>
|
||||
<ul class="md-nav__list" data-md-scrollfix>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Central-position SNP model
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Gamma rate heterogeneity
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sankoff model
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
Sampling design
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -599,6 +987,28 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#resource-usage" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Resource usage
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#merging-an-index-that-was-already-merged" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Merging an index that was already merged
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1028,6 +1438,34 @@
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="../../formats/vcf/" class="md-nav__link">
|
||||
|
||||
|
||||
|
||||
<span class="md-ellipsis">
|
||||
|
||||
|
||||
VCF pseudo-alignment
|
||||
|
||||
|
||||
|
||||
</span>
|
||||
|
||||
|
||||
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
</ul>
|
||||
</nav>
|
||||
|
||||
@@ -1119,6 +1557,28 @@
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#resource-usage" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Resource usage
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
<li class="md-nav__item">
|
||||
<a href="#merging-an-index-that-was-already-merged" class="md-nav__link">
|
||||
<span class="md-ellipsis">
|
||||
|
||||
Merging an index that was already merged
|
||||
|
||||
</span>
|
||||
</a>
|
||||
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
@@ -1188,15 +1648,23 @@
|
||||
<td>Disambiguate duplicate genome labels (<code>.1</code>, <code>.2</code>, …) instead of failing</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><code>--budget-fraction</code></td>
|
||||
<td><code>0.5</code></td>
|
||||
<td>Fraction of available RAM reserved as the memory budget for parallel partition merging</td>
|
||||
<td><code>--dense</code></td>
|
||||
<td>off</td>
|
||||
<td>Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see <a href="../pack/">pack</a>)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<h2 id="behaviour">Behaviour</h2>
|
||||
<p>The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. <code>--force-presence</code> forces presence/absence regardless of the sources.</p>
|
||||
<p>By default, merging two indexes that share a genome label fails with an error; <code>--rename-duplicates</code> instead appends a numeric suffix to keep both copies.</p>
|
||||
<p>The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; <code>--dense</code> keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The <a href="../pack/">pack</a> page explains the trade-off between the two formats.</p>
|
||||
<h2 id="resource-usage">Resource usage</h2>
|
||||
<p>The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.</p>
|
||||
<p>Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.</p>
|
||||
<h2 id="merging-an-index-that-was-already-merged">Merging an index that was already merged</h2>
|
||||
<p>An index produced by <code>merge</code> can be used as one of the sources of a later merge, which is how a collection is extended without starting again from the original genomes. There is one rule: among the sources, at least one must not have been through a merge. A merged index is stored in a compact packed format (see <a href="../pack/">pack</a>) that is read very efficiently but cannot be extended in place. The merge therefore takes a freshly built index as its starting point and brings the packed index in as a source. A typical case is a merged index of many genomes plus one or a few newly built single-genome indexes, and this works.</p>
|
||||
<p>If every source has already been merged or packed, the command stops before doing any work and says so. In that case, merge the original single-genome indexes together instead.</p>
|
||||
<p>The genome columns of the result follow the order of the starting index first, then the other sources, so they may not come out in the order the sources were listed. The genome labels always identify each column unambiguously.</p>
|
||||
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user