docs: restructure documentation for kmer indexing and phylogeny theory

Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
This commit is contained in:
Eric Coissac committed 2026-11-19 17:47:59 +01:00
1 parent d78a6b4501
commit 33d7a59dd9
43 files changed
+25375 -1117

No files matched your search

+471 -21
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis">
Kmers and super-kmers
Kmer indexing
@@ -272,11 +283,19 @@
</a>
</li>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link">
<a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link">
<a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link">
<a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link">
<a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
@@ -655,6 +1043,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#disk-usage" class="md-nav__link">
<span class="md-ellipsis">
Disk usage
</span>
</a>
</li>
</ul>
@@ -1028,6 +1427,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul>
</nav>
@@ -1119,6 +1546,17 @@
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#disk-usage" class="md-nav__link">
<span class="md-ellipsis">
Disk usage
</span>
</a>
</li>
</ul>
@@ -1139,8 +1577,8 @@
<h1 id="select">select</h1>
<p>Project and/or aggregate the genome columns of an index into a new (or in-place) index. Where <a href="../filter/"><code>filter</code></a> selects rows (kmers), <code>select</code> operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.</p>
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>SOURCE<span class="w"> </span><span class="o">(</span>--output<span class="w"> </span>OUTPUT<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="o">)</span><span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
<p>Project and/or aggregate the genome columns of an index into a new index. Where <a href="../filter/"><code>filter</code></a> selects rows (kmers), <code>select</code> operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.</p>
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>SOURCE<span class="w"> </span>--output<span class="w"> </span>OUTPUT<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
</code></pre></div>
<h2 id="arguments">Arguments</h2>
<table>
@@ -1168,14 +1606,9 @@
</thead>
<tbody>
<tr>
<td><code>--output</code></td>
<td><code>-o, --output</code></td>
<td>—</td>
<td>Output index directory (mutually exclusive with <code>--in-place</code>)</td>
</tr>
<tr>
<td><code>--in-place</code></td>
<td>off</td>
<td>Rewrite the source index in place (mutually exclusive with <code>--output</code>)</td>
<td>Output index directory (required)</td>
</tr>
<tr>
<td><code>-f, --force</code></td>
@@ -1212,12 +1645,29 @@
<td><code>0</code></td>
<td>Minimum count for a genome to be considered a carrier (logical operators only)</td>
</tr>
<tr>
<td><code>--dense</code></td>
<td>off</td>
<td>Pack the output's presence matrices in the dense format instead of the default sparse one</td>
</tr>
<tr>
<td><code>--force-copy</code></td>
<td>off</td>
<td>Copy each layer's unchanged kmer-identity files (mphf/unitigs/evidence/fingerprint) instead of hard-linking them</td>
</tr>
</tbody>
</table>
<h2 id="aggregation-operators">Aggregation operators</h2>
<p><code>any</code>, <code>all</code>, <code>none</code> (logical, evaluated against <code>--presence-threshold</code>), <code>sum</code>, <code>min</code>, <code>max</code> (numeric, count index only). If a group's operator is left unspecified, it defaults to <code>any</code> when the source is a presence/absence index and <code>sum</code> when it stores counts.</p>
<p>A <code>select</code> never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in <code>--select</code>) is a cheap copy.</p>
<p>At least one of <code>--output</code>/<code>--in-place</code> is required, and at least one output column must be defined; every name listed in <code>--select</code> must resolve to either a defined group or an existing genome label. See <a href="../predicates/">Genome predicates and taxonomy paths</a> for the predicate syntax used by <code>--group</code>.</p>
<p>At least one output column must be defined; every name listed in <code>--select</code> must resolve to either a defined group or an existing genome label. See <a href="../predicates/">Genome predicates and taxonomy paths</a> for the predicate syntax used by <code>--group</code>.</p>
<h2 id="disk-usage">Disk usage</h2>
<p><code>select</code> always writes to a new output directory — there is no in-place mode. Each layer's kmer-identity files (MPHF, unitigs, evidence, fingerprint) never change under a column projection/aggregation, so they are hard-linked into the output rather than copied: no extra disk is used for them, even on a very large index. Linking falls back to a real copy automatically if it fails (e.g. <code>SOURCE</code>/<code>OUTPUT</code> on different filesystems). Use <code>--force-copy</code> to always copy instead — needed when the output must be able to survive independently of the source on disk (a hard link shares the same underlying data, so overwriting one path outside <code>select</code> itself would affect the other).</p>
<p>To replace an index with a selected version of itself, select to a temporary directory and swap it in:</p>
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>INDEX<span class="w"> </span>--output<span class="w"> </span>INDEX.tmp<span class="w"> </span>--group<span class="w"> </span>...<span class="w"> </span>--group-op<span class="w"> </span>...<span class="w"> </span>--select<span class="w"> </span>...
rm<span class="w"> </span>-rf<span class="w"> </span>INDEX
mv<span class="w"> </span>INDEX.tmp<span class="w"> </span>INDEX
</code></pre></div>