Refactor documentation for kmer indexing and phylogenetic models
This commit updates documentation across the board, introducing definitions for kmer constraints, super-kmers, k-mer set distances, SNP models, and sampling design principles. It also clarifies the index construction pipeline and the underlying mathematical models used for genome pair calibration.
This commit is contained in:
1 parent
9da0876ee6
commit
e715528d34
19 files changed
+582
-712
No files matched your search
@@ -1,6 +1,6 @@
|
|||||||
# Architecture notes for advanced use
|
# Architecture notes for advanced use
|
||||||
|
|
||||||
This page describes execution-level behavior relevant to sizing and running `obikmer` on large datasets or multi-socket machines. It complements the [index format](formats/index_layout.md) and [theory](theory/indexing_architecture.md) pages.
|
This page describes execution-level behavior relevant to sizing and running `obikmer` on large datasets or multi-socket machines. It complements the [index format](formats/index_layout.md) and [theory](theory/kmer_indexing/indexing_architecture.md) pages.
|
||||||
|
|
||||||
## Sequence invariant
|
## Sequence invariant
|
||||||
|
|
||||||
@@ -8,7 +8,7 @@ Every input sequence is treated purely as a compact representation of a set of o
|
|||||||
|
|
||||||
- Only the `A`/`C`/`G`/`T` alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.
|
- Only the `A`/`C`/`G`/`T` alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.
|
||||||
- Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.
|
- Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.
|
||||||
- Kmers are always handled in canonical form (see [DNA encoding](theory/encoding.md)), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.
|
- Kmers are always handled in canonical form (see [DNA encoding](theory/kmer_indexing/encoding.md)), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.
|
||||||
|
|
||||||
## Index dimensioning
|
## Index dimensioning
|
||||||
|
|
||||||
|
|||||||
@@ -2,9 +2,9 @@
|
|||||||
|
|
||||||
## Construction pipeline
|
## Construction pipeline
|
||||||
|
|
||||||
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/indexing_architecture.md)):
|
Building an index ([`index`](../usage/index_command.md)) proceeds through a fixed sequence of phases, each operating independently per partition (see [Partitioning and indexing architecture](../theory/kmer_indexing/indexing_architecture.md)):
|
||||||
|
|
||||||
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition.
|
1. **Scatter.** A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see [Low-complexity kmer filter](../theory/kmer_indexing/entropy_filter.md)), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by `hash(minimizer) mod n_partitions` into one file per partition.
|
||||||
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
|
2. **Dereplication.** Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.
|
||||||
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.
|
3. **Exact counting.** Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.
|
||||||
4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set.
|
4. **Quorum filtering.** Kmers outside the `--min-abundance`/`--max-abundance` range are dropped, and super-kmers are recompacted around the surviving kmer set.
|
||||||
|
|||||||
+4
-4
@@ -6,11 +6,11 @@ All functionality is exposed through a single binary, `obikmer`, organized as su
|
|||||||
|
|
||||||
## Core principles
|
## Core principles
|
||||||
|
|
||||||
- Kmers are of fixed, odd length $k$, chosen at index-construction time in the range $[11, 31]$ (see [Kmers and super-kmers](theory/kmers_and_superkmers.md)).
|
- Kmers are of fixed, odd length $k$, chosen at index-construction time in the range $[11, 31]$ (see [Kmers](theory/kmer_indexing/kmers.md)).
|
||||||
- Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see [DNA encoding](theory/encoding.md)).
|
- Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see [DNA encoding](theory/kmer_indexing/encoding.md)).
|
||||||
- Kmers are handled in **canonical form** ($\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))$), making counting strand-independent.
|
- Kmers are handled in **canonical form** ($\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))$), making counting strand-independent.
|
||||||
- Sequences are decomposed into **super-kmers** before storage, anchored on a hash-selected **minimizer** (see [Minimizer selection](theory/minimizer_selection.md)), then routed to one of several **partitions** for parallel, memory-bounded processing (see [Partitioning and indexing architecture](theory/indexing_architecture.md)).
|
- Sequences are decomposed into **super-kmers** before storage, anchored on a hash-selected **minimizer** (see [Minimizer selection](theory/kmer_indexing/minimizer_selection.md)), then routed to one of several **partitions** for parallel, memory-bounded processing (see [Partitioning and indexing architecture](theory/kmer_indexing/indexing_architecture.md)).
|
||||||
- Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see [Low-complexity kmer filter](theory/entropy_filter.md)).
|
- Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see [Low-complexity kmer filter](theory/kmer_indexing/entropy_filter.md)).
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
|
|||||||
@@ -345,3 +345,224 @@
|
|||||||
url = {https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html},
|
url = {https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html},
|
||||||
urldate = {2026-09-12}
|
urldate = {2026-09-12}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Hamming1950-cw,
|
||||||
|
title = "{Error detecting and error correcting codes}",
|
||||||
|
author = "Hamming, R W",
|
||||||
|
journal = "The Bell System Technical Journal",
|
||||||
|
publisher = "Institute of Electrical and Electronics Engineers (IEEE)",
|
||||||
|
volume = 29,
|
||||||
|
number = 2,
|
||||||
|
pages = "147--160",
|
||||||
|
abstract = "The author was led to the study given in this paper from a
|
||||||
|
consideration of large scale computing machines in which a large
|
||||||
|
number of operations must be performed without a single error in
|
||||||
|
the end result. This problem of “doing things right” on a large
|
||||||
|
scale is not essentially new; in a telephone central office, for
|
||||||
|
example, a very large number of operations are performed while
|
||||||
|
the errors leading to wrong numbers are kept well under control,
|
||||||
|
though they have not been completely eliminated. This has been
|
||||||
|
achieved, in part, through the use of self-checking circuits. The
|
||||||
|
occasional failure that escapes routine checking is still
|
||||||
|
detected by the customer and will, if it persists, result in
|
||||||
|
customer complaint, while if it is transient it will produce only
|
||||||
|
occasional wrong numbers. At the same time the rest of the
|
||||||
|
central office functions satisfactorily. In a digital computer,
|
||||||
|
on the other hand, a single failure usually means the complete
|
||||||
|
failure, in the sense that if it is detected no more computing
|
||||||
|
can be done until the failure is located and corrected, while if
|
||||||
|
it escapes detection then it invalidates all subsequent
|
||||||
|
operations of the machine. Put in other words, in a telephone
|
||||||
|
central office there are a number of parallel paths which are
|
||||||
|
more or less independent of each other; in a digital machine
|
||||||
|
there is usually a single long path which passes through the same
|
||||||
|
piece of equipment many, many times before the answer is
|
||||||
|
obtained.",
|
||||||
|
month = apr,
|
||||||
|
year = 1950,
|
||||||
|
url = "http://dx.doi.org/10.1002/j.1538-7305.1950.tb00463.x",
|
||||||
|
doi = "10.1002/j.1538-7305.1950.tb00463.x",
|
||||||
|
issn = "0005-8580"
|
||||||
|
}
|
||||||
|
|
||||||
|
@ARTICLE{Ondov2016-dq,
|
||||||
|
title = "{Mash: fast genome and metagenome distance estimation using
|
||||||
|
{MinHash}}",
|
||||||
|
author = "Ondov, Brian D and Treangen, Todd J and Melsted, Páll and
|
||||||
|
Mallonee, Adam B and Bergman, Nicholas H and Koren, Sergey and
|
||||||
|
Phillippy, Adam M",
|
||||||
|
journal = "Genome biology",
|
||||||
|
volume = 17,
|
||||||
|
number = 1,
|
||||||
|
pages = 132,
|
||||||
|
abstract = "Mash extends the MinHash dimensionality-reduction technique to
|
||||||
|
include a pairwise mutation distance and P value significance
|
||||||
|
test, enabling the efficient clustering and search of massive
|
||||||
|
sequence collections. Mash reduces large sequences and sequence
|
||||||
|
sets to small, representative sketches, from which global mutation
|
||||||
|
distances can be rapidly estimated. We demonstrate several use
|
||||||
|
cases, including the clustering of all 54,118 NCBI RefSeq genomes
|
||||||
|
in 33 CPU h; real-time database search using assembled or
|
||||||
|
unassembled Illumina, Pacific Biosciences, and Oxford Nanopore
|
||||||
|
data; and the scalable clustering of hundreds of metagenomic
|
||||||
|
samples by composition. Mash is freely released under a BSD
|
||||||
|
license ( https://github.com/marbl/mash ).",
|
||||||
|
month = jun,
|
||||||
|
year = 2016,
|
||||||
|
url = "http://dx.doi.org/10.1186/s13059-016-0997-x",
|
||||||
|
keywords = "Alignment; Comparative genomics; Genomic distance; Metagenomics;
|
||||||
|
Nanopore; Sequencing",
|
||||||
|
doi = "10.1186/s13059-016-0997-x",
|
||||||
|
pmc = "PMC4915045",
|
||||||
|
pmid = 27323842,
|
||||||
|
issn = "1465-6906",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Bray1957-vv,
|
||||||
|
title = "{An ordination of the upland forest communities of southern
|
||||||
|
Wisconsin}",
|
||||||
|
author = "Bray, J Roger and Curtis, J T",
|
||||||
|
journal = "Ecological Monographs",
|
||||||
|
publisher = "Wiley",
|
||||||
|
volume = 27,
|
||||||
|
number = 4,
|
||||||
|
pages = "325--349",
|
||||||
|
month = oct,
|
||||||
|
year = 1957,
|
||||||
|
url = "http://dx.doi.org/10.2307/1942268",
|
||||||
|
doi = "10.2307/1942268",
|
||||||
|
issn = "0012-9615,1557-7015",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Gower1986-ri,
|
||||||
|
title = "{Metric and Euclidean properties of dissimilarity coefficients}",
|
||||||
|
author = "Gower, J C and Legendre, P",
|
||||||
|
journal = "Journal of Classification",
|
||||||
|
publisher = "Springer Science and Business Media LLC",
|
||||||
|
volume = 3,
|
||||||
|
number = 1,
|
||||||
|
pages = "5--48",
|
||||||
|
abstract = "We assemble here properties of certain dissimilarity coefficients
|
||||||
|
and are specially concerned with their metric and Euclidean
|
||||||
|
status. No attempt is made to be exhaustive as far as
|
||||||
|
coefficients are concerned, but certain mathematical results that
|
||||||
|
we have found useful are presented and should help establish
|
||||||
|
similar properties for other coefficients. The response to
|
||||||
|
different types of data is investigated, leading to guidance on
|
||||||
|
the choice of an appropriate coefficient.",
|
||||||
|
month = mar,
|
||||||
|
year = 1986,
|
||||||
|
url = "http://dx.doi.org/10.1007/bf01896809",
|
||||||
|
doi = "10.1007/bf01896809",
|
||||||
|
issn = "0176-4268,1432-1343",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Legendre2001-cq,
|
||||||
|
title = "{Ecologically meaningful transformations for ordination of species
|
||||||
|
data}",
|
||||||
|
author = "Legendre, Pierre and Gallagher, Eugene D",
|
||||||
|
journal = "Oecologia",
|
||||||
|
volume = 129,
|
||||||
|
number = 2,
|
||||||
|
pages = "271--280",
|
||||||
|
abstract = "This paper examines how to obtain species biplots in unconstrained
|
||||||
|
or constrained ordination without resorting to the Euclidean
|
||||||
|
distance [used in principal-component analysis (PCA) and
|
||||||
|
redundancy analysis (RDA)] or the chi-square distance [preserved
|
||||||
|
in correspondence analysis (CA) and canonical correspondence
|
||||||
|
analysis (CCA)] which are not always appropriate for the analysis
|
||||||
|
of community composition data. To achieve this goal,
|
||||||
|
transformations are proposed for species data tables. They allow
|
||||||
|
ecologists to use ordination methods such as PCA and RDA, which
|
||||||
|
are Euclidean-based, for the analysis of community data, while
|
||||||
|
circumventing the problems associated with the Euclidean distance,
|
||||||
|
and avoiding CA and CCA which present problems of their own in
|
||||||
|
some cases. This allows the use of the original (transformed)
|
||||||
|
species data in RDA carried out to test for relationships with
|
||||||
|
explanatory variables (i.e. environmental variables, or factors of
|
||||||
|
a multifactorial analysis-of-variance model); ecologists can then
|
||||||
|
draw biplots displaying the relationships of the species to the
|
||||||
|
explanatory variables. Another application allows the use of
|
||||||
|
species data in other methods of multivariate data analysis which
|
||||||
|
optimize a least-squares loss function; an example is K-means
|
||||||
|
partitioning.",
|
||||||
|
month = oct,
|
||||||
|
year = 2001,
|
||||||
|
url = "http://dx.doi.org/10.1007/s004420100716",
|
||||||
|
keywords = "Biplot diagram; Canonical correspondence analysis; Correspondence
|
||||||
|
analysis; Principal-component analysis; Redundancy analysis",
|
||||||
|
doi = "10.1007/s004420100716",
|
||||||
|
pmid = 28547606,
|
||||||
|
issn = "0029-8549,1432-1939",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Anderson2011-rq,
|
||||||
|
title = "{Navigating the multiple meanings of β diversity: a roadmap for
|
||||||
|
the practicing ecologist}",
|
||||||
|
author = "Anderson, Marti J and Crist, Thomas O and Chase, Jonathan M and
|
||||||
|
Vellend, Mark and Inouye, Brian D and Freestone, Amy L and
|
||||||
|
Sanders, Nathan J and Cornell, Howard V and Comita, Liza S and
|
||||||
|
Davies, Kendi F and Harrison, Susan P and Kraft, Nathan J B and
|
||||||
|
Stegen, James C and Swenson, Nathan G",
|
||||||
|
journal = "Ecology Letters",
|
||||||
|
publisher = "Wiley",
|
||||||
|
volume = 14,
|
||||||
|
number = 1,
|
||||||
|
pages = "19--28",
|
||||||
|
abstract = "A recent increase in studies of β diversity has yielded a
|
||||||
|
confusing array of concepts, measures and methods. Here, we
|
||||||
|
provide a roadmap of the most widely used and ecologically
|
||||||
|
relevant approaches for analysis through a series of mission
|
||||||
|
statements. We distinguish two types of β diversity: directional
|
||||||
|
turnover along a gradient vs. non-directional variation.
|
||||||
|
Different measures emphasize different properties of ecological
|
||||||
|
data. Such properties include the degree of emphasis on
|
||||||
|
presence/absence vs. relative abundance information and the
|
||||||
|
inclusion vs. exclusion of joint absences. Judicious use of
|
||||||
|
multiple measures in concert can uncover the underlying nature of
|
||||||
|
patterns in β diversity for a given dataset. A case study of
|
||||||
|
Indonesian coral assemblages shows the utility of a multi-faceted
|
||||||
|
approach. We advocate careful consideration of relevant
|
||||||
|
questions, matched by appropriate analyses. The rigorous
|
||||||
|
application of null models will also help to reveal potential
|
||||||
|
processes driving observed patterns in β diversity.",
|
||||||
|
month = jan,
|
||||||
|
year = 2011,
|
||||||
|
url = "http://dx.doi.org/10.1111/j.1461-0248.2010.01552.x",
|
||||||
|
doi = "10.1111/j.1461-0248.2010.01552.x",
|
||||||
|
pmid = 21070562,
|
||||||
|
issn = "1461-023X,1461-0248",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
|
@ARTICLE{Jaccard1901-mx,
|
||||||
|
title = "{Étude comparative de la distribution florale dans une portion
|
||||||
|
des Alpes et du Jura}",
|
||||||
|
author = "Jaccard, Paul",
|
||||||
|
journal = "Bulletin de la Société vaudoise des sciences naturelles",
|
||||||
|
publisher = "Imprimerie Corbaz \& Comp.",
|
||||||
|
volume = 37,
|
||||||
|
number = 142,
|
||||||
|
pages = "547--579",
|
||||||
|
abstract = "Schweizer Zeitschriften online - Ein Service der ETH-Bibliothek.
|
||||||
|
Revues suisses en ligne - Un service de ETH Library. Swiss
|
||||||
|
journals online - A service by ETH Library.",
|
||||||
|
year = 1901,
|
||||||
|
url = "http://dx.doi.org/10.5169/seals-266450",
|
||||||
|
doi = "10.5169/seals-266450",
|
||||||
|
issn = "0037-9603",
|
||||||
|
language = "de"
|
||||||
|
}
|
||||||
|
@ARTICLE{Kosub2019-xt,
|
||||||
|
title = "{A note on the triangle inequality for the Jaccard distance}",
|
||||||
|
author = "Kosub, Sven",
|
||||||
|
journal = "Pattern Recognition Letters",
|
||||||
|
publisher = "Elsevier BV",
|
||||||
|
volume = 120,
|
||||||
|
pages = "36--38",
|
||||||
|
month = apr,
|
||||||
|
year = 2019,
|
||||||
|
url = "http://dx.doi.org/10.1016/j.patrec.2018.12.007",
|
||||||
|
doi = "10.1016/j.patrec.2018.12.007",
|
||||||
|
issn = "0167-8655,1872-7344",
|
||||||
|
language = "en"
|
||||||
|
}
|
||||||
@@ -21,8 +21,8 @@ Reverse complement is computed by bit manipulation directly on the packed word,
|
|||||||
|
|
||||||
## Canonical form
|
## Canonical form
|
||||||
|
|
||||||
The canonical form of a kmer is the lexicographic minimum of the kmer and its reverse complement:
|
The canonical form of a packed 2-bit sequence is the lexicographic minimum of the sequence and its reverse complement:
|
||||||
|
|
||||||
$$\text{canonical}(kmer) = \min\big(kmer,\ \text{revcomp}(kmer)\big)$$
|
$$\text{canonical}(x) = \min\big(x,\ \text{revcomp}(x)\big)$$
|
||||||
|
|
||||||
Using the canonical form halves the kmer space and makes counting strand-independent: a kmer and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.
|
This is a single operation on the packed representation, so it applies identically regardless of what `x` represents — a kmer (see [Kmers](kmers.md)), an m-mer (see [Minimizer selection](minimizer_selection.md)), or a super-kmer (see [Super-kmers](superkmers.md)). Using the canonical form halves the relevant space and makes counting/selection strand-independent: a sequence and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.
|
||||||
File renamed without changes.
+4
-2
@@ -4,13 +4,15 @@ An index is split into a fixed number of **partitions**, each handling an indepe
|
|||||||
|
|
||||||
## Routing
|
## Routing
|
||||||
|
|
||||||
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition:
|
The canonical minimizer of a super-kmer (see [Minimizer selection](minimizer_selection.md)) is hashed to produce a $p$-bit routing value that selects the destination partition — the low $p$ bits of the minimizer's hash:
|
||||||
|
|
||||||
```
|
```
|
||||||
canonical minimizer → hash(minimizer) → p-bit value → partition index
|
canonical minimizer → hash(minimizer) → p-bit value → partition index
|
||||||
```
|
```
|
||||||
|
|
||||||
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
|
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
|
||||||
|
|
||||||
|
Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see [On-disk storage](../../formats/index_layout.md)); the minimizer plays no further role once a super-kmer has reached its partition.
|
||||||
|
|
||||||
## Parameter guidance
|
## Parameter guidance
|
||||||
|
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
# Kmers
|
||||||
|
|
||||||
|
A **kmer** is a DNA subsequence of fixed length $k$. Two constraints apply to $k$, both enforced when a command starts (an invalid value exits immediately with an error):
|
||||||
|
|
||||||
|
- $k \in [11, 31]$: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base ($k \le 32$ is the hard limit; $k < 11$ gives insufficient specificity).
|
||||||
|
- $k$ **is odd**: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see [DNA encoding](encoding.md)) to be well defined.
|
||||||
+3
-17
@@ -2,15 +2,7 @@
|
|||||||
|
|
||||||
## Definition
|
## Definition
|
||||||
|
|
||||||
The **minimizer** of a k-mer is the canonical form of the m-mer that is smallest according to a chosen ordering among the $k-m+1$ overlapping m-mers contained in the k-mer, with $m<k$ [@Roberts2004-rz].
|
The **minimizer** of a k-mer is the canonical form of the m-mer that is smallest according to a chosen ordering among the $k-m+1$ overlapping m-mers contained in the k-mer, with $m<k$ [@Roberts2004-rz]. Canonical form is the same operation defined for a full kmer (see [DNA encoding](encoding.md)), applied here to an m-mer $s$ instead: $s^c = \min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right)$.
|
||||||
|
|
||||||
For an m-mer $s$, its **canonical form** is defined as
|
|
||||||
|
|
||||||
$$
|
|
||||||
s^c = \min_{\mathrm{lex}}\left(s,\operatorname{RC}(s)\right),
|
|
||||||
$$
|
|
||||||
|
|
||||||
where $\operatorname{RC}(s)$ denotes the reverse complement of $s$.
|
|
||||||
|
|
||||||
In **OBIkmer**, canonical m-mers are ordered according to a deterministic hash function $H$ applied to their canonical form:
|
In **OBIkmer**, canonical m-mers are ordered according to a deterministic hash function $H$ applied to their canonical form:
|
||||||
|
|
||||||
@@ -36,7 +28,7 @@ $$
|
|||||||
|
|
||||||
In other words, the hash function defines the ordering of canonical m-mers, while the minimizer itself is the canonical form of the m-mer selected by that ordering.
|
In other words, the hash function defines the ordering of canonical m-mers, while the minimizer itself is the canonical form of the m-mer selected by that ordering.
|
||||||
|
|
||||||
Applying the minimizer to consecutive k-mers partitions a sequence into super-kmers, defined as maximal runs of consecutive k-mers sharing the same minimizer (see [Kmers and super-kmers](kmers_and_superkmers.md)).
|
Applying the minimizer to consecutive k-mers partitions a sequence into super-kmers (see [Super-kmers](superkmers.md)).
|
||||||
|
|
||||||
## Hash-based minimizer ordering
|
## Hash-based minimizer ordering
|
||||||
|
|
||||||
@@ -87,10 +79,4 @@ Exhaustive enumeration of all $4^m$ m-mers confirms that, with $\sigma=\texttt{0
|
|||||||
|
|
||||||
When the minimal period of the sequence is $m$, as observed in the table above, the sequence is primitive: it cannot be represented as repetitions of a shorter sequence.
|
When the minimal period of the sequence is $m$, as observed in the table above, the sequence is primitive: it cannot be represented as repetitions of a shorter sequence.
|
||||||
|
|
||||||
## Partition routing
|
See [Partitioning and indexing architecture](indexing_architecture.md) for how this hash routes a super-kmer to a partition.
|
||||||
|
|
||||||
A super-kmer's partition is the low $p$ bits of its minimizer's hash:
|
|
||||||
|
|
||||||
$$\text{partition} = H(\text{minimizer}) \bmod 2^p$$
|
|
||||||
|
|
||||||
See [Partitioning and indexing architecture](indexing_architecture.md) for more details.
|
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# Super-kmers
|
||||||
|
|
||||||
|
A **super-kmer** is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see [Minimizer selection](minimizer_selection.md)). Each kmer in the run overlaps the next by $k-1$ nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
|
||||||
|
|
||||||
|
For a random minimizer of length $m$ over kmers of length $k$, the expected length of a super-kmer is approximately [@Zheng2020-ji; @Golan2025-xf]:
|
||||||
|
|
||||||
|
$$L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1$$
|
||||||
|
|
||||||
|
For $k=31$, $m=13$ this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.
|
||||||
|
|
||||||
|
## Canonical super-kmers
|
||||||
|
|
||||||
|
A **canonical super-kmer** is the canonical form of a super-kmer (see [DNA encoding](encoding.md)): the lexicographic minimum of the super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.
|
||||||
|
|
||||||
|
Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
# Kmers and super-kmers
|
|
||||||
|
|
||||||
## Kmers
|
|
||||||
|
|
||||||
A **kmer** is a DNA subsequence of fixed length $k$. Two constraints apply to $k$, both enforced when a command starts (an invalid value exits immediately with an error):
|
|
||||||
|
|
||||||
- $k \in [11, 31]$: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base ($k \le 32$ is the hard limit; $k < 11$ gives insufficient specificity).
|
|
||||||
- $k$ **is odd**: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see [DNA encoding](encoding.md)) to be well defined.
|
|
||||||
|
|
||||||
## Super-kmers
|
|
||||||
|
|
||||||
A **super-kmer** is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see [Minimizer selection](minimizer_selection.md)). Each kmer in the run overlaps the next by $k-1$ nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.
|
|
||||||
|
|
||||||
For a random minimizer of length $m$ over kmers of length $k$, the expected length of a super-kmer is approximately [@Zheng2020-ji; @Golan2025-xf]:
|
|
||||||
|
|
||||||
$$L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1$$
|
|
||||||
|
|
||||||
For $k=31$, $m=13$ this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.
|
|
||||||
|
|
||||||
### Canonical super-kmers
|
|
||||||
|
|
||||||
A **canonical super-kmer** is the lexicographic minimum of a super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.
|
|
||||||
|
|
||||||
Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.
|
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
# Whole k-mer set distances
|
||||||
|
|
||||||
|
The k-mer composition of a genome provides a representation of its sequence content that can be compared directly between genomes. The distances considered here quantify dissimilarity between genomes from their complete k-mer composition, independently of any particular representation, storage scheme, or indexing strategy.
|
||||||
|
|
||||||
|
Two complementary classes of comparison arise according to the information retained about k-mer occurrences [@Anderson2011-rq]. **Presence/absence-based distances** consider only whether each distinct k-mer occurs in a genome, whereas **abundance-based distances** also incorporate its observed frequency. Abundance-based measures may be computed either from raw k-mer counts or from the corresponding relative frequencies.
|
||||||
|
|
||||||
|
Let $A$ and $B$ denote the k-mer sets of two genomes. For each distinct k-mer $i$, let $c_i^A$ and $c_i^B$ denote its counts in the two genomes, and let
|
||||||
|
|
||||||
|
$$
|
||||||
|
p_i^A=\frac{c_i^A}{\sum_j c_j^A},
|
||||||
|
\qquad
|
||||||
|
p_i^B=\frac{c_i^B}{\sum_j c_j^B}
|
||||||
|
$$
|
||||||
|
|
||||||
|
denote the corresponding relative frequencies.
|
||||||
|
|
||||||
|
The resulting measures can be classified along two independent dimensions: the **information retained about k-mer occurrences** (presence/absence versus abundance) and the **metric properties** of the resulting dissimilarity, in particular whether the triangle inequality is satisfied.
|
||||||
|
|
||||||
|
## Presence/absence-based distances
|
||||||
|
|
||||||
|
When only occurrence is retained, each genome is represented by its incidence vector over the k-mer space. The comparison therefore depends exclusively on the presence or absence of each distinct k-mer and is insensitive to its multiplicity.
|
||||||
|
|
||||||
|
The **Jaccard distance** compares two k-mer sets from the cardinalities of their intersection and union [@Jaccard1901-mx]:
|
||||||
|
|
||||||
|
$$
|
||||||
|
D = 1 - \frac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}.
|
||||||
|
$$
|
||||||
|
|
||||||
|
Jaccard distance is a metric and therefore satisfies the triangle inequality [@Kosub2019-xt].
|
||||||
|
|
||||||
|
The **Hamming distance** is the $L^1$ distance between the corresponding binary incidence vectors:
|
||||||
|
|
||||||
|
$$
|
||||||
|
D = \sum_i \mathbb{1}[a_i \ne b_i],
|
||||||
|
$$
|
||||||
|
|
||||||
|
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded.
|
||||||
|
|
||||||
|
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
|
||||||
|
|
||||||
|
$$
|
||||||
|
J = 1-D_{\mathrm{Jaccard}},
|
||||||
|
$$
|
||||||
|
|
||||||
|
the Mash distance is defined as
|
||||||
|
|
||||||
|
$$
|
||||||
|
D = -\frac{1}{k}\ln\left(\frac{2J}{1+J}\right)
|
||||||
|
= -\frac{1}{k}\ln\left(\frac{2(1-J_{\mathrm{Jaccard}})}
|
||||||
|
{2-J_{\mathrm{Jaccard}}}\right),
|
||||||
|
$$
|
||||||
|
|
||||||
|
with the result clamped to $1.0$ when the Jaccard similarity reaches zero. This transformation is monotone increasing with Jaccard distance and therefore preserves the ordering of pairwise dissimilarities. Monotonicity alone, however, does not guarantee preservation of the triangle inequality. Mash should consequently be regarded as an evolutionary distance estimator derived from k-mer similarity rather than assumed to be a metric.
|
||||||
|
|
||||||
|
## Abundance-based distances
|
||||||
|
|
||||||
|
When k-mer abundances are retained, genomes are represented by non-negative vectors of k-mer counts or frequencies. Distances can then distinguish genomes that contain the same k-mers but differ in their relative abundances.
|
||||||
|
|
||||||
|
The **Euclidean distance** is the ordinary $L^2$ distance between the corresponding vectors. On raw counts it is
|
||||||
|
|
||||||
|
$$
|
||||||
|
D = \sqrt{\sum_i (c_i^A-c_i^B)^2},
|
||||||
|
$$
|
||||||
|
|
||||||
|
and the same definition applied to relative frequencies gives `relfreq-euclidean`. In both cases, Euclidean distance is a metric.
|
||||||
|
|
||||||
|
The **Hellinger distance** applies the Euclidean distance to the square-root-transformed relative frequencies [@Legendre2001-cq]:
|
||||||
|
|
||||||
|
$$
|
||||||
|
D =
|
||||||
|
\frac{1}{\sqrt{2}}\sqrt{\sum_i\left(\sqrt{p_i^A}-\sqrt{p_i^B}\right)^2}.
|
||||||
|
$$
|
||||||
|
|
||||||
|
It is bounded between 0 and 1 and is a metric because it is a positive rescaling of the Euclidean distance between the transformed vectors $\sqrt{p^A}$ and $\sqrt{p^B}$. Its unnormalized form, `hellinger-euclidean`,
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
D &=\sqrt{\sum_i\left(\sqrt{p_i^A}-\sqrt{p_i^B}\right)^2} \\
|
||||||
|
&=\sqrt{2}\,D_{\mathrm{Hellinger}},
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
is likewise a metric, since multiplication by a positive constant preserves the triangle inequality.
|
||||||
|
|
||||||
|
The **Bray–Curtis dissimilarity** [@Bray1957-vv] is defined from the amount of abundance shared between two vectors:
|
||||||
|
|
||||||
|
$$
|
||||||
|
D = 1-\frac{2\sum_i\min(c_i^A,c_i^B)}{\sum_i c_i^A+\sum_i c_i^B}.
|
||||||
|
$$
|
||||||
|
|
||||||
|
Using
|
||||||
|
|
||||||
|
$$
|
||||||
|
a+b-2\min(a,b)=|a-b|,
|
||||||
|
$$
|
||||||
|
|
||||||
|
this can equivalently be written as
|
||||||
|
|
||||||
|
$$
|
||||||
|
D =
|
||||||
|
\frac{\sum_i|c_i^A-c_i^B|}
|
||||||
|
{\sum_i c_i^A+\sum_i c_i^B}.
|
||||||
|
$$
|
||||||
|
|
||||||
|
For arbitrary count vectors, **Bray–Curtis** is not a metric [@Gower1986-ri]: the pair-dependent normalization by the combined abundance can lead to violations of the triangle inequality. This distinguishes it from the $L^1$-based distances above, whose normalization factor is fixed independently of the pair being compared.
|
||||||
|
|
||||||
|
The metric property is recovered when all vectors under comparison have the same total abundance $S$. In that case,
|
||||||
|
|
||||||
|
$$
|
||||||
|
\sum_i c_i^A=\sum_i c_i^B=S
|
||||||
|
$$
|
||||||
|
|
||||||
|
for every pair, and **Bray–Curtis** reduces to
|
||||||
|
|
||||||
|
$$
|
||||||
|
D =
|
||||||
|
\frac{1}{2S}
|
||||||
|
\sum_i|c_i^A-c_i^B|.
|
||||||
|
$$
|
||||||
|
|
||||||
|
It is therefore a positive rescaling of the $L^1$ distance and is a metric on this fixed-total space.
|
||||||
|
|
||||||
|
Relative frequencies satisfy this condition by construction, since
|
||||||
|
|
||||||
|
$$
|
||||||
|
\sum_i p_i^A=\sum_i p_i^B=1.
|
||||||
|
$$
|
||||||
|
|
||||||
|
Consequently, `relfreq-bray-curtis` reduces to
|
||||||
|
|
||||||
|
$$
|
||||||
|
D =
|
||||||
|
\frac12\sum_i|p_i^A-p_i^B|,
|
||||||
|
$$
|
||||||
|
|
||||||
|
which is the **total variation distance** between the two frequency distributions. It is therefore a metric.
|
||||||
|
|
||||||
|
Thus, `bray-curtis` and `relfreq-bray-curtis` have the same algebraic form but operate on different domains: the former compares count vectors whose total abundance may vary between genomes, whereas the latter compares normalized frequency vectors with a fixed total. This difference in normalization is sufficient to change the metric properties of the measure.
|
||||||
|
|
||||||
|
See [`phylo`](../../../usage/phylo.md) for how to select a distance and the resulting output formats.
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
# Central-position SNP model
|
||||||
|
|
||||||
|
## Family
|
||||||
|
|
||||||
|
A **family** is the set of up to 4 kmers sharing identical flanking sequence and differing only at the central base — the odd kmer length guarantees a single, well-defined central position (see [Kmers](../../kmer_indexing/kmers.md)). A family is **eligible** for a genome pair $(i,j)$ only if both genomes carry exactly one of its observed forms (single-copy, unambiguous) — this excludes multi-copy and absent loci from the comparison rather than folding them into an undifferentiated "not identical" bucket the way a whole-index distance would.
|
||||||
|
|
||||||
|
Conditioning on eligibility this way restricts every comparison to loci that are directly, positively confirmed comparable in both genomes: a locus never enters the statistic because of a genuinely absent homologous region, a diverged paralogous copy, or a genome-size asymmetry — only because a real single-copy substitution (or lack of one) was observed at flanks confirmed intact in both genomes.
|
||||||
|
|
||||||
|
## Substitution corrections (`snp-*`)
|
||||||
|
|
||||||
|
For a genome pair, let $L$ be its total number of eligible loci, $p$ the raw proportion of substitutions among those loci, $P$/$Q$ the transition/transversion proportions, $Q_1$/$Q_2$ Kimura's two transversion categories (A↔C & G↔T vs. A↔T & C↔G), $P_1$/$P_2$ the purine (A↔G) / pyrimidine (C↔T) transition proportions, and $\pi_A,\pi_C,\pi_G,\pi_T$ the pair's pooled base frequencies.
|
||||||
|
|
||||||
|
**`snp-raw`**
|
||||||
|
|
||||||
|
$$d = p$$
|
||||||
|
|
||||||
|
**`snp-jc`**
|
||||||
|
|
||||||
|
$$d = -\frac{3}{4}\ln\!\left(1-\frac{4p}{3}\right)$$
|
||||||
|
|
||||||
|
**`snp-k2p`**
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
a_1 &= 1-2P-Q \\
|
||||||
|
a_2 &= 1-2Q \\
|
||||||
|
d &= -\frac{1}{2}\ln a_1-\frac{1}{4}\ln a_2
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
**`snp-k81`**
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
a_1 &= 1-2P-2Q_1 \\
|
||||||
|
a_2 &= 1-2P-2Q_2 \\
|
||||||
|
a_3 &= 1-2Q_1-2Q_2 \\
|
||||||
|
d &= -\frac{1}{4}\left(\ln a_1+\ln a_2+\ln a_3\right)
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
**`snp-f81`**
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
E &= 1-\left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right) \\
|
||||||
|
d &= -E\ln\!\left(1-\frac{p}{E}\right)
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
**`snp-t92`**
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
g &= \pi_C+\pi_G \\
|
||||||
|
w &= 2g(1-g) \\
|
||||||
|
a_1 &= 1-\frac{P}{w}-Q \\
|
||||||
|
a_2 &= 1-2Q \\
|
||||||
|
d &= -w\ln a_1-\frac{1}{2}(1-w)\ln a_2
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
**`snp-tn93`**
|
||||||
|
|
||||||
|
$$
|
||||||
|
\begin{aligned}
|
||||||
|
g_R &= \pi_A+\pi_G \\
|
||||||
|
g_Y &= \pi_C+\pi_T \\
|
||||||
|
k_1 &= \frac{2\pi_A\pi_G}{g_R} \\
|
||||||
|
k_2 &= \frac{2\pi_C\pi_T}{g_Y} \\
|
||||||
|
k_3 &= 2\left(g_Rg_Y-\frac{\pi_A\pi_G\,g_Y}{g_R}-\frac{\pi_C\pi_T\,g_R}{g_Y}\right) \\
|
||||||
|
w_1 &= 1-\frac{P_1}{k_1}-\frac{Q}{2g_R} \\
|
||||||
|
w_2 &= 1-\frac{P_2}{k_2}-\frac{Q}{2g_Y} \\
|
||||||
|
w_3 &= 1-\frac{Q}{2g_Rg_Y} \\
|
||||||
|
d &= -k_1\ln w_1-k_2\ln w_2-k_3\ln w_3
|
||||||
|
\end{aligned}
|
||||||
|
$$
|
||||||
|
|
||||||
|
**`snp-tv`** — transversions only, deliberately uncorrected:
|
||||||
|
|
||||||
|
$$d = Q$$
|
||||||
|
|
||||||
|
A rate-heterogeneity correction applies to every formula above except `snp-raw` and `snp-tv`: each $-\ln(x)$ term is replaced by $\alpha\left(x^{-1/\alpha}-1\right)$ (same weight, same $x$) — see [Gamma rate heterogeneity](gamma_rate_heterogeneity.md) for the correction itself and how $\alpha$ is estimated.
|
||||||
|
|
||||||
|
See [`phylo`](../../../usage/phylo.md) for how to select a `snp-*` value and the prerequisite index annex it requires.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Gamma rate heterogeneity
|
||||||
|
|
||||||
|
Real substitution rates vary across sites rather than being uniform, which a plain substitution correction (see [Central-position SNP model](central_snp_model.md)) ignores. The standard $+\Gamma$ correction models this by replacing each $-\ln(x)$ term in a correction formula with $\alpha\left(x^{-1/\alpha}-1\right)$ — the same $x$, the same weight, under the assumption that the per-site rate is $\mathrm{Gamma}(\alpha,\alpha)$-distributed (mean 1) rather than fixed. A small $\alpha$ indicates strong among-site rate heterogeneity (many near-invariant sites, a few fast ones); as $\alpha$ grows large the correction converges to the uncorrected formula.
|
||||||
|
|
||||||
|
## Automatic $\alpha$ estimation
|
||||||
|
|
||||||
|
$\alpha$ can be estimated automatically from the index itself, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.
|
||||||
|
|
||||||
|
The estimator pools substitution counts by **partition** rather than by genome pair: for partition $i$, let $n_i$ be the total number of substitutions observed across every genome pair, and $L_i$ the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:
|
||||||
|
|
||||||
|
$$R_i = \frac{n_i}{L_i}$$
|
||||||
|
|
||||||
|
Under a single shared substitution rate with no among-site heterogeneity, each $R_i$ would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as in the correction itself, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:
|
||||||
|
|
||||||
|
$$\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}$$
|
||||||
|
|
||||||
|
where $\mu$ is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own $L_i$ removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:
|
||||||
|
|
||||||
|
$$\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}}$$
|
||||||
|
|
||||||
|
$$\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}$$
|
||||||
|
|
||||||
|
If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), $\alpha$ is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged.
|
||||||
|
|
||||||
|
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
|
||||||
|
|
||||||
|
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation.
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Sampling design
|
||||||
|
|
||||||
|
Exhaustive computation over every non-monomorphic family in an index (see [Central-position SNP model](central_snp_model.md)) is the default, but can be bounded to approximately `N` families instead. Sampling is designed around two properties: staying representative of the whole index, and letting the user favor families that actually carry signal.
|
||||||
|
|
||||||
|
## Proportional sampling
|
||||||
|
|
||||||
|
Families are drawn in proportion to how many candidate families each part of the index actually holds, rather than uniformly across index partitions — this keeps the sample's composition representative of the whole index regardless of how unevenly candidate families happen to be distributed across partitions.
|
||||||
|
|
||||||
|
## Per-family informativeness: Shannon entropy
|
||||||
|
|
||||||
|
A family's informativeness is measured directly rather than assumed uniform. For a family, entropy is computed over its observed states across genomes:
|
||||||
|
|
||||||
|
| Quantity | Definition |
|
||||||
|
|---|---|
|
||||||
|
| `entropy15` | Shannon entropy (bits) over the 16 possible states (the 15 non-empty subsets of `{A,C,G,T}`), genomes absent from the family excluded from the count |
|
||||||
|
| `entropy4` | the same, reduced to the 4 plain bases |
|
||||||
|
|
||||||
|
A family where every genome carries the same single state has entropy 0 (uninformative); one where genomes are spread evenly across several states has higher entropy (more informative for distinguishing genomes).
|
||||||
|
|
||||||
|
## Entropy-biased sampling
|
||||||
|
|
||||||
|
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
||||||
|
|
||||||
|
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---subsample---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# The 16-state Sankoff model
|
||||||
|
|
||||||
|
Each family (see [Central-position SNP model](central_snp_model.md)) is treated as a character with 16 possible states: one per subset of the 4 possible central bases, including the empty subset (no observed form). A genome pair's calibration reduces to two separately-estimated first-order Markov models, both restricted to genome pairs below a configurable SNP-ratio ceiling (excluding saturated pairs from skewing the calibration):
|
||||||
|
|
||||||
|
- **Cardinality transitions** ($5 \times 5$, row-stochastic): the probability that a family observed with $i$ forms in one genome ($i \in \{0,\dots,4\}$) is observed with $j$ forms in the other.
|
||||||
|
- **Composition transitions** ($4 \times 4$, row-stochastic): the probability that an unambiguous, single-copy base observed as one of A/C/G/T in one genome is observed as another base in the other genome.
|
||||||
|
|
||||||
|
## Combining into a 16×16 cost matrix
|
||||||
|
|
||||||
|
The two matrices are *not* combined as a plain tensor product. For a pair of states $(A, B)$ — each a subset of $\{A,C,G,T\}$ — let `shared` $= A \cap B$, `lost` $= A \setminus B$, `gained` $= B \setminus A$. The log-probability of the transition $A \to B$ is:
|
||||||
|
|
||||||
|
$$\log P(A,B) = \underbrace{\log P_{\text{card}}(|A|,|B|)}_{\text{cardinality term, omitted when loss events are treated as free}} \;+\; \underbrace{\sum_{i \,\in\, \text{shared}} \log P_{\text{comp}}(i,i)}_{\text{bases conserved on both sides}} \;-\; \underbrace{\text{best\_pairing\_cost}(\text{lost}, \text{gained})}_{\text{substitutions among the differing bases}}$$
|
||||||
|
|
||||||
|
`best_pairing_cost` enumerates every injection pairing elements of `lost` with elements of `gained` and keeps the one minimizing $-\sum \log P_{\text{comp}}$ over the pairs — the discrete analogue of "prefer a substitution over an independent loss and gain": if a base is lost from one side and a different base is gained on the other, that is scored as a single substitution between them (cheaper, under any sane calibration, than treating them as two unrelated cardinality-changing events) whenever such a pairing is possible.
|
||||||
|
|
||||||
|
The resulting $16 \times 16$ log-probability matrix is row-normalized in log-space (log-sum-exp, not a direct `exp()`/divide, for numerical stability), giving a genuine row-stochastic transition matrix $P$. The cost matrix is then $\text{cost}(A,B) = -\ln P(A,B)$, symmetrized by simple averaging:
|
||||||
|
|
||||||
|
$$\text{sym}(A,B) = \frac{\text{cost}(A,B) + \text{cost}(B,A)}{2}$$
|
||||||
|
|
||||||
|
This calibrated cost matrix is what every downstream export (for TNT, PhyG, IQ-TREE) reuses, recoded into each tool's own format.
|
||||||
|
|
||||||
|
See [`phylo`](../../../usage/phylo.md#sankoff-calibration-and-phylogenetic-exports) for how to run the calibration and produce these exports.
|
||||||
+9
-125
@@ -17,131 +17,17 @@ obikmer phylo INDEX [OPTIONS]
|
|||||||
| Option | Default | Description |
|
| Option | Default | Description |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `--distance` | `jaccard` | See the two tables below for the full list of accepted values |
|
| `--distance` | `jaccard` | See the two tables below for the full list of accepted values |
|
||||||
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see below). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data (see "Automatic $\alpha$ estimation" below). No effect on the other values; rejected if given together with a value that doesn't support it |
|
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
|
||||||
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
|
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
|
||||||
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
|
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
|
||||||
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index metric, not a `snp-*` value |
|
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index distance, not a `snp-*` value |
|
||||||
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
|
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
|
||||||
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
|
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
|
||||||
| `-o, --output` | none (stdout) | Output file prefix |
|
| `-o, --output` | none (stdout) | Output file prefix |
|
||||||
|
|
||||||
Every value routes to one of two independent computations:
|
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
||||||
|
|
||||||
### Whole-index metrics
|
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||||
|
|
||||||
| Value | Definition |
|
|
||||||
|---|---|
|
|
||||||
| `jaccard` | $D = 1 - \dfrac{\lvert A \cap B \rvert}{\lvert A \cup B \rvert}$ over the sets of kmers present in each genome |
|
|
||||||
| `mash` | derived from the Jaccard distance via $D = -\dfrac{1}{k} \ln\!\left(\dfrac{2J}{1+J}\right)$ where $J = 1 - D_{\text{jaccard}}$ and $k$ is the index's kmer size; clamped to 1.0 when $J \le 0$ |
|
|
||||||
| `hamming` | number of kmer positions where presence differs between the two genomes (presence index only, not normalized): $D = \sum_i \mathbb{1}[a_i \ne b_i]$ |
|
|
||||||
| `bray-curtis` | $D = 1 - \dfrac{2 \sum_i \min(c_i^A, c_i^B)}{\sum_i c_i^A + \sum_i c_i^B}$ on raw per-kmer counts |
|
|
||||||
| `relfreq-bray-curtis` | the same formula computed on per-genome relative frequencies $p_i = c_i / \sum_j c_j$ instead of raw counts |
|
|
||||||
| `euclidean` | $D = \sqrt{\sum_i (c_i^A - c_i^B)^2}$ on raw counts |
|
|
||||||
| `relfreq-euclidean` | the same formula on relative frequencies |
|
|
||||||
| `hellinger` | $D = \dfrac{1}{\sqrt{2}} \sqrt{\sum_i \left(\sqrt{p_i^A} - \sqrt{p_i^B}\right)^2}$ on relative frequencies, bounded in $[0, 1]$ |
|
|
||||||
| `hellinger-euclidean` | the unnormalized variant, $D = \sqrt{2} \times D_{\text{hellinger}}$ |
|
|
||||||
|
|
||||||
`hamming` requires a presence/absence index; the others work on either index type.
|
|
||||||
|
|
||||||
### `snp-*` corrections
|
|
||||||
|
|
||||||
Computed from the central-position SNP model (see "Central-position SNP model" below): a family is the set of up to 4 kmers sharing identical flanking sequence and differing only at the central base. These values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--subsample N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same flag `--pseudo-alignment`/`--sankoff` use, but optional here).
|
|
||||||
|
|
||||||
For a genome pair, let $L$ be its total number of eligible loci (both genomes single-copy at that family), $p$ the raw proportion of substitutions among those loci, $P$/$Q$ the transition/transversion proportions, $Q_1$/$Q_2$ Kimura's two transversion categories (A↔C & G↔T vs. A↔T & C↔G), $P_1$/$P_2$ the purine (A↔G) / pyrimidine (C↔T) transition proportions, and $\pi_A,\pi_C,\pi_G,\pi_T$ the pair's pooled base frequencies.
|
|
||||||
|
|
||||||
**`snp-raw`**
|
|
||||||
|
|
||||||
$$d = p$$
|
|
||||||
|
|
||||||
**`snp-jc`**
|
|
||||||
|
|
||||||
$$d = -\frac{3}{4}\ln\!\left(1-\frac{4p}{3}\right)$$
|
|
||||||
|
|
||||||
**`snp-k2p`**
|
|
||||||
|
|
||||||
$$
|
|
||||||
\begin{aligned}
|
|
||||||
a_1 &= 1-2P-Q \\
|
|
||||||
a_2 &= 1-2Q \\
|
|
||||||
d &= -\frac{1}{2}\ln a_1-\frac{1}{4}\ln a_2
|
|
||||||
\end{aligned}
|
|
||||||
$$
|
|
||||||
|
|
||||||
**`snp-k81`**
|
|
||||||
|
|
||||||
$$
|
|
||||||
\begin{aligned}
|
|
||||||
a_1 &= 1-2P-2Q_1 \\
|
|
||||||
a_2 &= 1-2P-2Q_2 \\
|
|
||||||
a_3 &= 1-2Q_1-2Q_2 \\
|
|
||||||
d &= -\frac{1}{4}\left(\ln a_1+\ln a_2+\ln a_3\right)
|
|
||||||
\end{aligned}
|
|
||||||
$$
|
|
||||||
|
|
||||||
**`snp-f81`**
|
|
||||||
|
|
||||||
$$
|
|
||||||
\begin{aligned}
|
|
||||||
E &= 1-\left(\pi_A^2+\pi_C^2+\pi_G^2+\pi_T^2\right) \\
|
|
||||||
d &= -E\ln\!\left(1-\frac{p}{E}\right)
|
|
||||||
\end{aligned}
|
|
||||||
$$
|
|
||||||
|
|
||||||
**`snp-t92`**
|
|
||||||
|
|
||||||
$$
|
|
||||||
\begin{aligned}
|
|
||||||
g &= \pi_C+\pi_G \\
|
|
||||||
w &= 2g(1-g) \\
|
|
||||||
a_1 &= 1-\frac{P}{w}-Q \\
|
|
||||||
a_2 &= 1-2Q \\
|
|
||||||
d &= -w\ln a_1-\frac{1}{2}(1-w)\ln a_2
|
|
||||||
\end{aligned}
|
|
||||||
$$
|
|
||||||
|
|
||||||
**`snp-tn93`**
|
|
||||||
|
|
||||||
$$
|
|
||||||
\begin{aligned}
|
|
||||||
g_R &= \pi_A+\pi_G \\
|
|
||||||
g_Y &= \pi_C+\pi_T \\
|
|
||||||
k_1 &= \frac{2\pi_A\pi_G}{g_R} \\
|
|
||||||
k_2 &= \frac{2\pi_C\pi_T}{g_Y} \\
|
|
||||||
k_3 &= 2\left(g_Rg_Y-\frac{\pi_A\pi_G\,g_Y}{g_R}-\frac{\pi_C\pi_T\,g_R}{g_Y}\right) \\
|
|
||||||
w_1 &= 1-\frac{P_1}{k_1}-\frac{Q}{2g_R} \\
|
|
||||||
w_2 &= 1-\frac{P_2}{k_2}-\frac{Q}{2g_Y} \\
|
|
||||||
w_3 &= 1-\frac{Q}{2g_Rg_Y} \\
|
|
||||||
d &= -k_1\ln w_1-k_2\ln w_2-k_3\ln w_3
|
|
||||||
\end{aligned}
|
|
||||||
$$
|
|
||||||
|
|
||||||
**`snp-tv`** — transversions only, deliberately uncorrected:
|
|
||||||
|
|
||||||
$$d = Q$$
|
|
||||||
|
|
||||||
`--gamma-shape ALPHA` applies to every value above except `snp-raw` and `snp-tv`: each $-\ln(x)$ term in the formulas above is replaced by $\alpha\left(x^{-1/\alpha}-1\right)$ (the same weight, same $x$).
|
|
||||||
|
|
||||||
### Automatic $\alpha$ estimation (`--gamma-shape auto`)
|
|
||||||
|
|
||||||
`--gamma-shape auto` (or the equivalent `--gamma-shape estimate`) fits $\alpha$ from the index itself instead of requiring a user-supplied value, using a method-of-moments estimator computed once, from the same sampling pass that builds the pairwise substitution tally — no extra scan of the index.
|
|
||||||
|
|
||||||
The estimator pools substitution counts by **partition** rather than by genome pair: for partition $i$, let $n_i$ be the total number of substitutions observed across every genome pair, and $L_i$ the total number of eligible loci across every genome pair, in that partition. Define the partition's observed substitution rate:
|
|
||||||
|
|
||||||
$$R_i = \frac{n_i}{L_i}$$
|
|
||||||
|
|
||||||
Under a single shared substitution rate with no among-site heterogeneity, each $R_i$ would vary only by Poisson sampling noise. Rate heterogeneity is modeled, as elsewhere in this correction, by a $\mathrm{Gamma}(\alpha,\alpha)$-distributed multiplicative rate (mean 1) shared by every locus in a partition — the classical Poisson–Gamma (negative-binomial) mixture. Under that model:
|
|
||||||
|
|
||||||
$$\mathbb{E}[R_i] = \mu \qquad \mathrm{Var}[R_i] = \frac{\mu}{L_i} + \frac{\mu^2}{\alpha}$$
|
|
||||||
|
|
||||||
where $\mu$ is the pooled substitution rate across every partition. Weighting each partition's squared deviation by its own $L_i$ removes the first (Poisson) term before attributing what's left to genuine rate heterogeneity:
|
|
||||||
|
|
||||||
$$\hat\mu = \frac{\sum_i n_i}{\sum_i L_i} \qquad V = \frac{\sum_i L_i\,(R_i-\hat\mu)^2}{\sum_i L_i} \qquad \bar L = \frac{\sum_i L_i}{\text{number of partitions}}$$
|
|
||||||
|
|
||||||
$$\hat\alpha = \frac{\hat\mu^2}{V - \hat\mu/\bar L}$$
|
|
||||||
|
|
||||||
If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (no detectable over-dispersion across partitions — the data are consistent with a single shared rate), $\alpha$ is left undefined: the correction is silently disabled for that run rather than applying a fabricated value, and a warning is logged. When an estimate is produced, it's logged at the `info` level before the distance matrix is computed.
|
|
||||||
|
|
||||||
Note: this is a method-of-moments estimator derived from the standard Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition — it is not part of Jin & Nei's (1990) original publication, which only defines the `+Γ` distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ (`--gamma-shape 1`) rather than proposing a way to estimate it from data. `alpha < 1` indicates strong among-site rate heterogeneity (many near-invariant loci, a few fast ones); `alpha` growing large makes the correction converge to the uncorrected formula.
|
|
||||||
|
|
||||||
### Output
|
### Output
|
||||||
|
|
||||||
@@ -152,13 +38,13 @@ Without `-o`, the matrix goes to stdout in relaxed-PHYLIP format (`n` on the fir
|
|||||||
| Option | Description |
|
| Option | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `--exclude-genome LABEL` | Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by `--pseudo-alignment`/`--sankoff`/a `snp-*` `--distance` value. Does not change the value computed for any remaining pair |
|
| `--exclude-genome LABEL` | Exclude a genome (repeatable). Drops its row/column from the distance/shared-kmer matrix output, and removes it from the sampling used by `--pseudo-alignment`/`--sankoff`/a `snp-*` `--distance` value. Does not change the value computed for any remaining pair |
|
||||||
| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` — never to the whole-index metrics or their matrix/NJ/UPGMA output |
|
| `--min-shared-family N` | Auto-exclude, on top of `--exclude-genome`, any genome whose mean shared-family count against every other genome (see "Family Overlap" below) falls below `N`. Applies only to `--pseudo-alignment`/`--sankoff`/`snp-*` `--distance` — never to the whole-index distances or their matrix/NJ/UPGMA output |
|
||||||
|
|
||||||
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
|
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
|
||||||
|
|
||||||
## Central-position SNP model
|
## Central-position SNP model
|
||||||
|
|
||||||
Requires the sibling annex, built once per index:
|
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index:
|
||||||
|
|
||||||
| Option | Description |
|
| Option | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -169,8 +55,6 @@ Requires the sibling annex, built once per index:
|
|||||||
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
||||||
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--subsample N` |
|
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--subsample N` |
|
||||||
|
|
||||||
A family is eligible for a genome pair $(i,j)$ only if both genomes carry exactly one of its observed forms (single-copy, unambiguous).
|
|
||||||
|
|
||||||
### `--sibling-stats`
|
### `--sibling-stats`
|
||||||
|
|
||||||
`<prefix>_siblings.csv` — family size = number of distinct central bases observed at a family (1-4).
|
`<prefix>_siblings.csv` — family size = number of distinct central bases observed at a family (1-4).
|
||||||
@@ -219,7 +103,7 @@ A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sa
|
|||||||
|
|
||||||
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
||||||
|
|
||||||
Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead, drawn in proportion to how many candidate families each part of the index actually holds, so the sample stays representative of the whole index. If the index has fewer than `N` candidate families, `--subsample` has no effect.
|
Without `--subsample`, every variable family (family size ≥ 2) is used. With `--subsample N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If the index has fewer than `N` candidate families, `--subsample` has no effect.
|
||||||
|
|
||||||
### `--shannon`: measuring how informative a family is
|
### `--shannon`: measuring how informative a family is
|
||||||
|
|
||||||
@@ -238,7 +122,7 @@ Run with `--subsample N --shannon` to get a bounded diagnostic sample instead of
|
|||||||
|
|
||||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||||
|
|
||||||
By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff. The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
By default, `--subsample` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--subsample N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||||
|
|
||||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
||||||
|
|
||||||
@@ -257,7 +141,7 @@ The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays
|
|||||||
|
|
||||||
### The 16-state model
|
### The 16-state model
|
||||||
|
|
||||||
Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). Calibration combines a $5 \times 5$ transition matrix over family cardinality (0-4 observed forms) and a $4 \times 4$ base-substitution matrix from unambiguous single-copy loci, both restricted to genome pairs at or below `--sankoff-ratio-ceiling`, into a row-normalized $16 \times 16$ transition probability matrix $P$, converted to a symmetric cost matrix via $\text{cost}(a,b) = -\ln P(a,b)$.
|
Each family is a character with 16 possible states: one per subset of the 4 possible central bases (including the empty subset). See [The 16-state Sankoff model](../theory/phylogeny/snp_based/sankoff_model.md) for how the calibrated cost matrix is built from cardinality and base-composition transitions.
|
||||||
|
|
||||||
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character). It does not run any external tool.
|
`--sankoff` alone writes the cost matrix, the calibration parameters, and a pseudo-alignment recoded so the empty state uses the symbol `0` (never a gap character). It does not run any external tool.
|
||||||
|
|
||||||
|
|||||||
+15
-5
@@ -31,11 +31,21 @@ nav:
|
|||||||
- Home: index.md
|
- Home: index.md
|
||||||
- Installation: installation.md
|
- Installation: installation.md
|
||||||
- Theory:
|
- Theory:
|
||||||
- Kmers and super-kmers: theory/kmers_and_superkmers.md
|
- Kmer indexing:
|
||||||
- DNA encoding: theory/encoding.md
|
- DNA encoding: theory/kmer_indexing/encoding.md
|
||||||
- Low-complexity kmer filter: theory/entropy_filter.md
|
- Kmers: theory/kmer_indexing/kmers.md
|
||||||
- Minimizer selection: theory/minimizer_selection.md
|
- Minimizer selection: theory/kmer_indexing/minimizer_selection.md
|
||||||
- Partitioning and indexing architecture: theory/indexing_architecture.md
|
- Super-kmers: theory/kmer_indexing/superkmers.md
|
||||||
|
- Partitioning and indexing architecture: theory/kmer_indexing/indexing_architecture.md
|
||||||
|
- Low-complexity kmer filter: theory/kmer_indexing/entropy_filter.md
|
||||||
|
- Phylogeny:
|
||||||
|
- Kmer-based:
|
||||||
|
- Whole k-mer set distances: theory/phylogeny/kmer_based/kmer_set_distances.md
|
||||||
|
- SNP-based:
|
||||||
|
- Central-position SNP model: theory/phylogeny/snp_based/central_snp_model.md
|
||||||
|
- Gamma rate heterogeneity: theory/phylogeny/snp_based/gamma_rate_heterogeneity.md
|
||||||
|
- Sankoff model: theory/phylogeny/snp_based/sankoff_model.md
|
||||||
|
- Sampling design: theory/phylogeny/snp_based/sampling_design.md
|
||||||
- Usage:
|
- Usage:
|
||||||
- superkmer: usage/superkmer.md
|
- superkmer: usage/superkmer.md
|
||||||
- index: usage/index_command.md
|
- index: usage/index_command.md
|
||||||
|
|||||||
@@ -1,528 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Convert the MkDocs user documentation (UserDocMD/) into a Gitea/Forgejo wiki tree.
|
|
||||||
|
|
||||||
Gitea/Forgejo wikis use a different link syntax than MkDocs and cannot render
|
|
||||||
multi-line `$$...$$` display math (a known Goldmark/KaTeX limitation: the
|
|
||||||
parser only recognizes the delimiters when the whole block sits on one
|
|
||||||
physical line). This script rewrites the three constructs that would
|
|
||||||
otherwise break, flattens the tree onto page basenames, and copies everything
|
|
||||||
else verbatim:
|
|
||||||
|
|
||||||
* Markdown links [text](path.md) -> [text](PageName)
|
|
||||||
Relative paths (including MkDocs root-relative `/path.md` links) are
|
|
||||||
resolved against the source tree and then flattened to a wiki page id;
|
|
||||||
only the target is rewritten, the label is left untouched. Gitea/Forgejo's
|
|
||||||
own `[[page|label]]` shortlink syntax is deliberately not used: it is
|
|
||||||
post-processed on rendered HTML text nodes, so it cannot match when the
|
|
||||||
label contains inline formatting (a `` `code` `` span splits the text
|
|
||||||
into separate DOM nodes around it) — confirmed empirically on the target
|
|
||||||
instance, where such labels never became links at all, and even a plain
|
|
||||||
label put target and display text in the position swapped from what the
|
|
||||||
MediaWiki-style `[[target|label]]` convention would suggest. A native
|
|
||||||
Markdown link is a real AST link node, so it has neither problem.
|
|
||||||
External links (http(s)://, mailto:) and links pointing outside the
|
|
||||||
source tree are left untouched.
|
|
||||||
* Pandoc citations [@key; @key] -> formatted, CSL-styled citation
|
|
||||||
Gitea/Forgejo's Goldmark does not support the [@key] citation syntax, and
|
|
||||||
a bare `(@key)` fallback is meaningless to a reader without the .bib file
|
|
||||||
open. Files containing citations are instead piped through `pandoc
|
|
||||||
--citeproc` with the project's own `references.bib` and `ecology-letters
|
|
||||||
.csl` — the same files `mkdocs-bibtex` uses for the real site — so the
|
|
||||||
wiki gets the identical citation style, e.g. `[@Zheng2020-ji]` becomes
|
|
||||||
`(Zheng et al. 2020)`, with a formatted bibliography (one `<div id="ref-
|
|
||||||
KEY">` per entry, real title/journal/DOI) appended to that page. This
|
|
||||||
duplicates entries across pages that cite the same source rather than
|
|
||||||
centralizing them on one wiki page — an acceptable tradeoff for a
|
|
||||||
handful of references, and far simpler than inventing a cross-page
|
|
||||||
anchor scheme. Files with no `[@key]` at all skip pandoc entirely.
|
|
||||||
* Multi-line display math $$\\n...\\n$$ -> single-line $$...$$
|
|
||||||
Collapsed onto one line (interior newlines replaced by spaces; `\\\\`
|
|
||||||
row separators are preserved) so Gitea/Forgejo's renderer recognizes it.
|
|
||||||
Inline `$...$` and already single-line `$$...$$` need no change and pass
|
|
||||||
through as-is.
|
|
||||||
|
|
||||||
None of these rewrites touch fenced code blocks (```` ``` ```` or `~~~`,
|
|
||||||
including openings longer than three characters, e.g. ```` ```` ```` wrapping
|
|
||||||
a literal triple-backtick example).
|
|
||||||
|
|
||||||
Wiki page ids are flat (no directories), so a source file living in a
|
|
||||||
sub-directory gets its directory path folded into the page id: a file at
|
|
||||||
`theory/encoding.md` becomes the page `theory-encoding`. Files at the root keep
|
|
||||||
their bare stem, except `index.md` which maps to `Home` — the page name
|
|
||||||
Gitea/Forgejo treats as the wiki's landing page. The same mapping is used to
|
|
||||||
resolve internal links, so cross-directory references stay consistent. If
|
|
||||||
folding still produces a collision the script refuses to write rather than
|
|
||||||
silently dropping a page.
|
|
||||||
|
|
||||||
The `nav:` section of the MkDocs config is reproduced as a `_Sidebar.md` so the
|
|
||||||
wiki keeps the same top-level grouping (Home / Installation / Theory / Usage /
|
|
||||||
Formats / Architecture). Nav entries are usually `{label: path}` mappings, but
|
|
||||||
a bare `path` entry (no explicit label) is also honored: the label is taken
|
|
||||||
from the target file's first `# ` heading, falling back to a prettified
|
|
||||||
filename. The sidebar is static and shown on every page, which is the only
|
|
||||||
way Gitea/Forgejo expose a structured menu; the file is hidden from the page
|
|
||||||
list because its name starts with `_`.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
mk_wiki.py OUTPUT [OPTIONS]
|
|
||||||
|
|
||||||
Options:
|
|
||||||
--source DIR Source tree (default: UserDocMD next to this script's parent)
|
|
||||||
--config FILE MkDocs user config (default: <repo>/mkdocs-user.yml)
|
|
||||||
--csl FILE CSL citation style (default: <source>/ecology-letters.csl)
|
|
||||||
--bib FILE BibTeX bibliography (default: <source>/references.bib)
|
|
||||||
--no-sidebar do not generate _Sidebar.md
|
|
||||||
--verbose print every conversion
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import yaml
|
|
||||||
|
|
||||||
|
|
||||||
class ConversionError(Exception):
|
|
||||||
"""Raised when a source file cannot be converted faithfully."""
|
|
||||||
|
|
||||||
|
|
||||||
def _make_lenient_loader() -> type[yaml.SafeLoader]:
|
|
||||||
"""A SafeLoader that ignores unknown YAML tags instead of erroring.
|
|
||||||
|
|
||||||
The MkDocs config uses `!!python/name:pymdownx.superfences.fence_code_format`
|
|
||||||
on a custom-fence entry; PyYAML cannot resolve that tag without an unsafe
|
|
||||||
loader, but the script only needs the `nav:` section, which is plain YAML.
|
|
||||||
Treating unknown tags as their literal scalar value is enough.
|
|
||||||
"""
|
|
||||||
|
|
||||||
class _LenientLoader(yaml.SafeLoader):
|
|
||||||
pass
|
|
||||||
|
|
||||||
def _unknown_constructor(loader: yaml.Loader, node: yaml.Node): # type: ignore[no-untyped-def]
|
|
||||||
if isinstance(node, yaml.ScalarNode):
|
|
||||||
return loader.construct_scalar(node)
|
|
||||||
if isinstance(node, yaml.SequenceNode):
|
|
||||||
return loader.construct_sequence(node)
|
|
||||||
if isinstance(node, yaml.MappingNode):
|
|
||||||
return loader.construct_mapping(node)
|
|
||||||
return None
|
|
||||||
|
|
||||||
_LenientLoader.add_constructor(None, _unknown_constructor)
|
|
||||||
return _LenientLoader
|
|
||||||
|
|
||||||
|
|
||||||
def find_repo_root(start: Path) -> Path:
|
|
||||||
"""Walk up until we find a directory containing UserDocMD/."""
|
|
||||||
for candidate in [start.resolve(), *start.resolve().parents]:
|
|
||||||
if (candidate / "UserDocMD").is_dir():
|
|
||||||
return candidate
|
|
||||||
return start.resolve()
|
|
||||||
|
|
||||||
|
|
||||||
def is_external(target: str) -> bool:
|
|
||||||
return bool(re.match(r"^[a-zA-Z][a-zA-Z0-9+.-]*://", target)) or target.startswith("mailto:")
|
|
||||||
|
|
||||||
|
|
||||||
def split_anchor(target: str) -> tuple[str, str]:
|
|
||||||
if "#" in target:
|
|
||||||
path, anchor = target.split("#", 1)
|
|
||||||
return path, "#" + anchor
|
|
||||||
return target, ""
|
|
||||||
|
|
||||||
|
|
||||||
def wiki_page_id(src_root: Path, abs_target: Path) -> str:
|
|
||||||
"""Map a source file to its wiki page id.
|
|
||||||
|
|
||||||
Wikis are flat, so the directory path is folded into the id: a file at
|
|
||||||
`theory/encoding.md` becomes `theory-encoding`, while a root file keeps its
|
|
||||||
bare stem, except `index.md` which maps to `Home` — the page name
|
|
||||||
Gitea/Forgejo renders as the wiki's landing page.
|
|
||||||
"""
|
|
||||||
rel = abs_target.relative_to(src_root.resolve())
|
|
||||||
parts = list(rel.parts)
|
|
||||||
if len(parts) > 1:
|
|
||||||
# Drop the .md suffix from the last part, join the rest with '-'.
|
|
||||||
return "-".join([*parts[:-1], parts[-1][:-3]])
|
|
||||||
stem = parts[0][:-3]
|
|
||||||
return "Home" if stem == "index" else stem
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_wiki_path(src_root: Path, src_file: Path, target: str) -> str | None:
|
|
||||||
"""Turn a relative markdown target into a wiki page id.
|
|
||||||
|
|
||||||
Returns None for targets we should leave as plain Markdown links
|
|
||||||
(external URLs, non-markdown files, missing files, or targets outside the
|
|
||||||
source tree).
|
|
||||||
"""
|
|
||||||
path, anchor = split_anchor(target)
|
|
||||||
if is_external(path) or path == "":
|
|
||||||
return None
|
|
||||||
if path.startswith("/"):
|
|
||||||
# MkDocs root-relative link: resolved against the docs root, not the
|
|
||||||
# current file's directory (Path("a") / "/b" would silently discard
|
|
||||||
# "a" and resolve "/b" from the filesystem root, which is wrong here).
|
|
||||||
abs_target = (src_root / path.lstrip("/")).resolve()
|
|
||||||
else:
|
|
||||||
abs_target = (src_file.parent / path).resolve()
|
|
||||||
try:
|
|
||||||
abs_target.relative_to(src_root.resolve())
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
if abs_target.suffix != ".md":
|
|
||||||
return None
|
|
||||||
if not abs_target.is_file():
|
|
||||||
return None
|
|
||||||
return wiki_page_id(src_root, abs_target) + anchor
|
|
||||||
|
|
||||||
|
|
||||||
LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)\s]+)\)")
|
|
||||||
FENCE_RE = re.compile(r"^(`{3,}|~{3,})")
|
|
||||||
# Pandoc citation syntax, e.g. [@Zheng2020-ji; @Golan2025-xf] — just used to
|
|
||||||
# detect whether a file has any citation worth running through pandoc at all.
|
|
||||||
HAS_CITATION_RE = re.compile(r"\[@[^\]]+\]")
|
|
||||||
|
|
||||||
# pandoc's markdown writer must be told, relative to its own defaults, to:
|
|
||||||
# -smart leave dashes/quotes as written (no em-dash -> "---")
|
|
||||||
# -fenced_divs emit the bibliography as raw HTML <div> (Goldmark
|
|
||||||
# has no ``:::`` fenced-div extension)
|
|
||||||
# -simple_tables, -multiline_tables, -grid_tables, +pipe_tables
|
|
||||||
# keep GFM pipe-table syntax, the only kind Goldmark
|
|
||||||
# parses, instead of pandoc's space-aligned tables
|
|
||||||
# +fenced_code_blocks (pandoc still emits an indented block for a fence
|
|
||||||
# with no language tag regardless of this extension —
|
|
||||||
# such fences need a language hint in the source)
|
|
||||||
# -header_attributes Goldmark has no `{#id .class}` heading-attribute
|
|
||||||
# syntax; without this the "Bibliography" heading pandoc
|
|
||||||
# adds (see BIBLIOGRAPHY_TITLE below) would render with
|
|
||||||
# a literal `{#bibliography .unnumbered}` after it
|
|
||||||
PANDOC_MARKDOWN_FORMAT = (
|
|
||||||
"markdown-smart-fenced_divs-simple_tables-multiline_tables-grid_tables"
|
|
||||||
"-header_attributes+pipe_tables+fenced_code_blocks"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Title pandoc gives the generated bibliography section (always emitted as a
|
|
||||||
# level-1 `# ` heading, with no option to pick the level) — demoted to `##`
|
|
||||||
# below to match the rest of this doc set's heading hierarchy.
|
|
||||||
BIBLIOGRAPHY_TITLE = "Bibliography"
|
|
||||||
|
|
||||||
|
|
||||||
class FenceState:
|
|
||||||
"""Tracks whether the current line is inside a fenced code block.
|
|
||||||
|
|
||||||
A closing fence must use the same character and be at least as long as
|
|
||||||
the opening one (CommonMark rule) — this is what lets a source file wrap
|
|
||||||
a literal ``` example inside a longer ```` fence.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self.marker = ""
|
|
||||||
|
|
||||||
def toggle(self, line: str) -> bool:
|
|
||||||
"""Update state for a fence delimiter line; return True if it is one."""
|
|
||||||
m = FENCE_RE.match(line.rstrip("\n"))
|
|
||||||
if not m:
|
|
||||||
return False
|
|
||||||
run = m.group(1)
|
|
||||||
if not self.marker:
|
|
||||||
self.marker = run
|
|
||||||
elif run[0] == self.marker[0] and len(run) >= len(self.marker):
|
|
||||||
self.marker = ""
|
|
||||||
return True
|
|
||||||
|
|
||||||
@property
|
|
||||||
def active(self) -> bool:
|
|
||||||
return bool(self.marker)
|
|
||||||
|
|
||||||
|
|
||||||
def convert_links(text: str, src_root: Path, src_file: Path, verbose: bool) -> str:
|
|
||||||
out_lines: list[str] = []
|
|
||||||
fence = FenceState()
|
|
||||||
|
|
||||||
for line in text.splitlines(keepends=True):
|
|
||||||
if fence.toggle(line) or fence.active:
|
|
||||||
out_lines.append(line)
|
|
||||||
continue
|
|
||||||
|
|
||||||
def repl(m: re.Match) -> str:
|
|
||||||
label, target = m.group(1), m.group(2)
|
|
||||||
wiki = resolve_wiki_path(src_root, src_file, target)
|
|
||||||
if wiki is None:
|
|
||||||
return m.group(0) # external or non-md: leave as Markdown
|
|
||||||
if verbose:
|
|
||||||
print(f" {src_file.relative_to(src_root)}: [{label}]({target}) -> [{label}]({wiki})")
|
|
||||||
return f"[{label}]({wiki})"
|
|
||||||
|
|
||||||
out_lines.append(LINK_RE.sub(repl, line))
|
|
||||||
|
|
||||||
return "".join(out_lines)
|
|
||||||
|
|
||||||
|
|
||||||
def convert_citations(
|
|
||||||
text: str, src_root: Path, src_file: Path, csl_path: Path, bib_path: Path, verbose: bool
|
|
||||||
) -> str:
|
|
||||||
"""Render Pandoc `[@key; @key]` citations with the project's real CSL style.
|
|
||||||
|
|
||||||
Skips files with no citation entirely, so a missing `--csl`/`--bib` or a
|
|
||||||
missing `pandoc` binary is only an error for files that actually need it.
|
|
||||||
"""
|
|
||||||
if not HAS_CITATION_RE.search(text):
|
|
||||||
return text
|
|
||||||
|
|
||||||
if not csl_path.is_file():
|
|
||||||
raise ConversionError(f"{src_file.relative_to(src_root)}: CSL style not found: {csl_path}")
|
|
||||||
if not bib_path.is_file():
|
|
||||||
raise ConversionError(f"{src_file.relative_to(src_root)}: bibliography not found: {bib_path}")
|
|
||||||
|
|
||||||
try:
|
|
||||||
result = subprocess.run(
|
|
||||||
[
|
|
||||||
"pandoc",
|
|
||||||
"--citeproc",
|
|
||||||
f"--csl={csl_path}",
|
|
||||||
f"--bibliography={bib_path}",
|
|
||||||
"--wrap=none",
|
|
||||||
f"--metadata=link-citations=true",
|
|
||||||
f"--metadata=reference-section-title={BIBLIOGRAPHY_TITLE}",
|
|
||||||
"-f",
|
|
||||||
"markdown-smart",
|
|
||||||
"-t",
|
|
||||||
PANDOC_MARKDOWN_FORMAT,
|
|
||||||
],
|
|
||||||
input=text,
|
|
||||||
capture_output=True,
|
|
||||||
text=True,
|
|
||||||
)
|
|
||||||
except FileNotFoundError:
|
|
||||||
raise ConversionError(
|
|
||||||
f"{src_file.relative_to(src_root)}: 'pandoc' not found on PATH, needed to render citations"
|
|
||||||
) from None
|
|
||||||
|
|
||||||
if result.returncode != 0:
|
|
||||||
raise ConversionError(
|
|
||||||
f"{src_file.relative_to(src_root)}: pandoc failed to render citations:\n{result.stderr}"
|
|
||||||
)
|
|
||||||
|
|
||||||
# pandoc always emits the bibliography title as a level-1 heading; demote
|
|
||||||
# it to `##` to match the rest of this doc set's hierarchy.
|
|
||||||
out = result.stdout.replace(f"\n# {BIBLIOGRAPHY_TITLE}\n", f"\n## {BIBLIOGRAPHY_TITLE}\n")
|
|
||||||
|
|
||||||
if verbose:
|
|
||||||
n = len(HAS_CITATION_RE.findall(text))
|
|
||||||
print(f" {src_file.relative_to(src_root)}: rendered {n} citation group(s) via pandoc/{csl_path.name}")
|
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def convert_math(text: str, src_root: Path, src_file: Path, verbose: bool) -> str:
|
|
||||||
"""Collapse multi-line `$$...$$` display blocks onto a single line.
|
|
||||||
|
|
||||||
Gitea/Forgejo's math renderer only recognizes `$$...$$` when it sits on
|
|
||||||
one physical line; a block opened and closed on their own bare `$$` line
|
|
||||||
(the natural Pandoc/MkDocs style) renders as literal text otherwise.
|
|
||||||
Interior line breaks are replaced by spaces (insignificant in math mode);
|
|
||||||
`\\\\` row separators inside `aligned`/`cases` environments are untouched.
|
|
||||||
Already single-line `$$...$$` and inline `$...$` need no change.
|
|
||||||
"""
|
|
||||||
lines = text.split("\n")
|
|
||||||
out: list[str] = []
|
|
||||||
fence = FenceState()
|
|
||||||
i = 0
|
|
||||||
n = len(lines)
|
|
||||||
while i < n:
|
|
||||||
line = lines[i]
|
|
||||||
if fence.toggle(line) or fence.active:
|
|
||||||
out.append(line)
|
|
||||||
i += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
if line.strip() == "$$":
|
|
||||||
body: list[str] = []
|
|
||||||
i += 1
|
|
||||||
closed = False
|
|
||||||
while i < n:
|
|
||||||
if lines[i].strip() == "$$":
|
|
||||||
closed = True
|
|
||||||
i += 1
|
|
||||||
break
|
|
||||||
body.append(lines[i].strip())
|
|
||||||
i += 1
|
|
||||||
if not closed:
|
|
||||||
raise ConversionError(
|
|
||||||
f"{src_file.relative_to(src_root)}: unterminated $$ display math block"
|
|
||||||
)
|
|
||||||
collapsed = "$$" + " ".join(part for part in body if part) + "$$"
|
|
||||||
if verbose:
|
|
||||||
print(f" {src_file.relative_to(src_root)}: collapsed {len(body)}-line $$ block")
|
|
||||||
out.append(collapsed)
|
|
||||||
continue
|
|
||||||
|
|
||||||
out.append(line)
|
|
||||||
i += 1
|
|
||||||
|
|
||||||
return "\n".join(out)
|
|
||||||
|
|
||||||
|
|
||||||
def convert_file(
|
|
||||||
src_root: Path, src_file: Path, csl_path: Path, bib_path: Path, verbose: bool
|
|
||||||
) -> str:
|
|
||||||
text = src_file.read_text(encoding="utf-8")
|
|
||||||
# Citations first, before pandoc has seen any of our own wiki-specific
|
|
||||||
# rewrites: it needs to parse plain, unmodified Markdown.
|
|
||||||
text = convert_citations(text, src_root, src_file, csl_path, bib_path, verbose)
|
|
||||||
text = convert_math(text, src_root, src_file, verbose)
|
|
||||||
text = convert_links(text, src_root, src_file, verbose)
|
|
||||||
return text
|
|
||||||
|
|
||||||
|
|
||||||
def _default_label(src_root: Path, rel_path: str) -> str:
|
|
||||||
"""Derive a nav label for a bare `path.md` entry (no explicit label).
|
|
||||||
|
|
||||||
Mirrors MkDocs' own fallback: use the page's first `# ` heading if it has
|
|
||||||
one, otherwise prettify the filename.
|
|
||||||
"""
|
|
||||||
target = src_root / rel_path
|
|
||||||
if target.is_file():
|
|
||||||
for line in target.read_text(encoding="utf-8").splitlines():
|
|
||||||
if line.startswith("# "):
|
|
||||||
return line[2:].strip()
|
|
||||||
return Path(rel_path).stem.replace("_", " ").replace("-", " ").title()
|
|
||||||
|
|
||||||
|
|
||||||
def build_sidebar(src_root: Path, config_path: Path) -> str | None:
|
|
||||||
"""Render the MkDocs `nav:` section as a Gitea/Forgejo `_Sidebar.md`.
|
|
||||||
|
|
||||||
Returns None if the config is missing or has no usable `nav:` section.
|
|
||||||
Each nav entry is either a bare `path` string or a `{label: path}` pair
|
|
||||||
(or a nested list for a group); the path is mapped through `wiki_page_id`
|
|
||||||
so the sidebar links to the flattened page ids.
|
|
||||||
"""
|
|
||||||
if not config_path.is_file():
|
|
||||||
print(f"warning: config not found, skipping sidebar: {config_path}")
|
|
||||||
return None
|
|
||||||
|
|
||||||
try:
|
|
||||||
with config_path.open(encoding="utf-8") as fh:
|
|
||||||
cfg = yaml.safe_load(fh)
|
|
||||||
except yaml.YAMLError:
|
|
||||||
# The MkDocs config carries `!!python/name:...` tags that SafeLoader
|
|
||||||
# cannot resolve; fall back to a lenient loader that treats unknown
|
|
||||||
# tags as plain scalars so the `nav:` section is still readable.
|
|
||||||
try:
|
|
||||||
with config_path.open(encoding="utf-8") as fh:
|
|
||||||
cfg = yaml.load(fh, Loader=_make_lenient_loader())
|
|
||||||
except yaml.YAMLError as exc2:
|
|
||||||
print(f"warning: could not parse {config_path}: {exc2}")
|
|
||||||
return None
|
|
||||||
|
|
||||||
nav = cfg.get("nav") if isinstance(cfg, dict) else None
|
|
||||||
if not nav:
|
|
||||||
print(f"warning: no `nav:` section in {config_path}, skipping sidebar")
|
|
||||||
return None
|
|
||||||
|
|
||||||
def link_line(depth: int, rel_path: str, label: str) -> str:
|
|
||||||
page = wiki_page_id(src_root, (src_root / rel_path).resolve())
|
|
||||||
return f"{' ' * depth}- [{label}]({page})"
|
|
||||||
|
|
||||||
def render_entry(entry: object, depth: int) -> list[str]:
|
|
||||||
if isinstance(entry, str):
|
|
||||||
return [link_line(depth, entry, _default_label(src_root, entry))]
|
|
||||||
if not isinstance(entry, dict):
|
|
||||||
return []
|
|
||||||
lines: list[str] = []
|
|
||||||
for label, target in entry.items():
|
|
||||||
if isinstance(target, str):
|
|
||||||
lines.append(link_line(depth, target, label))
|
|
||||||
elif isinstance(target, list):
|
|
||||||
# Headings must never be indented: CommonMark treats a line
|
|
||||||
# indented by 4+ spaces as a code block, not an ATX heading,
|
|
||||||
# which a depth-scaled indent would trigger from depth 2 on.
|
|
||||||
lines.append(f"## {label}")
|
|
||||||
lines.extend(render_subtree(target, depth + 1))
|
|
||||||
return lines
|
|
||||||
|
|
||||||
def render_subtree(nodes: list[object], depth: int) -> list[str]:
|
|
||||||
out: list[str] = []
|
|
||||||
for node in nodes:
|
|
||||||
out.extend(render_entry(node, depth))
|
|
||||||
return out
|
|
||||||
|
|
||||||
body = render_subtree(nav, 0)
|
|
||||||
if not body:
|
|
||||||
return None
|
|
||||||
|
|
||||||
return "# Wiki sidebar\n\n" + "\n".join(body) + "\n"
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
||||||
ap.add_argument("output", help="destination wiki directory (created if missing)")
|
|
||||||
ap.add_argument("--source", default=None, help="source Markdown tree (default: <repo>/UserDocMD)")
|
|
||||||
ap.add_argument("--config", default=None,
|
|
||||||
help="MkDocs user config (default: <repo>/mkdocs-user.yml)")
|
|
||||||
ap.add_argument("--csl", default=None,
|
|
||||||
help="CSL citation style (default: <source>/ecology-letters.csl)")
|
|
||||||
ap.add_argument("--bib", default=None,
|
|
||||||
help="BibTeX bibliography (default: <source>/references.bib)")
|
|
||||||
ap.add_argument("--no-sidebar", action="store_true", help="do not generate _Sidebar.md")
|
|
||||||
ap.add_argument("--verbose", action="store_true", help="print every conversion")
|
|
||||||
args = ap.parse_args()
|
|
||||||
|
|
||||||
script_dir = Path(__file__).resolve().parent
|
|
||||||
repo_root = find_repo_root(script_dir)
|
|
||||||
src_root = Path(args.source).resolve() if args.source else (repo_root / "UserDocMD")
|
|
||||||
out_root = Path(args.output).resolve()
|
|
||||||
config_path = Path(args.config).resolve() if args.config else (repo_root / "mkdocs-user.yml")
|
|
||||||
csl_path = Path(args.csl).resolve() if args.csl else (src_root / "ecology-letters.csl")
|
|
||||||
bib_path = Path(args.bib).resolve() if args.bib else (src_root / "references.bib")
|
|
||||||
|
|
||||||
if not src_root.is_dir():
|
|
||||||
print(f"error: source tree not found: {src_root}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
|
|
||||||
md_files = sorted(p for p in src_root.rglob("*.md") if p.is_file())
|
|
||||||
if not md_files:
|
|
||||||
print(f"error: no Markdown files under {src_root}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
|
|
||||||
# A wiki is flat, so the destination filename is the source path folded into
|
|
||||||
# a page id. Detect collisions up front so a silent overwrite does not drop
|
|
||||||
# a page.
|
|
||||||
by_name: dict[str, list[Path]] = {}
|
|
||||||
for src in md_files:
|
|
||||||
by_name.setdefault(wiki_page_id(src_root, src), []).append(src)
|
|
||||||
collisions = {name: paths for name, paths in by_name.items() if len(paths) > 1}
|
|
||||||
if collisions:
|
|
||||||
for name, paths in sorted(collisions.items()):
|
|
||||||
print(f"error: wiki page id collision '{name}':", file=sys.stderr)
|
|
||||||
for p in paths:
|
|
||||||
print(f" {p.relative_to(src_root)}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
|
|
||||||
out_root.mkdir(parents=True, exist_ok=True)
|
|
||||||
written = 0
|
|
||||||
for src in md_files:
|
|
||||||
dst = out_root / (wiki_page_id(src_root, src) + ".md")
|
|
||||||
try:
|
|
||||||
content = convert_file(src_root, src, csl_path, bib_path, args.verbose)
|
|
||||||
except ConversionError as exc:
|
|
||||||
print(f"error: {exc}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
dst.write_text(content, encoding="utf-8")
|
|
||||||
written += 1
|
|
||||||
|
|
||||||
if not args.no_sidebar:
|
|
||||||
sidebar = build_sidebar(src_root, config_path)
|
|
||||||
if sidebar is not None:
|
|
||||||
(out_root / "_Sidebar.md").write_text(sidebar, encoding="utf-8")
|
|
||||||
written += 1
|
|
||||||
|
|
||||||
print(f"converted {written} Markdown file(s) from {src_root} -> {out_root}")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
Reference in new issue
Block a user