docs: restructure documentation for kmer indexing and phylogeny theory

Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
This commit is contained in:
Eric Coissac committed 2026-11-19 17:47:59 +01:00
1 parent d78a6b4501
commit 33d7a59dd9
43 files changed
+25349 -1091

No files matched your search

+1 -1
View File
@@ -14,7 +14,7 @@ Nothing here comes from a reference genome. The file has no real coordinates, an
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
``` ```
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, together with an already-built sibling annex (see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites. `--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, (the sibling annex it reads is built automatically if missing; see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
## Layout ## Layout
+232
View File
@@ -857,3 +857,235 @@
doi = "10.1214/aop/1176996461", doi = "10.1214/aop/1176996461",
issn = "0091-1798,2168-894X" issn = "0091-1798,2168-894X"
} }
@ARTICLE{Karlin1997-lm,
title = "{Compositional differences within and between eukaryotic genomes}",
author = "Karlin, S and Mrázek, J",
journal = "Proceedings of the National Academy of Sciences of the United
States of America",
publisher = "Proceedings of the National Academy of Sciences",
volume = 94,
number = 19,
pages = "10227--10232",
abstract = "Eukaryotic genome similarity relationships are inferred using
sequence information derived from large aggregates of genomic
sequences. Comparisons within and between species sample
sequences are based on the profile of dinucleotide relative
abundance values (The profile is ρ* XY = f * XY / f * X f * Y for
all XY , where f * X denotes the frequency of the nucleotide X
and f * XY denotes the frequency of the dinucleotide XY , both
computed from the sequence concatenated with its inverted
complement). Previous studies with respect to prokaryotes and
this study document that profiles of different DNA sequence
samples (sample size ≥50 kb) from the same organism are generally
much more similar to each other than they are to profiles from
other organisms, and that closely related organisms generally
have more similar profiles than do distantly related organisms.
On this basis we refer to the collection {ρ* XY } as the genome
signature . This paper identifies ρ* XY extremes and compares
genome signature differences for a diverse range of eukaryotic
species. Interpretations on the mechanisms maintaining these
profile differences center on genome-wide replication, repair,
DNA structures, and context-dependent mutational biases. It is
also observed that mitochondrial genome signature differences
between species parallel the corresponding nuclear genome
signature differences despite large differences between
corresponding mitochondrial and nuclear signatures. The genome
signature differences also have implications for contrasts
between rodents and other mammals, and between monocot and dicot
plants, as well as providing evidence for similarities among
fungi and the diversity of protists.",
month = sep,
year = 1997,
url = "http://dx.doi.org/10.1073/pnas.94.19.10227",
doi = "10.1073/pnas.94.19.10227",
pmc = "PMC23344",
pmid = 9294192,
issn = "0027-8424,1091-6490",
language = "en"
}
@ARTICLE{Karlin1997-vm,
title = "{Compositional biases of bacterial genomes and evolutionary
implications}",
author = "Karlin, S and Mrázek, J and Campbell, A M",
journal = "Journal of Bacteriology",
publisher = "American Society for Microbiology",
volume = 179,
number = 12,
pages = "3899--3913",
abstract = "We compare and contrast genome-wide compositional biases and
distributions of short oligonucleotides across 15 diverse
prokaryotes that have substantial genomic sequence collections.
These include seven complete genomes (Escherichia coli,
Haemophilus influenzae, Mycoplasma genitalium, Mycoplasma
pneumoniae, Synechocystis sp. strain PCC6803, Methanococcus
jannaschii, and Pyrobaculum aerophilum). A key observation
concerns the constancy of the dinucleotide relative abundance
profiles over multiple 50-kb disjoint contigs within the same
genome. (The profile is rhoXY* = fXY*/fX*fY* for all XY, where
fX* denotes the frequency of the nucleotide X and fY* denotes the
frequency of the dinucleotide XY, both computed from the sequence
concatenated with its inverted complementary sequence.) On the
basis of this constancy, we refer to the collection [rhoXY*] as
the genome signature. We establish that the differences between
[rhoXY*] vectors of 50-kb sample contigs of different genomes
virtually always exceed the differences between those of the same
genomes. Various di- and tetranucleotide biases are identified.
In particular, we find that the dinucleotide CpG=CG is
underrepresented in many thermophiles (e.g., M. jannaschii,
Sulfolobus sp., and M. thermoautotrophicum) but overrepresented
in halobacteria. TA is broadly underrepresented in prokaryotes
and eukaryotes, but normal counts appear in Sulfolobus and P.
aerophilum sequences. More than for any other bacterial genome,
palindromic tetranucleotides are underrepresented in H.
influenzae. The M. jannaschii sequence is unprecedented in its
extreme underrepresentation of CTAG tetranucleotides and in the
anomalous distribution of CTAG sites around the genome.
Comparative analysis of numbers of long tetranucleotide
microsatellites distinguishes H. influenzae. Dinucleotide
relative abundance differences between bacterial sequences are
compared. For example, in these assessments of differences, the
cyanobacteria Synechocystis, Synechococcus, and Anabaena do not
form a coherent group and are as far from each other as general
gram-negative sequences are from general gram-positive sequences.
The difference of M. jannaschii from low-G+C gram-positive
proteobacteria is one-half of the difference from gram-negative
proteobacteria. Interpretations and hypotheses center on the role
of the genome signature in highlighting similarities and
dissimilarities across different classes of prokaryotic species,
possible mechanisms underlying the genome signature, the form and
level of genome compositional flux, the use of the genome
signature as a chronometer of molecular phylogeny, and
implications with respect to the three putative eubacterial,
archaeal, and eukaryote domains of life and to the origin and
early evolution of eukaryotes.",
month = jun,
year = 1997,
url = "http://dx.doi.org/10.1128/jb.179.12.3899-3913.1997",
doi = "10.1128/jb.179.12.3899-3913.1997",
pmc = "PMC179198",
pmid = 9190805,
issn = "0021-9193,1098-5530",
language = "en"
}
@ARTICLE{Karlin1995-lt,
title = "{Dinucleotide relative abundance extremes: a genomic signature}",
author = "Karlin, S and Burge, C",
journal = "Trends in Genetics: TIG",
publisher = "Elsevier BV",
volume = 11,
number = 7,
pages = "283--290",
abstract = "Early biochemical experiments established that the set of
dinucleotide odds ratios or 'general design' is a remarkably
stable property of the DNA of an organism, which is essentially
the same in protein-coding DNA, bulk genomic DNA, and in
different renaturation rate and density gradient fractions of
genomic DNA in many organisms. Analysis of currently available
genomic sequence data has extended these earlier results, showing
that the general designs of disjoint samples of a genome are
substantially more similar to each other than to those of
sequences from other organisms and that closely related organisms
have similar general designs. From this perspective, the set of
dinucleotide odds ratio (relative abundance) values constitute a
signature of each DNA genome, which can discriminate between
sequences from different organisms. Dinucleotide-odds ratio
values appear to reflect not only the chemistry of dinucleotide
stacking energies and base-step conformational preferences, but
also the species-specific properties of DNA modification,
replication and repair mechanisms.",
month = jul,
year = 1995,
url = "http://dx.doi.org/10.1016/s0168-9525(00)89076-9",
doi = "10.1016/s0168-9525(00)89076-9",
pmid = 7482779,
issn = "0168-9525,1362-4555",
language = "en"
}
@ARTICLE{Karlin1994-on,
title = "{Heterogeneity of genomes: measures and values}",
author = "Karlin, S and Ladunga, I and Blaisdell, B E",
journal = "Proceedings of the National Academy of Sciences of the United
States of America",
publisher = "Proceedings of the National Academy of Sciences",
volume = 91,
number = 26,
pages = "12837--12841",
abstract = "Genomic homogeneity is investigated for a broad base of DNA
sequences in terms of dinucleotide relative abundance distances
(abbreviated delta-distances) and of oligonucleotide
compositional extremes. It is shown that delta-distances between
different genomic sequences in the same species are low, only
about 2 or 3 times the distance found in random DNA, and are
generally smaller than the between-species delta-distances.
Extremes in short oligonucleotides include underrepresentation of
TpA and overrepresentation of GpC in most temperate bacteriophage
sequences; underrepresentation of CTAG in most eubacterial
genomes; underrepresentation of GATC in most bacteriophage; CpG
suppression in vertebrates, in all animal mitochondrial genomes,
and in many thermophilic bacterial sequences; and
overrepresentation of GpG/CpC in all animal mitochondrial sets
and chloroplast genomes. Interpretations center on DNA structures
(dinucleotide stacking energies, DNA curvature and superhelicity,
nucleosome organization), context-dependent mutational events,
methylation effects, and processes of replication and repair.",
month = dec,
year = 1994,
url = "http://dx.doi.org/10.1073/pnas.91.26.12837",
doi = "10.1073/pnas.91.26.12837",
pmc = "PMC45535",
pmid = 7809131,
issn = "0027-8424,1091-6490",
language = "en"
}
@ARTICLE{Bennett2013-cl,
title = "{Small, smaller, smallest: the origins and evolution of ancient
dual symbioses in a Phloem-feeding insect}",
author = "Bennett, Gordon M and Moran, Nancy A",
journal = "Genome Biology and Evolution",
publisher = "Oxford University Press (OUP)",
volume = 5,
number = 9,
pages = "1675--1688",
abstract = "Many insects rely on bacterial symbionts with tiny genomes
specialized for provisioning nutrients lacking in host diets.
Xylem sap and phloem sap are both deficient as insect diets, but
differ dramatically in nutrient content, potentially affecting
symbiont genome evolution. For sap-feeding insects, sequenced
symbiont genomes are available only for phloem-feeding examples
from the suborder Sternorrhyncha and xylem-feeding examples from
the suborder Auchenorrhyncha, confounding comparisons. We
sequenced genomes of the obligate symbionts, Sulcia muelleri and
Nasuia deltocephalinicola, of the phloem-feeding pest insect,
Macrosteles quadrilineatus (Auchenorrhyncha: Cicadellidae). Our
results reveal that Nasuia-ALF has the smallest bacterial genome
yet sequenced (112 kb), and that the Sulcia-ALF genome (190 kb)
is smaller than that of Sulcia in other insect lineages.
Together, these symbionts retain the capability to synthesize the
10 essential amino acids, as observed for several symbiont pairs
from xylem-feeding Auchenorrhyncha. Nasuia retains genes enabling
synthesis of two amino acids, DNA replication, transcription, and
translation. Both symbionts have lost genes underlying ATP
synthesis through oxidative phosphorylation, possibly as a
consequence of the enriched sugar content of phloem. Shared
genomic features, including reassignment of the UGA codon from
Stop to tryptophan, and phylogenetic results suggest that
Nasuia-ALF is most closely related to Zinderia, the
betaproteobacterial symbiont of spittlebugs. Thus,
Nasuia/Zinderia and Sulcia likely represent ancient associates
that have co-resided in hosts since the divergence of leafhoppers
and spittlebugs >200 Ma, and possibly since the origin of the
Auchenorrhyncha, >260 Ma.",
month = aug,
year = 2013,
url = "http://dx.doi.org/10.1093/gbe/evt118",
keywords = "Nasuia deltocephalinicola; Sulcia muelleri; gene loss; genome
evolution; leafhopper; nutritional symbioses",
doi = "10.1093/gbe/evt118",
pmc = "PMC3787670",
pmid = 23918810,
issn = "1759-6653",
language = "en"
}
+67 -26
View File
@@ -47,7 +47,9 @@ P(n_K > 0) &= 1-e^{-\lambda} \\
\end{aligned} \end{aligned}
$$ $$
Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore ##### Assuming equal genome lengths
The number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
$$ $$
@@ -56,11 +58,22 @@ $$
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
$$
I_p(r,S-r+1)
=
\frac{1}{B(r,S-r+1)}
\int_0^p t^{r-1}(1-t)^{S-r}\,dt
$$
and
$$ $$
p = 1-e^{-2G/4^k}. p = 1-e^{-2G/4^k}.
$$ $$
Different genome sizes the regularized incomplete beta function is available in standard statistical libraries.
##### Different genome sizes
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
@@ -82,39 +95,63 @@ Using the mean genome size therefore provides a conservative estimate of the FDR
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$. When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
### Exact tail probability ##### Exact tail probability
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time. With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does: Let $P_i(j)$ denote the probability that the k-mer occurs in exactly $j$ of the first $i$ genomes. Initially,
$$
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
$$
When genome $i$ is added, the k-mer is either absent, with probability $1-p_i$, or present, with probability $p_i$. The probability of observing exactly $j$ presences among the first $i$ genomes is therefore
$$
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
$$
This dynamic programming algorithm requires $O(S^2)$ operations and $O(S)$ memory. However, to compute $P(X\geq r)$, only the probabilities for fewer than $r$ presences need to be tracked, since
$$
P(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
$$
This reduces the computational cost to $O(Sr)$ operations and $O(r)$ memory.
#### Distinguishing random from non-random k-mer sharing
The probability $P(X\geq r)$ describes the occurrence of a given k-mer in at least $r$ genomes under the random-sequence model. However, the index contains only k-mers observed in at least one genome. Therefore, when considering the k-mers recorded in the index, the relevant probability is the conditional probability:
$$
q_r = P(X\geq r\mid X\geq 1)
= \frac{P(X\geq r)}{P(X\geq 1)}.
$$
Let $k_r$ be the number of distinct k-mers observed in at least $r$ genomes, among the $k_1$ distinct k-mers recorded in the index. The observed proportion of k-mers shared by at least $r$ genomes is:
$$
O_r = \frac{k_r}{k_1}.
$$
Under the random-sequence model, the expected proportion is $q_r$. We can therefore quantify the deviation from random expectations using the difference:
$$ $$
\left \{
\begin{aligned} \begin{aligned}
d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\ D_r &= O_r-q_r \\
d_0&=1 &= \frac{k_r}{k_1}-q_r.
\end{aligned} \end{aligned}
\right .
$$ $$
before processing the first genome. Positive values of $D_r$ indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.
The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$ is bounded:
$$ $$
P(X\geq r)=1-\sum_{j=0}^{r-1}d_j. -1\leq D_r\leq 1.
$$ $$
Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$. More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By construction, $D_1=0$, since all k-mers recorded in the index occur in at least one genome. Unlike a ratio-based measure, this difference remains defined even when no k-mers are observed in at least $r$ genomes. It thus provides a bounded measure of the excess or deficit of k-mer sharing relative to the random-sequence model.
When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean
$$
\lambda=\sum_{i=1}^{S}p_i.
$$
The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe].
******* *******
@@ -150,11 +187,12 @@ Consequently, the appropriate value of $k$ depends on the total amount of sequen
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Mycoplasma genitalium* having a size $G=580\,Mb$ [@Fraser1995-jc], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate: Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Nasuia deltocephalinicola* having a size $G=112\,Kb$ [@Bennett2013-cl], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
$$ $$
\begin{aligned} \begin{aligned}
P(C\geq1) &= 1- e^{-\frac{580.000^2}{2 \cdot 4^{11}}} \\ P(C\geq1) &= 1- e^{-\frac{112\,000^2}{2 \cdot 4^{11}}} \\
&= 1- e^{-1.5\times10^{3}} \\
&\simeq 1 &\simeq 1
\end{aligned} \end{aligned}
$$ $$
@@ -182,7 +220,7 @@ The distinction between the simple $4^k$ model and the Karlin--Ost framework is
These scales have very different implications for different genome sizes. These scales have very different implications for different genome sizes.
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995; @Karlin1997]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers. The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995-lt; @Karlin1994-on]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity. This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.
@@ -232,7 +270,7 @@ $$
The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985]. The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985].
<!--
$o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ». $o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ».
Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ : Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ :
@@ -265,7 +303,7 @@ Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nomb
- **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant. - **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant.
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$. Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
-->
## Chance sharing across genomes and the FDR of homology ## Chance sharing across genomes and the FDR of homology
@@ -359,6 +397,7 @@ Chance alone produces millions of pairwise shared 21-mers at this genome size, a
--- ---
<!-- Notes perso (bac à sable), non publiées dans le HTML
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`. Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
## 1. FDR d'un ensemble de k-mers sélectionnés ## 1. FDR d'un ensemble de k-mers sélectionnés
@@ -414,3 +453,5 @@ Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Ell
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total. **Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ? Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
-->
@@ -34,7 +34,7 @@ $$
D = \sum_i \mathbb{1}[a_i \ne b_i], D = \sum_i \mathbb{1}[a_i \ne b_i],
$$ $$
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded. where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded. On an index that stores counts, a k-mer is taken as present when its count reaches the presence threshold (`--presence-threshold`, 1 by default).
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If **Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
@@ -24,4 +24,4 @@ If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction. **This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation. See [`phylo`](../../../usage/phylo.md#distance-estimation-and-tree-reconstruction) for how to select a fixed $\alpha$ or request automatic estimation.
@@ -31,4 +31,4 @@ How well a pair of genomes is represented in a sample depends on how many famili
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy. By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior. See [`phylo`](../../../usage/phylo.md#snp-character-selection-and-sampling) for how to bound and bias the sample.
+112 -91
View File
@@ -1,6 +1,13 @@
# phylo # phylo
Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE). Compare the genomes stored in an index and build phylogenetic trees from them. Two complementary approaches are available:
- **Whole-genome distances** compare the k-mer content of the genomes, without any alignment.
- **SNP-based distances and analyses** compare the nucleotide variants found at the central position of related k-mers.
Distance matrices can be used to build Neighbor-Joining or UPGMA trees. SNP-based analyses can also produce a pseudo-alignment and calibrated substitution costs, with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
## Usage
```bash ```bash
obikmer phylo INDEX [OPTIONS] obikmer phylo INDEX [OPTIONS]
@@ -12,28 +19,43 @@ obikmer phylo INDEX [OPTIONS]
|---|---| |---|---|
| `INDEX` | Index directory | | `INDEX` | Index directory |
## Distance matrix (`--distance`) ## General options
| Option | Default | Description | | Option | Default | Description |
|---|---|---| |---|---|---|
| `--distance` | none | Compute and write a distance matrix. See the two tables below for the full list of accepted values. No matrix is computed unless this is given, or `--nj`, `--upgma` or `--shared-kmers` is (they need a matrix and then use `jaccard`) | | `--cpu-max N` | all available cores | Maximum number of CPU cores to use. It caps every thread pool of the process and can only lower the number of cores, never raise it above those available. Common to all subcommands |
| `-o, --output PREFIX` | none | Prefix of the output files. Without it, the distance matrix goes to standard output and every other file is written in the current directory under its name without the prefix. See "Output files" below |
## SNP families
SNP-based analyses use groups of k-mers that differ only at their central nucleotide. These groups are called families (see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the exact definitions and eligibility rules). A variable family is one character of the analysis, and the variants a genome carries give its state for that character.
## Distance estimation and tree reconstruction
| Option | Default | Description |
|---|---|---|
| `--distance` | none | Compute and write a distance matrix. The accepted values are listed just below the table. No matrix is computed unless this is given, or `--nj` or `--upgma` is (they need a matrix and then use `jaccard`) |
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it | | `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index | | `--presence-threshold` | `1` | How counts are turned into presence/absence on an index that stores counts: a kmer is present in a genome only when its count reaches this value. Used by `jaccard`, `mash` and `hamming`, and by the shared-kmer counts of `--shared-kmers`. No effect on an index without counts, nor on the distances that use the counts themselves |
| `--session DIR`, `--session-force` | none | Save and reuse the sample of a `snp-*` distance. Currently unavailable, see `--session` below |
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format | | `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index distance, not a `snp-*` value | | `--shared-kmers` | off | Write the shared-kmer count matrix (`<prefix>_shared.csv`). It can be used alone, without any distance, or together with a whole-index distance; it is refused with a `snp-*` value |
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) | | `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
| `--upgma` | off | Compute and write a UPGMA tree (Newick) | | `--upgma` | off | Compute and write a UPGMA tree (Newick) |
| `-o, --output` | none (stdout) | Output file prefix |
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions). Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `jaccard`, `mash` and `hamming` compare which kmers are present and work on either index type; the six others (`bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`) use the counts and need an index that stores them), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here). `snp-*` values read the sibling annex (below), which is built automatically the first time it is needed, and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "SNP character selection and sampling" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
### Output ### Matrix format
When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below. When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
## `--exclude-genome`, `--min-shared-family` ### `--session`: currently unavailable
`--session DIR` and `--session-force` are meant to save the sample of a `snp-*` `--distance` run in a directory, so that a later run with identical selection parameters reuses it instead of drawing again. This feature is being rebuilt and is not available for now: giving either option stops the command with an error. Until it returns, keep the output files of a run (the alignment, or the distance matrix) rather than relying on re-drawing the same sample.
## Genome selection and filtering
| Option | Description | | Option | Description |
|---|---| |---|---|
@@ -42,21 +64,79 @@ When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHY
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`. Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
## Central-position SNP model ## SNP character selection and sampling
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index: SNP-based analyses can use every eligible variable family, or a sample drawn from the index. The options below control which characters are selected, which genomes and states are kept, and how the sample is spread over the index.
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags.
Every invocation draws its own fresh random sample: running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately.
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
| Option | Default | Description |
|---|---|---|
| `--n-sites N\|all` | none (mandatory for `--pseudo-alignment` and `--sankoff`) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
### `--quartet`: keeping only families that can resolve a tree
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
### `--pair-balance`: representing every pair of genomes
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
### Describing a draw: `--sample-stats`, `--sample-plan`
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
`--sample-stats` writes three files:
| File | Content |
|---|---|
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. A site is informative under the same rule as `--quartet`: without `--free-loss`, absence is a state of its own and can form one of the two pairs; with `--free-loss` it is missing data and cannot. Excluded genomes are listed with zeros in the first file and count nowhere.
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
## SNP family preparation and diagnostics
SNP-based options read precomputed tables describing the families: the sibling annex, and tables derived from it (entropy, quartet, family overlap, pair weights). They are built automatically the first time they are needed and reused afterwards. The options below prepare them in advance, or describe the families.
| Option | Description | | Option | Description |
|---|---| |---|---|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the `--no-ambiguity` / `--only-diploid` convention given: `--entropy` the entropy annex, `--quartet` the quartet annex, `--pair-balance` the pair-weight and family-overlap annexes, `--family-overlap` or `--min-shared-family` the family-overlap annex. For example `--sibling-annex --quartet --only-diploid` prepares exactly the quartet annex of that convention, and `--sibling-annex` alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time | | `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the `--no-ambiguity` / `--only-diploid` convention given: `--entropy` the entropy annex, `--quartet` the quartet annex, `--pair-balance` the pair-weight and family-overlap annexes, `--family-overlap` or `--min-shared-family` the family-overlap annex. For example `--sibling-annex --quartet --only-diploid` prepares exactly the quartet annex of that convention, and `--sibling-annex` alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally | | `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only | | `--sibling-hist` | Write `<prefix>_sibling_hist.csv` (columns `size,count`): the global family-size histogram (1-4 members) only, without the per-genome breakdown |
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for | | `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan | | `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
| `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index | | `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index |
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
### `--sibling-stats` ### `--sibling-stats`
@@ -73,50 +153,6 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold. `--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
### `--pseudo-alignment`
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
#### VCF output (`--vcf`)
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
| Option | Default | Description |
|---|---|---|
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
## Sampling at scale: `--n-sites`, `--shannon`, `--entropy`
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
Every invocation, with or without `--session`, draws its own fresh random sample by default — running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately. `--session` does not change this: it makes a *specific* sample reusable on request, it does not make sampling itself reproducible from one independent run to the next.
### `--session`: reusing a sample across separate commands
| Option | Default | Description |
|---|---|---|
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
### `--shannon`: measuring how informative a family is ### `--shannon`: measuring how informative a family is
`<prefix>_entropy.csv` has one row per family visited: `<prefix>_entropy.csv` has one row per family visited:
@@ -134,43 +170,27 @@ With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`)
`--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample. `--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
### `--quartet`: keeping only families that can resolve a tree ### `--shannon-summary`
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
### Describing a draw: `--sample-stats`, `--sample-plan`
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
`--sample-stats` writes three files:
| File | Content |
|---|---|
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. Excluded genomes are listed with zeros in the first file and count nowhere.
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
`--shannon-summary` writes `<prefix>_shannon_summary.csv` (`entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families`): the number of variable families per entropy (`entropy15` over every carrier, `entropy4` over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (`bin_start` is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers `--no-ambiguity` loses, from the same scan as `--shannon` but without one line per family. `--shannon-summary` writes `<prefix>_shannon_summary.csv` (`entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families`): the number of variable families per entropy (`entropy15` over every carrier, `entropy4` over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (`bin_start` is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers `--no-ambiguity` loses, from the same scan as `--shannon` but without one line per family.
### `--pair-balance`: representing every pair of genomes ## SNP pseudo-alignment
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs. A pseudo-alignment has one column per variable family and one state per genome. Unlike a conventional sequence alignment, its columns come directly from groups of related k-mers of the index. The number of sites is set with `--n-sites`, and the options of "SNP character selection and sampling" above control which families are included and how ambiguous or absent states are written.
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first. | Option | Description |
|---|---|
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families ### `--pseudo-alignment`
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target). `<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data. #### VCF output (`--vcf`)
The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given. The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
## Sankoff calibration and phylogenetic exports ## Sankoff calibration and phylogenetic exports
@@ -205,7 +225,7 @@ TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in t
## Output files ## Output files
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix, if one is computed, is produced, on stdout. All matrices use genome labels as row/column headers, in index order. With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, the distance matrix goes to standard output and every other file is written in the current directory under its name without the `<prefix>_` part (for example `nj.nwk`, `entropy.csv`, `alignment.fasta`). All matrices use genome labels as row/column headers, in index order.
### Distance matrix ### Distance matrix
@@ -219,13 +239,14 @@ With `-o/--output PREFIX`, the relevant subset of the files below is written. Wi
CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`. CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
### Central-position SNP model ### SNP analyses
| File | Written by | Format | Content | | File | Written by | Format | Content |
|---|---|---|---| |---|---|---|---|
| `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global | | `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
| `<prefix>_sibling_hist.csv` | `--sibling-hist` | CSV table | global family-size histogram (`size,count`) |
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for | | `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above | | `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see `--shannon` above |
| `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers | | `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers |
| `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes | | `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes |
| `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept | | `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept |
+426 -10
View File
@@ -247,15 +247,26 @@
<li class="md-nav__item">
<a href="/theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -263,12 +274,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -276,7 +295,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="/theory/encoding/" class="md-nav__link"> <a href="/theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -304,14 +323,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="/theory/entropy_filter/" class="md-nav__link"> <a href="/theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -332,7 +351,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="/theory/minimizer_selection/" class="md-nav__link"> <a href="/theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -360,7 +379,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="/theory/indexing_architecture/" class="md-nav__link"> <a href="/theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -381,6 +428,347 @@
<li class="md-nav__item">
<a href="/theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="/theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="/theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="/theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -937,6 +1325,34 @@
<li class="md-nav__item">
<a href="/formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+429 -13
View File
@@ -9,7 +9,7 @@
<link rel="prev" href="../formats/index_layout/"> <link rel="prev" href="../formats/vcf/">
@@ -254,15 +254,26 @@
<li class="md-nav__item">
<a href="../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -270,12 +281,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -283,7 +302,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/encoding/" class="md-nav__link"> <a href="../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -311,14 +330,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/entropy_filter/" class="md-nav__link"> <a href="../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -339,7 +358,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/minimizer_selection/" class="md-nav__link"> <a href="../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -367,7 +386,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/indexing_architecture/" class="md-nav__link"> <a href="../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -388,6 +435,347 @@
<li class="md-nav__item">
<a href="../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -944,6 +1332,34 @@
<li class="md-nav__item">
<a href="../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1157,13 +1573,13 @@
<h1 id="architecture-notes-for-advanced-use">Architecture notes for advanced use</h1> <h1 id="architecture-notes-for-advanced-use">Architecture notes for advanced use</h1>
<p>This page describes execution-level behavior relevant to sizing and running <code>obikmer</code> on large datasets or multi-socket machines. It complements the <a href="../formats/index_layout/">index format</a> and <a href="../theory/indexing_architecture/">theory</a> pages.</p> <p>This page describes execution-level behavior relevant to sizing and running <code>obikmer</code> on large datasets or multi-socket machines. It complements the <a href="../formats/index_layout/">index format</a> and <a href="../theory/kmer_indexing/indexing_architecture/">theory</a> pages.</p>
<h2 id="sequence-invariant">Sequence invariant</h2> <h2 id="sequence-invariant">Sequence invariant</h2>
<p>Every input sequence is treated purely as a compact representation of a set of overlapping kmers:</p> <p>Every input sequence is treated purely as a compact representation of a set of overlapping kmers:</p>
<ul> <ul>
<li>Only the <code>A</code>/<code>C</code>/<code>G</code>/<code>T</code> alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.</li> <li>Only the <code>A</code>/<code>C</code>/<code>G</code>/<code>T</code> alphabet (case-insensitive) is recognized; a sequence is cut at any other character (including IUPAC ambiguity codes), so runs containing them are not represented in the index.</li>
<li>Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.</li> <li>Sequences are internally processed in chunks of at most 256 nucleotides; a chunk shorter than k is dropped. This is invisible to the user beyond the ACGT-only, minimum-length-k constraints above.</li>
<li>Kmers are always handled in canonical form (see <a href="../theory/encoding/">DNA encoding</a>), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.</li> <li>Kmers are always handled in canonical form (see <a href="../theory/kmer_indexing/encoding/">DNA encoding</a>), so the tool is strand-agnostic throughout: a kmer and its reverse complement are always the same entry.</li>
</ul> </ul>
<h2 id="index-dimensioning">Index dimensioning</h2> <h2 id="index-dimensioning">Index dimensioning</h2>
<p>An index directory is organized as <code>KmerIndex → partitions → layers</code>, with a canonical kmer belonging to exactly one (partition, layer) pair. This is what makes set operations (merge, filter, distance) parallel and coordination-free across partitions.</p> <p>An index directory is organized as <code>KmerIndex → partitions → layers</code>, with a canonical kmer belonging to exactly one (partition, layer) pair. This is what makes set operations (merge, filter, distance) parallel and coordination-free across partitions.</p>
+429 -13
View File
@@ -12,7 +12,7 @@
<link rel="prev" href="../../usage/predicates/"> <link rel="prev" href="../../usage/predicates/">
<link rel="next" href="../../architecture/"> <link rel="next" href="../vcf/">
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1039,6 +1427,34 @@
<li class="md-nav__item">
<a href="../vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1162,9 +1578,9 @@
<h1 id="index-construction-and-on-disk-layout">Index construction and on-disk layout</h1> <h1 id="index-construction-and-on-disk-layout">Index construction and on-disk layout</h1>
<h2 id="construction-pipeline">Construction pipeline</h2> <h2 id="construction-pipeline">Construction pipeline</h2>
<p>Building an index (<a href="../../usage/index_command/"><code>index</code></a>) proceeds through a fixed sequence of phases, each operating independently per partition (see <a href="../../theory/indexing_architecture/">Partitioning and indexing architecture</a>):</p> <p>Building an index (<a href="../../usage/index_command/"><code>index</code></a>) proceeds through a fixed sequence of phases, each operating independently per partition (see <a href="../../theory/kmer_indexing/indexing_architecture/">Partitioning and indexing architecture</a>):</p>
<ol> <ol>
<li><strong>Scatter.</strong> A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see <a href="../../theory/entropy_filter/">Low-complexity kmer filter</a>), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by <code>hash(minimizer) mod n_partitions</code> into one file per partition.</li> <li><strong>Scatter.</strong> A single streaming pass over the input. Each sequence fragment is cut at non-ACGT bases, passed through the low-complexity entropy filter (see <a href="../../theory/kmer_indexing/entropy_filter/">Low-complexity kmer filter</a>), and any resulting segment shorter than k is dropped. Surviving segments are decomposed into super-kmers, canonicalized, and routed by <code>hash(minimizer) mod n_partitions</code> into one file per partition.</li>
<li><strong>Dereplication.</strong> Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.</li> <li><strong>Dereplication.</strong> Within each partition, identical super-kmer sequences are merged and their occurrence counts summed. This count is per super-kmer, not per kmer — a kmer's true abundance is the sum of the counts of every super-kmer containing it.</li>
<li><strong>Exact counting.</strong> Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.</li> <li><strong>Exact counting.</strong> Every kmer in every dereplicated super-kmer is enumerated and its exact total count computed. A per-genome kmer frequency spectrum is produced at this stage.</li>
<li><strong>Quorum filtering.</strong> Kmers outside the <code>--min-abundance</code>/<code>--max-abundance</code> range are dropped, and super-kmers are recompacted around the surviving kmer set.</li> <li><strong>Quorum filtering.</strong> Kmers outside the <code>--min-abundance</code>/<code>--max-abundance</code> range are dropped, and super-kmers are recompacted around the surviving kmer set.</li>
File diff suppressed because it is too large. Load diff
+430 -14
View File
@@ -367,15 +367,26 @@
<li class="md-nav__item">
<a href="theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -383,12 +394,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -396,7 +415,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="theory/encoding/" class="md-nav__link"> <a href="theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -424,14 +443,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="theory/entropy_filter/" class="md-nav__link"> <a href="theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -452,7 +471,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="theory/minimizer_selection/" class="md-nav__link"> <a href="theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -480,7 +499,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="theory/indexing_architecture/" class="md-nav__link"> <a href="theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -501,6 +548,347 @@
<li class="md-nav__item">
<a href="theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1057,6 +1445,34 @@
<li class="md-nav__item">
<a href="formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1205,11 +1621,11 @@
<p>All functionality is exposed through a single binary, <code>obikmer</code>, organized as subcommands.</p> <p>All functionality is exposed through a single binary, <code>obikmer</code>, organized as subcommands.</p>
<h2 id="core-principles">Core principles</h2> <h2 id="core-principles">Core principles</h2>
<ul> <ul>
<li>Kmers are of fixed, odd length <span class="arithmatex">\(k\)</span>, chosen at index-construction time in the range <span class="arithmatex">\([11, 31]\)</span> (see <a href="theory/kmers_and_superkmers/">Kmers and super-kmers</a>).</li> <li>Kmers are of fixed, odd length <span class="arithmatex">\(k\)</span>, chosen at index-construction time in the range <span class="arithmatex">\([11, 31]\)</span> (see <a href="theory/kmer_indexing/kmers/">Kmers</a>).</li>
<li>Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see <a href="theory/encoding/">DNA encoding</a>).</li> <li>Each kmer fits in a 64-bit word using a 2-bit-per-base encoding (see <a href="theory/kmer_indexing/encoding/">DNA encoding</a>).</li>
<li>Kmers are handled in <strong>canonical form</strong> (<span class="arithmatex">\(\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))\)</span>), making counting strand-independent.</li> <li>Kmers are handled in <strong>canonical form</strong> (<span class="arithmatex">\(\text{canonical}(kmer) = \min(kmer, \text{revcomp}(kmer))\)</span>), making counting strand-independent.</li>
<li>Sequences are decomposed into <strong>super-kmers</strong> before storage, anchored on a hash-selected <strong>minimizer</strong> (see <a href="theory/minimizer_selection/">Minimizer selection</a>), then routed to one of several <strong>partitions</strong> for parallel, memory-bounded processing (see <a href="theory/indexing_architecture/">Partitioning and indexing architecture</a>).</li> <li>Sequences are decomposed into <strong>super-kmers</strong> before storage, anchored on a hash-selected <strong>minimizer</strong> (see <a href="theory/kmer_indexing/minimizer_selection/">Minimizer selection</a>), then routed to one of several <strong>partitions</strong> for parallel, memory-bounded processing (see <a href="theory/kmer_indexing/indexing_architecture/">Partitioning and indexing architecture</a>).</li>
<li>Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see <a href="theory/entropy_filter/">Low-complexity kmer filter</a>).</li> <li>Low-complexity kmers can be filtered out at index-construction time using an entropy-based score (see <a href="theory/kmer_indexing/entropy_filter/">Low-complexity kmer filter</a>).</li>
</ul> </ul>
<h2 id="commands">Commands</h2> <h2 id="commands">Commands</h2>
<table> <table>
+427 -11
View File
@@ -12,7 +12,7 @@
<link rel="prev" href=".."> <link rel="prev" href="..">
<link rel="next" href="../theory/kmers_and_superkmers/"> <link rel="next" href="../theory/kmer_indexing/encoding/">
@@ -442,15 +442,26 @@
<li class="md-nav__item">
<a href="../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -458,12 +469,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -471,7 +490,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/encoding/" class="md-nav__link"> <a href="../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -499,14 +518,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/entropy_filter/" class="md-nav__link"> <a href="../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -527,7 +546,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/minimizer_selection/" class="md-nav__link"> <a href="../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -555,7 +574,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../theory/indexing_architecture/" class="md-nav__link"> <a href="../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -576,6 +623,347 @@
<li class="md-nav__item">
<a href="../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1132,6 +1520,34 @@
<li class="md-nav__item">
<a href="../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+223
View File
@@ -0,0 +1,223 @@
<?xml version="1.0" encoding="utf-8"?>
<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" default-locale="en-US" demote-non-dropping-particle="sort-only">
<info>
<title>Molecular Ecology Modified</title>
<id>http://www.zotero.org/styles/molecular-ecology</id>
<link href="http://www.zotero.org/styles/molecular-ecology" rel="self"/>
<link href="http://www.zotero.org/styles/ecology-letters" rel="template"/>
<link href="http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291365-294X/homepage/ForAuthors.html" rel="documentation"/>
<author>
<name>Xiaodong Dang</name>
<email>dangxdong@gmail.com</email>
</author>
<contributor>
<name>Johan Asplund</name>
<email>asplundj@gmail.com</email>
</contributor>
<contributor>
<name>David Kaplan</name>
<email>david.kaplan@ird.fr</email>
</contributor>
<category citation-format="author-date"/>
<category field="biology"/>
<issn>0962-1083</issn>
<eissn>1365-294X</eissn>
<updated>2012-09-09T22:24:43+00:00</updated>
<rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
</info>
<macro name="editor-translator">
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
<name initialize-with="" delimiter=", "/>
<label form="short" prefix=", " text-case="capitalize-first" suffix="," strip-periods="true"/>
</names>
</macro>
<macro name="author">
<names variable="author">
<name name-as-sort-order="all" sort-separator=" " initialize-with="" delimiter=", " delimiter-precedes-et-al="never"/>
<label form="short" prefix=" (" suffix=")" text-case="capitalize-first"/>
<et-al font-style="italic"/>
<substitute>
<names variable="editor"/>
<names variable="translator"/>
<text macro="title"/>
</substitute>
</names>
</macro>
<macro name="author-short">
<names variable="author">
<name form="short" and="symbol" delimiter=", " initialize-with=". "/>
<et-al font-style="italic"/>
<substitute>
<names variable="editor"/>
<names variable="translator"/>
<choose>
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
<text variable="title" form="short" font-style="italic"/>
</if>
<else>
<text variable="title" form="short" quotes="true"/>
</else>
</choose>
</substitute>
</names>
</macro>
<macro name="title">
<choose>
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
<text variable="title" font-style="italic"/>
</if>
<else>
<text variable="title"/>
</else>
</choose>
</macro>
<macro name="access-detail">
<group delimiter=" ">
<text variable="URL" prefix="Available at: "/>
<group prefix="(" suffix=")" delimiter=" ">
<text term="accessed"/>
<date variable="accessed">
<date-part name="day" suffix=" "/>
<date-part name="month" form="long" suffix=" "/>
<date-part name="year"/>
</date>
</group>
</group>
</macro>
<macro name="access">
<choose>
<if type="webpage post post-weblog entry entry-encyclopedia dataset software report manuscript document" match="any">
<text macro="access-detail"/>
</if>
<else-if variable="accessed">
<text macro="access-detail"/>
</else-if>
</choose>
</macro>
<macro name="access-fallback">
<group delimiter=" ">
<choose>
<if variable="URL">
<text macro="access-detail" suffix="."/>
</if>
<else-if variable="publisher">
<text variable="publisher" prefix="Available at: " suffix="."/>
</else-if>
</choose>
<text variable="note" suffix="."/>
</group>
</macro>
<macro name="issued-year">
<choose>
<if variable="issued">
<date variable="issued">
<date-part name="year"/>
</date>
</if>
<else>
<text term="no date" form="short"/>
</else>
</choose>
</macro>
<macro name="publisher">
<group delimiter=", ">
<text variable="publisher"/>
<text variable="publisher-place"/>
</group>
</macro>
<citation collapse="year-suffix" et-al-min="3" et-al-use-first="1" disambiguate-add-year-suffix="true">
<sort>
<key variable="issued"/>
<key variable="author"/>
</sort>
<layout prefix="(" suffix=")" delimiter="; ">
<group delimiter=" ">
<text macro="author-short"/>
<text macro="issued-year"/>
<group>
<label variable="locator" form="short"/>
<text variable="locator" prefix=" "/>
</group>
</group>
</layout>
</citation>
<bibliography hanging-indent="true" et-al-min="6" et-al-use-first="1000" entry-spacing="0.5" line-spacing="1">
<sort>
<key macro="author-short"/>
<key variable="issued"/>
</sort>
<layout>
<text macro="author" suffix=" "/>
<text macro="issued-year" prefix="(" suffix=")"/>
<choose>
<if type="bill book graphic legal_case legislation motion_picture report song" match="any">
<group suffix=".">
<text macro="title" prefix=" "/>
<text macro="editor-translator" prefix=" "/>
</group>
<text prefix=" " suffix="." macro="publisher"/>
<text macro="access" prefix=" " suffix="."/>
</if>
<else-if type="thesis" match="any">
<group suffix=".">
<text macro="title" prefix=" " suffix="."/>
<text variable="genre" prefix=" " suffix=" Thesis."/>
<text prefix=" " macro="publisher"/>
</group>
<text macro="access" prefix=" " suffix="."/>
</else-if>
<else-if type="chapter paper-conference" match="any">
<text macro="title" prefix=" "/>
<group prefix=".">
<group prefix=" In:">
<text variable="container-title" font-style="italic" prefix=" "/>
<text variable="collection-title" prefix=" " suffix="."/>
<names variable="editor translator" prefix=" (" delimiter=", " suffix=")">
<label form="short" suffix=" " strip-periods="true"/>
<name name-as-sort-order="all" sort-separator=" " initialize-with="" delimiter=", " delimiter-precedes-last="never"/>
</names>
<group suffix=".">
<label variable="page" form="short" prefix=", " suffix=" "/>
<text variable="page"/>
<text macro="publisher" prefix=". " suffix="."/>
</group>
</group>
</group>
</else-if>
<else-if type="webpage post post-weblog entry entry-encyclopedia dataset software" match="any">
<group suffix=".">
<text macro="title" prefix=" "/>
<text macro="editor-translator" prefix=" "/>
</group>
<group prefix=" " suffix=".">
<text variable="container-title" text-case="title" font-style="italic"/>
<text variable="genre" prefix=", "/>
<text macro="publisher" prefix=". "/>
</group>
<text macro="access" prefix=" " suffix="."/>
</else-if>
<else>
<group suffix=".">
<text macro="title" prefix=" "/>
<text macro="editor-translator" prefix=" "/>
</group>
<group prefix=" " suffix=".">
<text variable="container-title" text-case="title" font-style="italic"/>
<group prefix=", ">
<text variable="volume" font-weight="bold"/>
</group>
<text variable="page" prefix=", "/>
</group>
<choose>
<if variable="container-title">
<text macro="access" prefix=" " suffix="."/>
</if>
<else>
<text macro="access-fallback" prefix=" "/>
</else>
</choose>
</else>
</choose>
</layout>
</bibliography>
</style>
+598
View File
@@ -259,3 +259,601 @@
url = {https://doi.org/10.1186/s12864-015-1647-5}, url = {https://doi.org/10.1186/s12864-015-1647-5},
volume = 16, volume = 16,
year = 2015} year = 2015}
@article{Roberts2004-rz,
title = "{Reducing storage requirements for biological sequence
comparison}",
author = "Roberts, Michael and Hayes, Wayne and Hunt, Brian R and Mount,
Stephen M and Yorke, James A",
journal = "Bioinformatics (Oxford, England)",
publisher = "Oxford University Press (OUP)",
volume = 20,
number = 18,
pages = "3363--3369",
abstract = "MOTIVATION: Comparison of nucleic acid and protein sequences is a
fundamental tool of modern bioinformatics. A dominant method of
such string matching is the 'seed-and-extend' approach, in which
occurrences of short subsequences called 'seeds' are used to
search for potentially longer matches in a large database of
sequences. Each such potential match is then checked to see if it
extends beyond the seed. To be effective, the seed-and-extend
approach needs to catalogue seeds from virtually every substring
in the database of search strings. Projects such as mammalian
genome assemblies and large-scale protein matching, however, have
such large sequence databases that the resulting list of seeds
cannot be stored in RAM on a single computer. This significantly
slows the matching process. RESULTS: We present a simple and
elegant method in which only a small fraction of seeds, called
'minimizers', needs to be stored. Using minimizers can speed up
string-matching computations by a large factor while missing only
a small fraction of the matches found using all seeds.",
month = dec,
year = 2004,
url = "https://dx.doi.org/10.1093/bioinformatics/bth408",
doi = "10.1093/bioinformatics/bth408",
pmid = 15256412,
issn = "1367-4803,1367-4811",
language = "en"
}
@INPROCEEDINGS{Steele2014-cz,
title = "{Fast splittable pseudorandom number generators}",
author = "Steele, Jr, Guy L and Lea, Doug and Flood, Christine H",
booktitle = "{Proceedings of the 2014 ACM International Conference on Object
Oriented Programming Systems Languages \& Applications}",
publisher = "ACM",
address = "New York, NY, USA",
abstract = "We describe a new algorithm SplitMix for an object-oriented and
splittable pseudorandom number generator (PRNG) that is quite
fast: 9 64-bit arithmetic/logical operations per 64 bits
generated. A conventional linear PRNG object provides a generate
method that returns one pseudorandom value and updates the state
of the PRNG, but a splittable PRNG object also has a second
operation, split, that replaces the original PRNG object with two
(seemingly) independent PRNG objects, by creating and returning a
new such object and updating the state of the original object.
Splittable PRNG objects make it easy to organize the use of
pseudorandom numbers in multithreaded programs structured using
fork-join parallelism. No locking or synchronization is required
(other than the usual memory fence immediately after object
creation). Because the generate method has no loops or
conditionals, it is suitable for SIMD or GPU implementation. We
derive SplitMix from the DotMix algorithm of Leiserson, Schardl,
and Sukha by making a series of program transformations and
engineering improvements. The end result is an object-oriented
version of the purely functional API used in the Haskell library
for over a decade, but SplitMix is faster and produces
pseudorandom sequences of higher quality; it is also far superior
in quality and speed to java.util.Random, and has been included
in Java JDK8 as the class java.util.SplittableRandom. We have
tested the pseudorandom sequences produced by SplitMix using two
standard statistical test suites (DieHarder and TestU01) and they
appear to be adequate for ``everyday'' use, such as in Monte
Carlo algorithms and randomized data structures where speed is
important.",
month = oct,
year = 2014,
url = "http://dx.doi.org/10.1145/2660193.2660195",
doi = "10.1145/2660193.2660195",
isbn = 9781450325851
}
@online{Stafford2011,
author = {Stafford, David},
title = {Better Bit Mixing - Improving on MurmurHash3's 64-bit Finalizer},
date = {2011-09-28},
url = {https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html},
urldate = {2026-09-12}
}
@ARTICLE{Hamming1950-cw,
title = "{Error detecting and error correcting codes}",
author = "Hamming, R W",
journal = "The Bell System Technical Journal",
publisher = "Institute of Electrical and Electronics Engineers (IEEE)",
volume = 29,
number = 2,
pages = "147--160",
abstract = "The author was led to the study given in this paper from a
consideration of large scale computing machines in which a large
number of operations must be performed without a single error in
the end result. This problem of “doing things right” on a large
scale is not essentially new; in a telephone central office, for
example, a very large number of operations are performed while
the errors leading to wrong numbers are kept well under control,
though they have not been completely eliminated. This has been
achieved, in part, through the use of self-checking circuits. The
occasional failure that escapes routine checking is still
detected by the customer and will, if it persists, result in
customer complaint, while if it is transient it will produce only
occasional wrong numbers. At the same time the rest of the
central office functions satisfactorily. In a digital computer,
on the other hand, a single failure usually means the complete
failure, in the sense that if it is detected no more computing
can be done until the failure is located and corrected, while if
it escapes detection then it invalidates all subsequent
operations of the machine. Put in other words, in a telephone
central office there are a number of parallel paths which are
more or less independent of each other; in a digital machine
there is usually a single long path which passes through the same
piece of equipment many, many times before the answer is
obtained.",
month = apr,
year = 1950,
url = "http://dx.doi.org/10.1002/j.1538-7305.1950.tb00463.x",
doi = "10.1002/j.1538-7305.1950.tb00463.x",
issn = "0005-8580"
}
@ARTICLE{Ondov2016-dq,
title = "{Mash: fast genome and metagenome distance estimation using
{MinHash}}",
author = "Ondov, Brian D and Treangen, Todd J and Melsted, Páll and
Mallonee, Adam B and Bergman, Nicholas H and Koren, Sergey and
Phillippy, Adam M",
journal = "Genome biology",
volume = 17,
number = 1,
pages = 132,
abstract = "Mash extends the MinHash dimensionality-reduction technique to
include a pairwise mutation distance and P value significance
test, enabling the efficient clustering and search of massive
sequence collections. Mash reduces large sequences and sequence
sets to small, representative sketches, from which global mutation
distances can be rapidly estimated. We demonstrate several use
cases, including the clustering of all 54,118 NCBI RefSeq genomes
in 33 CPU h; real-time database search using assembled or
unassembled Illumina, Pacific Biosciences, and Oxford Nanopore
data; and the scalable clustering of hundreds of metagenomic
samples by composition. Mash is freely released under a BSD
license ( https://github.com/marbl/mash ).",
month = jun,
year = 2016,
url = "http://dx.doi.org/10.1186/s13059-016-0997-x",
keywords = "Alignment; Comparative genomics; Genomic distance; Metagenomics;
Nanopore; Sequencing",
doi = "10.1186/s13059-016-0997-x",
pmc = "PMC4915045",
pmid = 27323842,
issn = "1465-6906",
language = "en"
}
@ARTICLE{Bray1957-vv,
title = "{An ordination of the upland forest communities of southern
Wisconsin}",
author = "Bray, J Roger and Curtis, J T",
journal = "Ecological Monographs",
publisher = "Wiley",
volume = 27,
number = 4,
pages = "325--349",
month = oct,
year = 1957,
url = "http://dx.doi.org/10.2307/1942268",
doi = "10.2307/1942268",
issn = "0012-9615,1557-7015",
language = "en"
}
@ARTICLE{Gower1986-ri,
title = "{Metric and Euclidean properties of dissimilarity coefficients}",
author = "Gower, J C and Legendre, P",
journal = "Journal of Classification",
publisher = "Springer Science and Business Media LLC",
volume = 3,
number = 1,
pages = "5--48",
abstract = "We assemble here properties of certain dissimilarity coefficients
and are specially concerned with their metric and Euclidean
status. No attempt is made to be exhaustive as far as
coefficients are concerned, but certain mathematical results that
we have found useful are presented and should help establish
similar properties for other coefficients. The response to
different types of data is investigated, leading to guidance on
the choice of an appropriate coefficient.",
month = mar,
year = 1986,
url = "http://dx.doi.org/10.1007/bf01896809",
doi = "10.1007/bf01896809",
issn = "0176-4268,1432-1343",
language = "en"
}
@ARTICLE{Legendre2001-cq,
title = "{Ecologically meaningful transformations for ordination of species
data}",
author = "Legendre, Pierre and Gallagher, Eugene D",
journal = "Oecologia",
volume = 129,
number = 2,
pages = "271--280",
abstract = "This paper examines how to obtain species biplots in unconstrained
or constrained ordination without resorting to the Euclidean
distance [used in principal-component analysis (PCA) and
redundancy analysis (RDA)] or the chi-square distance [preserved
in correspondence analysis (CA) and canonical correspondence
analysis (CCA)] which are not always appropriate for the analysis
of community composition data. To achieve this goal,
transformations are proposed for species data tables. They allow
ecologists to use ordination methods such as PCA and RDA, which
are Euclidean-based, for the analysis of community data, while
circumventing the problems associated with the Euclidean distance,
and avoiding CA and CCA which present problems of their own in
some cases. This allows the use of the original (transformed)
species data in RDA carried out to test for relationships with
explanatory variables (i.e. environmental variables, or factors of
a multifactorial analysis-of-variance model); ecologists can then
draw biplots displaying the relationships of the species to the
explanatory variables. Another application allows the use of
species data in other methods of multivariate data analysis which
optimize a least-squares loss function; an example is K-means
partitioning.",
month = oct,
year = 2001,
url = "http://dx.doi.org/10.1007/s004420100716",
keywords = "Biplot diagram; Canonical correspondence analysis; Correspondence
analysis; Principal-component analysis; Redundancy analysis",
doi = "10.1007/s004420100716",
pmid = 28547606,
issn = "0029-8549,1432-1939",
language = "en"
}
@ARTICLE{Anderson2011-rq,
title = "{Navigating the multiple meanings of β diversity: a roadmap for
the practicing ecologist}",
author = "Anderson, Marti J and Crist, Thomas O and Chase, Jonathan M and
Vellend, Mark and Inouye, Brian D and Freestone, Amy L and
Sanders, Nathan J and Cornell, Howard V and Comita, Liza S and
Davies, Kendi F and Harrison, Susan P and Kraft, Nathan J B and
Stegen, James C and Swenson, Nathan G",
journal = "Ecology Letters",
publisher = "Wiley",
volume = 14,
number = 1,
pages = "19--28",
abstract = "A recent increase in studies of β diversity has yielded a
confusing array of concepts, measures and methods. Here, we
provide a roadmap of the most widely used and ecologically
relevant approaches for analysis through a series of mission
statements. We distinguish two types of β diversity: directional
turnover along a gradient vs. non-directional variation.
Different measures emphasize different properties of ecological
data. Such properties include the degree of emphasis on
presence/absence vs. relative abundance information and the
inclusion vs. exclusion of joint absences. Judicious use of
multiple measures in concert can uncover the underlying nature of
patterns in β diversity for a given dataset. A case study of
Indonesian coral assemblages shows the utility of a multi-faceted
approach. We advocate careful consideration of relevant
questions, matched by appropriate analyses. The rigorous
application of null models will also help to reveal potential
processes driving observed patterns in β diversity.",
month = jan,
year = 2011,
url = "http://dx.doi.org/10.1111/j.1461-0248.2010.01552.x",
doi = "10.1111/j.1461-0248.2010.01552.x",
pmid = 21070562,
issn = "1461-023X,1461-0248",
language = "en"
}
@ARTICLE{Jaccard1901-mx,
title = "{Étude comparative de la distribution florale dans une portion
des Alpes et du Jura}",
author = "Jaccard, Paul",
journal = "Bulletin de la Société vaudoise des sciences naturelles",
publisher = "Imprimerie Corbaz \& Comp.",
volume = 37,
number = 142,
pages = "547--579",
abstract = "Schweizer Zeitschriften online - Ein Service der ETH-Bibliothek.
Revues suisses en ligne - Un service de ETH Library. Swiss
journals online - A service by ETH Library.",
year = 1901,
url = "http://dx.doi.org/10.5169/seals-266450",
doi = "10.5169/seals-266450",
issn = "0037-9603",
language = "de"
}
@ARTICLE{Kosub2019-xt,
title = "{A note on the triangle inequality for the Jaccard distance}",
author = "Kosub, Sven",
journal = "Pattern Recognition Letters",
publisher = "Elsevier BV",
volume = 120,
pages = "36--38",
month = apr,
year = 2019,
url = "http://dx.doi.org/10.1016/j.patrec.2018.12.007",
doi = "10.1016/j.patrec.2018.12.007",
issn = "0167-8655,1872-7344",
language = "en"
}
@ARTICLE{Fleischmann2014-ed,
title = {Evolution of genome size and chromosome number in the
carnivorous plant genus Genlisea (Lentibulariaceae), with a new
estimate of the minimum genome size in angiosperms},
author = "Fleischmann, Andreas and Michael, Todd P and Rivadavia, Fernando
and Sousa, Aretuza and Wang, Wenqin and Temsch, Eva M and
Greilhuber, Johann and Müller, Kai F and Heubl, Günther",
journal = "Annals of Botany",
publisher = "Oxford University Press (OUP)",
volume = 114,
number = 8,
pages = "1651--1663",
abstract = "BACKGROUND AND AIMS: Some species of Genlisea possess ultrasmall
nuclear genomes, the smallest known among angiosperms, and some
have been found to have chromosomes of diminutive size, which may
explain why chromosome numbers and karyotypes are not known for
the majority of species of the genus. However, other members of
the genus do not possess ultrasmall genomes, nor do most taxa
studied in related genera of the family or order. This study
therefore examined the evolution of genome sizes and chromosome
numbers in Genlisea in a phylogenetic context. The correlations
of genome size with chromosome number and size, with the
phylogeny of the group and with growth forms and habitats were
also examined. METHODS: Nuclear genome sizes were measured from
cultivated plant material for a comprehensive sampling of taxa,
including nearly half of all species of Genlisea and representing
all major lineages. Flow cytometric measurements were conducted
in parallel in two laboratories in order to compare the
consistency of different methods and controls. Chromosome counts
were performed for the majority of taxa, comparing different
staining techniques for the ultrasmall chromosomes. KEY RESULTS:
Genome sizes of 15 taxa of Genlisea are presented and interpreted
in a phylogenetic context. A high degree of congruence was found
between genome size distribution and the major phylogenetic
lineages. Ultrasmall genomes with 1C values of <100 Mbp were
almost exclusively found in a derived lineage of South American
species. The ancestral haploid chromosome number was inferred to
be n = 8. Chromosome numbers in Genlisea ranged from 2n = 2x = 16
to 2n = 4x = 32. Ascendant dysploid series (2n = 36, 38) are
documented for three derived taxa. The different ploidy levels
corresponded to the two subgenera, but were not directly
correlated to differences in genome size; the three different
karyotype ranges mirrored the different sections of the genus.
The smallest known plant genomes were not found in G. margaretae,
as previously reported, but in G. tuberosa (1C ≈ 61 Mbp) and some
strains of G. aurea (1C ≈ 64 Mbp). CONCLUSIONS: Genlisea is an
ideal candidate model organism for the understanding of genome
reduction as the genus includes species with both relatively
large (∼1700 Mbp) and ultrasmall (∼61 Mbp) genomes. This
comparative, phylogeny-based analysis of genome sizes and
karyotypes in Genlisea provides essential data for selection of
suitable species for comparative whole-genome analyses, as well
as for further studies on both the molecular and cytogenetic
basis of genome reduction in plants.",
month = dec,
year = 2014,
url = "http://dx.doi.org/10.1093/aob/mcu189",
keywords = "Bladderwort; Genlisea; Lamiales; Lentibulariaceae; carnivorous
plant; chromosome number; flow cytometry; genome miniaturization;
genome size",
doi = "10.1093/aob/mcu189",
pmc = "PMC4649684",
pmid = 25274549,
issn = "0305-7364,1095-8290",
language = "en"
}
@ARTICLE{Fernandez2024-kd,
title = "A 160 Gbp fork fern genome shatters size record for eukaryotes",
author = "Fernández, Pol and Amice, Rémy and Bruy, David and Christenhusz,
Maarten J M and Leitch, Ilia J and Leitch, Andrew L and Pokorny,
Lisa and Hidalgo, Oriane and Pellicer, Jaume",
journal = "iScience",
publisher = "Elsevier BV",
volume = 27,
number = 6,
pages = 109889,
abstract = "Vascular plants are exceptional among eukaryotes due to their
outstanding genome size diversity which ranges ∼2,400-fold,
including the largest genome so far recorded in the angiosperm
Paris japonica (148.89 Gbp/1C). Despite available data showing
that giant genomes are restricted across the Tree of Life, the
biological limits to genome size expansion remain to be
established. Here, we report the discovery of an even larger
eukaryotic genome in Tmesipteris oblanceolata, a New Caledonian
fork fern. At 160.45 Gbp/1C, this record-breaking genome
challenges current understanding and opens new avenues to explore
the evolutionary dynamics of genomic gigantism.",
month = jun,
year = 2024,
url = "http://dx.doi.org/10.1016/j.isci.2024.109889",
keywords = "Plant evolution; Plant genetics",
doi = "10.1016/j.isci.2024.109889",
pmc = "PMC11270024",
pmid = 39055604,
issn = "2589-0042",
language = "en"
}
@ARTICLE{Fraser1995-jc,
title = "The minimal gene complement of Mycoplasma genitalium",
author = "Fraser, C M and Gocayne, J D and White, O and Adams, M D and
Clayton, R A and Fleischmann, R D and Bult, C J and Kerlavage, A
R and Sutton, G and Kelley, J M and Fritchman, R D and Weidman, J
F and Small, K V and Sandusky, M and Fuhrmann, J and Nguyen, D
and Utterback, T R and Saudek, D M and Phillips, C A and Merrick,
J M and Tomb, J F and Dougherty, B A and Bott, K F and Hu, P C
and Lucier, T S and Peterson, S N and Smith, H O and Hutchison,
3rd, C A and Venter, J C",
journal = "Science (New York, N.Y.)",
publisher = "American Association for the Advancement of Science (AAAS)",
volume = 270,
number = 5235,
pages = "397--403",
abstract = "The complete nucleotide sequence (580,070 base pairs) of the
Mycoplasma genitalium genome, the smallest known genome of any
free-living organism, has been determined by whole-genome random
sequencing and assembly. A total of only 470 predicted coding
regions were identified that include genes required for DNA
replication, transcription and translation, DNA repair, cellular
transport, and energy metabolism. Comparison of this genome to
that of Haemophilus influenzae suggests that differences in
genome content are reflected as profound differences in
physiology and metabolic capacity between these two organisms.",
month = oct,
year = 1995,
url = "http://dx.doi.org/10.1126/science.270.5235.397",
doi = "10.1126/science.270.5235.397",
pmid = 7569993,
issn = "0036-8075,1095-9203",
language = "en"
}
@ARTICLE{Schneiker2007-gs,
title = "Complete genome sequence of the myxobacterium Sorangium
cellulosum",
author = "Schneiker, Susanne and Perlova, Olena and Kaiser, Olaf and Gerth,
Klaus and Alici, Aysel and Altmeyer, Matthias O and Bartels,
Daniela and Bekel, Thomas and Beyer, Stefan and Bode, Edna and
Bode, Helge B and Bolten, Christoph J and Choudhuri, Jomuna V and
Doss, Sabrina and Elnakady, Yasser A and Frank, Bettina and
Gaigalat, Lars and Goesmann, Alexander and Groeger, Carolin and
Gross, Frank and Jelsbak, Lars and Jelsbak, Lotte and Kalinowski,
Jörn and Kegler, Carsten and Knauber, Tina and Konietzny,
Sebastian and Kopp, Maren and Krause, Lutz and Krug, Daniel and
Linke, Bukhard and Mahmud, Taifo and Martinez-Arias, Rosa and
McHardy, Alice C and Merai, Michelle and Meyer, Folker and
Mormann, Sascha and Muñoz-Dorado, Jose and Perez, Juana and
Pradella, Silke and Rachid, Shwan and Raddatz, Günter and
Rosenau, Frank and Rückert, Christian and Sasse, Florenz and
Scharfe, Maren and Schuster, Stephan C and Suen, Garret and
Treuner-Lange, Anke and Velicer, Gregory J and Vorhölter,
Frank-Jörg and Weissman, Kira J and Welch, Roy D and Wenzel,
Silke C and Whitworth, David E and Wilhelm, Susanne and Wittmann,
Christoph and Blöcker, Helmut and Pühler, Alfred and Müller, Rolf",
journal = "Nature Biotechnology",
publisher = "Springer Science and Business Media LLC",
volume = 25,
number = 11,
pages = "1281--1289",
abstract = "The genus Sorangium synthesizes approximately half of the
secondary metabolites isolated from myxobacteria, including the
anti-cancer metabolite epothilone. We report the complete genome
sequence of the model Sorangium strain S. cellulosum So ce56,
which produces several natural products and has morphological and
physiological properties typical of the genus. The circular
genome, comprising 13,033,779 base pairs, is the largest
bacterial genome sequenced to date. No global synteny with the
genome of Myxococcus xanthus is apparent, revealing an
unanticipated level of divergence between these myxobacteria. A
large percentage of the genome is devoted to regulation,
particularly post-translational phosphorylation, which probably
supports the strain's complex, social lifestyle. This regulatory
network includes the highest number of eukaryotic protein
kinase-like kinases discovered in any organism. Seventeen
secondary metabolite loci are encoded in the genome, as well as
many enzymes with potential utility in industry.",
month = nov,
year = 2007,
url = "http://dx.doi.org/10.1038/nbt1354",
doi = "10.1038/nbt1354",
pmid = 17965706,
issn = "1087-0156,1546-1696",
language = "en"
}
@incollection{Karlin1985,
author = {Karlin, Samuel and Ost, Fr{\'e}d{\'e}ric},
title = {Maximal segmental match length among random sequences from a finite alphabet},
booktitle = {Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer},
editor = {Le Cam, Lucien M. and Olshen, Richard A.},
volume = {1},
pages = {225--243},
year = {1985},
publisher = {Wadsworth},
address = {Monterey, CA}
}
@article{Karlin1988,
author = {Karlin, Samuel and Ost, Fr{\'e}d{\'e}ric},
title = {Maximal length of common words among random letter sequences},
journal = {The Annals of Probability},
volume = {16},
number = {2},
pages = {535--563},
year = {1988}
}
@ARTICLE{Le-Cam1960-pe,
title = "{An approximation theorem for the Poisson binomial distribution}",
author = "Le Cam, Lucien",
journal = "Pacific Journal of Mathematics",
publisher = "Mathematical Sciences Publishers",
volume = 10,
number = 4,
pages = "1181--1197",
month = dec,
year = 1960,
url = "http://dx.doi.org/10.2140/pjm.1960.10.1181",
doi = "10.2140/pjm.1960.10.1181",
issn = "0030-8730,1945-5844"
}
@ARTICLE{Hoeffding1956-bx,
title = "{On the distribution of the number of successes in independent
trials}",
author = "Hoeffding, Wassily",
journal = "The Annals of Mathematical Statistics",
publisher = "Institute of Mathematical Statistics",
volume = 27,
number = 3,
pages = "713--721",
abstract = "Let $S$ be the number of successes in $n$ independent trials, and
let $p_j$ denote the probability of success in the $j$th trial,
$j = 1, 2, \cdots, n$ (Poisson trials). We consider the problem
of finding the maximum and the minimum of $Eg(S),$ the expected
value of a given real-valued function of $S,$ when $ES = np$ is
fixed. It is well known that the maximum of the variance of $S$
is attained when $p_1 = p_2 = \cdots = p_n = p.$ This can be
interpreted as showing that the variability in the number of
successes is highest when the successes are equally probable
(Bernoulli trials). This interpretation is further supported by
the following two theorems, proved in this paper. If $b$ and $c$
are two integers, $0 \leqq b \leqq np \leqq c \leqq n,$ the
probability $P(b \leqq S \leqq c)$ attains its minimum if and
only if $p_1 = p_2 = \cdots = p_n = p,$ unless $b = 0$ and $c =
n$ (Theorem 5, a corollary of Theorem 4, which gives the maximum
and the minimum of $P(S \leqq c)$). If $g$ is a strictly convex
function, $Eg(S)$ attains its maximum if and only if $p_1 = p_2 =
\cdots = p_n = p$ (Theorem 3). These results are obtained with
the help of two theorems concerning the extrema of the expected
value of an arbitrary function $g(S)$ under the condition $ES =
np.$ Theorem 1 gives necessary conditions for the maximum and the
minimum of $Eg(S).$ Theorem 2 gives a partial characterization of
the set of points at which an extremum is attained. Corollary 2.1
states that the maximum and the minimum are attained when $p_1,
p_2, \cdots, p_n$ take on, at most, three different values, only
one of which is distinct from 0 and 1. Applications of Theorems 3
and 5 to problems of estimation and testing are pointed out in
Section 5.",
month = sep,
year = 1956,
url = "http://dx.doi.org/10.1214/aoms/1177728178",
doi = "10.1214/aoms/1177728178",
issn = "0003-4851,2168-8990",
language = "en"
}
@ARTICLE{Gleser1975-yu,
title = "{On the distribution of the number of successes in independent
trials}",
author = "Gleser, Leon Jay",
journal = "Annals of Probability",
publisher = "Institute of Mathematical Statistics",
volume = 3,
number = 1,
pages = "182--188",
abstract = "Let $S$ be the number of successes in $n$ independent Bernoulli
trials, where $p_j$ is the probability of success on the $j$th
trial. Let $\mathbf{p} = (p_1, p_2, \cdots, p_n)$, and for any
integer $c, 0 \leqq c \leqq n$, let $H(c \mid \mathbf{p}) = P\{S
\leqq c\}$. Let $\mathbf{p}^{(1)}$ be one possible choice of
$\mathbf{p}$ for which $E(S) = \lambda$. For any $n \times n$
doubly stochastic matrix $\Pi$, let $\mathbf{p}^{(2)} =
\mathbf{p}^{(1)}\Pi$. Then in the present paper it is shown that
$H(c \mid \mathbf{p}^{(1)}) \leqq H(c \mid \mathbf{p}^{(2)})$ for
$0 \leqq c \leqq \lbrack\lambda - 2\rbrack$, and $H(c \mid
\mathbf{p}^{(1)}) \geqq H(c \mid \mathbf{p}^{(2)})$ for
$\lbrack\lambda + 2\rbrack \leqq c \leqq n$. These results
provide a refinement of inequalities for $H(c \mid \mathbf{p})$
obtained by Hoeffding [3]. Their derivation is achieved by
applying consequences of the partial ordering of majorization.",
month = feb,
year = 1975,
url = "http://dx.doi.org/10.2307/2959277",
doi = "10.1214/aop/1176996461",
issn = "0091-1798,2168-894X"
}
Binary file not shown.
@@ -9,16 +9,16 @@
<link rel="prev" href="../kmers_and_superkmers/"> <link rel="prev" href="../../../installation/">
<link rel="next" href="../entropy_filter/"> <link rel="next" href="../kmers/">
<link rel="icon" href="../../assets/images/favicon.png"> <link rel="icon" href="../../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6"> <meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
@@ -27,7 +27,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css"> <link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
@@ -46,7 +46,7 @@
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script> <script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
@@ -80,7 +80,7 @@
<header class="md-header md-header--shadow" data-md-component="header"> <header class="md-header md-header--shadow" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header"> <nav class="md-header__inner md-grid" aria-label="Header">
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -138,7 +138,7 @@
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0"> <nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer"> <label class="md-nav__title" for="__drawer">
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -156,7 +156,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../.." class="md-nav__link"> <a href="../../.." class="md-nav__link">
@@ -183,7 +183,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../installation/" class="md-nav__link"> <a href="../../../installation/" class="md-nav__link">
@@ -258,15 +258,28 @@
<li class="md-nav__item">
<a href="../kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -274,12 +287,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -395,14 +416,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../entropy_filter/" class="md-nav__link"> <a href="../kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -450,6 +471,34 @@
<li class="md-nav__item">
<a href="../superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../indexing_architecture/" class="md-nav__link"> <a href="../indexing_architecture/" class="md-nav__link">
@@ -472,6 +521,347 @@
<li class="md-nav__item">
<a href="../entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -533,7 +923,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/superkmer/" class="md-nav__link"> <a href="../../../usage/superkmer/" class="md-nav__link">
@@ -561,7 +951,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/index_command/" class="md-nav__link"> <a href="../../../usage/index_command/" class="md-nav__link">
@@ -589,7 +979,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/merge/" class="md-nav__link"> <a href="../../../usage/merge/" class="md-nav__link">
@@ -617,7 +1007,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/filter/" class="md-nav__link"> <a href="../../../usage/filter/" class="md-nav__link">
@@ -645,7 +1035,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/select/" class="md-nav__link"> <a href="../../../usage/select/" class="md-nav__link">
@@ -673,7 +1063,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/query/" class="md-nav__link"> <a href="../../../usage/query/" class="md-nav__link">
@@ -701,7 +1091,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/dump/" class="md-nav__link"> <a href="../../../usage/dump/" class="md-nav__link">
@@ -729,7 +1119,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/annotate/" class="md-nav__link"> <a href="../../../usage/annotate/" class="md-nav__link">
@@ -757,7 +1147,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/phylo/" class="md-nav__link"> <a href="../../../usage/phylo/" class="md-nav__link">
@@ -785,7 +1175,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/unitig/" class="md-nav__link"> <a href="../../../usage/unitig/" class="md-nav__link">
@@ -813,7 +1203,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/estimate/" class="md-nav__link"> <a href="../../../usage/estimate/" class="md-nav__link">
@@ -841,7 +1231,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/convert/" class="md-nav__link"> <a href="../../../usage/convert/" class="md-nav__link">
@@ -869,7 +1259,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/utils/" class="md-nav__link"> <a href="../../../usage/utils/" class="md-nav__link">
@@ -897,7 +1287,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/pack/" class="md-nav__link"> <a href="../../../usage/pack/" class="md-nav__link">
@@ -925,7 +1315,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/predicates/" class="md-nav__link"> <a href="../../../usage/predicates/" class="md-nav__link">
@@ -1007,7 +1397,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../formats/index_layout/" class="md-nav__link"> <a href="../../../formats/index_layout/" class="md-nav__link">
@@ -1028,6 +1418,34 @@
<li class="md-nav__item">
<a href="../../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1042,7 +1460,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../architecture/" class="md-nav__link"> <a href="../../../architecture/" class="md-nav__link">
@@ -1172,9 +1590,9 @@
<p>A kmer of length <span class="arithmatex">\(k\)</span> (<span class="arithmatex">\(k \le 31\)</span>) fits in a single 64-bit word. The first nucleotide occupies the two most significant bits, each following nucleotide occupies the next two bits, and unused low-order bits are zero. Extracting nucleotide i (0-indexed from the 5′ end) is a shift-and-mask operation.</p> <p>A kmer of length <span class="arithmatex">\(k\)</span> (<span class="arithmatex">\(k \le 31\)</span>) fits in a single 64-bit word. The first nucleotide occupies the two most significant bits, each following nucleotide occupies the next two bits, and unused low-order bits are zero. Extracting nucleotide i (0-indexed from the 5′ end) is a shift-and-mask operation.</p>
<p>Reverse complement is computed by bit manipulation directly on the packed word, without any lookup table: complement every base, reverse the byte order, then reverse the order of 2-bit groups within each byte in two more passes, and finally realign the result to the most-significant bits.</p> <p>Reverse complement is computed by bit manipulation directly on the packed word, without any lookup table: complement every base, reverse the byte order, then reverse the order of 2-bit groups within each byte in two more passes, and finally realign the result to the most-significant bits.</p>
<h2 id="canonical-form">Canonical form</h2> <h2 id="canonical-form">Canonical form</h2>
<p>The canonical form of a kmer is the lexicographic minimum of the kmer and its reverse complement:</p> <p>The canonical form of a packed 2-bit sequence is the lexicographic minimum of the sequence and its reverse complement:</p>
<div class="arithmatex">\[\text{canonical}(kmer) = \min\big(kmer,\ \text{revcomp}(kmer)\big)\]</div> <div class="arithmatex">\[\text{canonical}(x) = \min\big(x,\ \text{revcomp}(x)\big)\]</div>
<p>Using the canonical form halves the kmer space and makes counting strand-independent: a kmer and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.</p> <p>This is a single operation on the packed representation, so it applies identically regardless of what <code>x</code> represents — a kmer (see <a href="../kmers/">Kmers</a>), an m-mer (see <a href="../minimizer_selection/">Minimizer selection</a>), or a super-kmer (see <a href="../superkmers/">Super-kmers</a>). Using the canonical form halves the relevant space and makes counting/selection strand-independent: a sequence and its reverse complement are always treated as the same entity, regardless of which DNA strand was sequenced.</p>
@@ -1224,10 +1642,10 @@
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script> <script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script> <script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
@@ -9,16 +9,16 @@
<link rel="prev" href="../encoding/"> <link rel="prev" href="../indexing_architecture/">
<link rel="next" href="../minimizer_selection/"> <link rel="next" href="../../phylogeny/kmer_based/kmer_set_distances/">
<link rel="icon" href="../../assets/images/favicon.png"> <link rel="icon" href="../../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6"> <meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
@@ -27,7 +27,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css"> <link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
@@ -46,7 +46,7 @@
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script> <script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
@@ -80,7 +80,7 @@
<header class="md-header md-header--shadow" data-md-component="header"> <header class="md-header md-header--shadow" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header"> <nav class="md-header__inner md-grid" aria-label="Header">
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -138,7 +138,7 @@
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0"> <nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer"> <label class="md-nav__title" for="__drawer">
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -156,7 +156,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../.." class="md-nav__link"> <a href="../../.." class="md-nav__link">
@@ -183,7 +183,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../installation/" class="md-nav__link"> <a href="../../../installation/" class="md-nav__link">
@@ -258,15 +258,64 @@
<li class="md-nav__item">
<a href="../kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../encoding/" class="md-nav__link">
<span class="md-ellipsis">
DNA encoding
@@ -287,14 +336,98 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../encoding/" class="md-nav__link"> <a href="../kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
DNA encoding Kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../minimizer_selection/" class="md-nav__link">
<span class="md-ellipsis">
Minimizer selection
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../indexing_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Partitioning and indexing architecture
@@ -438,6 +571,108 @@
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -445,14 +680,97 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../minimizer_selection/" class="md-nav__link"> <a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Minimizer selection Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
@@ -473,14 +791,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../indexing_architecture/" class="md-nav__link"> <a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Partitioning and indexing architecture Gamma rate heterogeneity
@@ -494,6 +812,78 @@
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -555,7 +945,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/superkmer/" class="md-nav__link"> <a href="../../../usage/superkmer/" class="md-nav__link">
@@ -583,7 +973,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/index_command/" class="md-nav__link"> <a href="../../../usage/index_command/" class="md-nav__link">
@@ -611,7 +1001,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/merge/" class="md-nav__link"> <a href="../../../usage/merge/" class="md-nav__link">
@@ -639,7 +1029,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/filter/" class="md-nav__link"> <a href="../../../usage/filter/" class="md-nav__link">
@@ -667,7 +1057,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/select/" class="md-nav__link"> <a href="../../../usage/select/" class="md-nav__link">
@@ -695,7 +1085,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/query/" class="md-nav__link"> <a href="../../../usage/query/" class="md-nav__link">
@@ -723,7 +1113,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/dump/" class="md-nav__link"> <a href="../../../usage/dump/" class="md-nav__link">
@@ -751,7 +1141,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/annotate/" class="md-nav__link"> <a href="../../../usage/annotate/" class="md-nav__link">
@@ -779,7 +1169,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/phylo/" class="md-nav__link"> <a href="../../../usage/phylo/" class="md-nav__link">
@@ -807,7 +1197,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/unitig/" class="md-nav__link"> <a href="../../../usage/unitig/" class="md-nav__link">
@@ -835,7 +1225,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/estimate/" class="md-nav__link"> <a href="../../../usage/estimate/" class="md-nav__link">
@@ -863,7 +1253,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/convert/" class="md-nav__link"> <a href="../../../usage/convert/" class="md-nav__link">
@@ -891,7 +1281,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/utils/" class="md-nav__link"> <a href="../../../usage/utils/" class="md-nav__link">
@@ -919,7 +1309,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/pack/" class="md-nav__link"> <a href="../../../usage/pack/" class="md-nav__link">
@@ -947,7 +1337,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/predicates/" class="md-nav__link"> <a href="../../../usage/predicates/" class="md-nav__link">
@@ -1029,7 +1419,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../formats/index_layout/" class="md-nav__link"> <a href="../../../formats/index_layout/" class="md-nav__link">
@@ -1050,6 +1440,34 @@
<li class="md-nav__item">
<a href="../../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1064,7 +1482,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../architecture/" class="md-nav__link"> <a href="../../../architecture/" class="md-nav__link">
@@ -1252,10 +1670,10 @@
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script> <script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script> <script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
@@ -9,16 +9,16 @@
<link rel="prev" href="../minimizer_selection/"> <link rel="prev" href="../superkmers/">
<link rel="next" href="../../usage/superkmer/"> <link rel="next" href="../entropy_filter/">
<link rel="icon" href="../../assets/images/favicon.png"> <link rel="icon" href="../../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6"> <meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
@@ -27,7 +27,7 @@
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css"> <link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
@@ -46,7 +46,7 @@
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script> <script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
@@ -80,7 +80,7 @@
<header class="md-header md-header--shadow" data-md-component="header"> <header class="md-header md-header--shadow" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header"> <nav class="md-header__inner md-grid" aria-label="Header">
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -138,7 +138,7 @@
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0"> <nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer"> <label class="md-nav__title" for="__drawer">
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -156,7 +156,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../.." class="md-nav__link"> <a href="../../.." class="md-nav__link">
@@ -183,7 +183,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../installation/" class="md-nav__link"> <a href="../../../installation/" class="md-nav__link">
@@ -258,15 +258,28 @@
<li class="md-nav__item">
<a href="../kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -274,12 +287,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -315,14 +336,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../entropy_filter/" class="md-nav__link"> <a href="../kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -370,6 +391,34 @@
<li class="md-nav__item">
<a href="../superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item md-nav__item--active"> <li class="md-nav__item md-nav__item--active">
@@ -439,17 +488,6 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#why-hashing-is-necessary" class="md-nav__link">
<span class="md-ellipsis">
Why hashing is necessary
</span>
</a>
</li> </li>
<li class="md-nav__item"> <li class="md-nav__item">
@@ -472,6 +510,347 @@
<li class="md-nav__item">
<a href="../entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -533,7 +912,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/superkmer/" class="md-nav__link"> <a href="../../../usage/superkmer/" class="md-nav__link">
@@ -561,7 +940,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/index_command/" class="md-nav__link"> <a href="../../../usage/index_command/" class="md-nav__link">
@@ -589,7 +968,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/merge/" class="md-nav__link"> <a href="../../../usage/merge/" class="md-nav__link">
@@ -617,7 +996,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/filter/" class="md-nav__link"> <a href="../../../usage/filter/" class="md-nav__link">
@@ -645,7 +1024,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/select/" class="md-nav__link"> <a href="../../../usage/select/" class="md-nav__link">
@@ -673,7 +1052,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/query/" class="md-nav__link"> <a href="../../../usage/query/" class="md-nav__link">
@@ -701,7 +1080,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/dump/" class="md-nav__link"> <a href="../../../usage/dump/" class="md-nav__link">
@@ -729,7 +1108,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/annotate/" class="md-nav__link"> <a href="../../../usage/annotate/" class="md-nav__link">
@@ -757,7 +1136,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/phylo/" class="md-nav__link"> <a href="../../../usage/phylo/" class="md-nav__link">
@@ -785,7 +1164,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/unitig/" class="md-nav__link"> <a href="../../../usage/unitig/" class="md-nav__link">
@@ -813,7 +1192,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/estimate/" class="md-nav__link"> <a href="../../../usage/estimate/" class="md-nav__link">
@@ -841,7 +1220,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/convert/" class="md-nav__link"> <a href="../../../usage/convert/" class="md-nav__link">
@@ -869,7 +1248,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/utils/" class="md-nav__link"> <a href="../../../usage/utils/" class="md-nav__link">
@@ -897,7 +1276,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/pack/" class="md-nav__link"> <a href="../../../usage/pack/" class="md-nav__link">
@@ -925,7 +1304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/predicates/" class="md-nav__link"> <a href="../../../usage/predicates/" class="md-nav__link">
@@ -1007,7 +1386,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../formats/index_layout/" class="md-nav__link"> <a href="../../../formats/index_layout/" class="md-nav__link">
@@ -1028,6 +1407,34 @@
<li class="md-nav__item">
<a href="../../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1042,7 +1449,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../architecture/" class="md-nav__link"> <a href="../../../architecture/" class="md-nav__link">
@@ -1097,17 +1504,6 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#why-hashing-is-necessary" class="md-nav__link">
<span class="md-ellipsis">
Why hashing is necessary
</span>
</a>
</li> </li>
<li class="md-nav__item"> <li class="md-nav__item">
@@ -1141,15 +1537,13 @@
<h1 id="partitioning-and-indexing-architecture">Partitioning and indexing architecture</h1> <h1 id="partitioning-and-indexing-architecture">Partitioning and indexing architecture</h1>
<p>An index is split into a fixed number of <strong>partitions</strong>, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.</p> <p>An index is split into a fixed number of <strong>partitions</strong>, each handling an independent, disjoint slice of the kmer space. Partitioning keeps the working set of each stage small enough to process efficiently and enables parallel construction and querying.</p>
<h2 id="routing">Routing</h2> <h2 id="routing">Routing</h2>
<p>The canonical minimizer of a super-kmer (see <a href="../minimizer_selection/">Minimizer selection</a>) is hashed to produce a <span class="arithmatex">\(p\)</span>-bit routing value that selects the destination partition:</p> <p>The canonical minimizer of a super-kmer (see <a href="../minimizer_selection/">Minimizer selection</a>) is hashed to produce a <span class="arithmatex">\(p\)</span>-bit routing value that selects the destination partition — the low <span class="arithmatex">\(p\)</span> bits of the minimizer's hash:</p>
<div class="highlight"><pre><span></span><code>canonical minimizer → hash(minimizer) → p-bit value → partition index <div class="highlight"><pre><span></span><code>canonical minimizer → hash(minimizer) → p-bit value → partition index
</code></pre></div> </code></pre></div>
<p>The routing value is recomputed whenever it is needed (during construction and again at query time) rather than stored — it is not part of the on-disk super-kmer representation.</p> <div class="arithmatex">\[\text{partition} = H(\text{minimizer}) \bmod 2^p\]</div>
<p>Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see <a href="../../formats/index_layout/">On-disk storage</a>); the minimizer plays no further role once a super-kmer has reached its partition.</p> <p>Within a partition, kmers are indexed as plain values via a minimal perfect hash function (see <a href="../../../formats/index_layout/">On-disk storage</a>); the minimizer plays no further role once a super-kmer has reached its partition.</p>
<h2 id="why-hashing-is-necessary">Why hashing is necessary</h2>
<p>A canonical minimizer is an m-mer (<span class="arithmatex">\(m \in \{9, 11, 13, 15\}\)</span>), and its distribution over all possible m-mer values is not uniform — as the lexicographic minimum of a window, small values are systematically over-represented (Golan &amp; Shur 2025; Kille <em>et al.</em> 2023; Pan &amp; Reinert 2024; Zheng <em>et al.</em> 2020, 2021)<sup id="fnref:Zheng2020-ji"><a class="footnote-ref" href="#fn:Zheng2020-ji">1</a></sup> <sup id="fnref:Zheng2021-cc"><a class="footnote-ref" href="#fn:Zheng2021-cc">2</a></sup> <sup id="fnref:Pan2024-hb"><a class="footnote-ref" href="#fn:Pan2024-hb">3</a></sup> <sup id="fnref:Kille2023-px"><a class="footnote-ref" href="#fn:Kille2023-px">4</a></sup> <sup id="fnref:Golan2025-xf"><a class="footnote-ref" href="#fn:Golan2025-xf">5</a></sup>. Routing directly on the raw minimizer value would therefore produce badly unbalanced partitions.</p>
<p>Hashing the minimizer before routing redistributes this skewed distribution uniformly across partitions. This works reliably because the number of partition-index bits <span class="arithmatex">\(p\)</span> is chosen well below the number of bits available in the minimizer (<span class="arithmatex">\(2m\)</span>): even with strong bias in the minimizer distribution, the hash has enough entropy margin to absorb it, provided the number of distinct minimizers actually observed is much larger than the number of partitions.</p>
<h2 id="parameter-guidance">Parameter guidance</h2> <h2 id="parameter-guidance">Parameter guidance</h2>
<p>Even though <span class="arithmatex">\(H\)</span> already makes minimizer values well-distributed (see <a href="../minimizer_selection/">Minimizer selection</a>), choosing <span class="arithmatex">\(p\)</span> well below the number of bits available in the minimizer (<span class="arithmatex">\(2m\)</span>) leaves a comfortable entropy margin, provided the number of distinct minimizers actually observed is much larger than the number of partitions.</p>
<table> <table>
<thead> <thead>
<tr> <tr>
@@ -1187,26 +1581,6 @@
</tbody> </tbody>
</table> </table>
<p>The number of partitions must satisfy <span class="arithmatex">\(p \le 2m\)</span>, and in practice <span class="arithmatex">\(p\)</span> is chosen well below that bound to leave a comfortable entropy margin. For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span>, <span class="arithmatex">\(p=10\)</span> (1024 partitions), partition load is well balanced on real genomic data.</p> <p>The number of partitions must satisfy <span class="arithmatex">\(p \le 2m\)</span>, and in practice <span class="arithmatex">\(p\)</span> is chosen well below that bound to leave a comfortable entropy margin. For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span>, <span class="arithmatex">\(p=10\)</span> (1024 partitions), partition load is well balanced on real genomic data.</p>
<div class="footnote">
<hr />
<ol>
<li id="fn:Zheng2020-ji">
<p>Zheng, H., Kingsford, C. &amp; Marçais, G. (2020). <a href="https://doi.org/10.1093/bioinformatics/btaa472">Improved design and analysis of practical minimizers</a>. <em>Bioinformatics (Oxford, England)</em>, 36, i119--i127.&#160;<a class="footnote-backref" href="#fnref:Zheng2020-ji" title="Jump back to footnote 1 in the text">&#8617;</a></p>
</li>
<li id="fn:Zheng2021-cc">
<p>Zheng, H., Kingsford, C. &amp; Marçais, G. (2021). <a href="https://doi.org/10.1093/bioinformatics/btab313">Sequence-specific minimizers via polar sets</a>. <em>Bioinformatics (Oxford, England)</em>, 37, i187--i195.&#160;<a class="footnote-backref" href="#fnref:Zheng2021-cc" title="Jump back to footnote 2 in the text">&#8617;</a></p>
</li>
<li id="fn:Pan2024-hb">
<p>Pan, C. &amp; Reinert, K. (2024). <a href="https://doi.org/10.1093/bioinformatics/btae045">A simple refined DNA minimizer operator enables 2-fold faster computation</a>. <em>Bioinformatics (Oxford, England)</em>, 40.&#160;<a class="footnote-backref" href="#fnref:Pan2024-hb" title="Jump back to footnote 3 in the text">&#8617;</a></p>
</li>
<li id="fn:Kille2023-px">
<p>Kille, B., Garrison, E., Treangen, T.J. &amp; Phillippy, A.M. (2023). <a href="https://doi.org/10.1093/bioinformatics/btad512">Minmers are a generalization of minimizers that enable unbiased local jaccard estimation</a>. <em>Bioinformatics (Oxford, England)</em>, 39.&#160;<a class="footnote-backref" href="#fnref:Kille2023-px" title="Jump back to footnote 4 in the text">&#8617;</a></p>
</li>
<li id="fn:Golan2025-xf">
<p>Golan, S. &amp; Shur, A.M. (2025). <a href="https://doi.org/10.1007/978-3-031-82670-2\_25">Expected density of random minimizers</a>. In: <em>Lecture notes in computer science</em>, Lecture notes in computer science. Springer Nature Switzerland, Cham, pp. 347--360.&#160;<a class="footnote-backref" href="#fnref:Golan2025-xf" title="Jump back to footnote 5 in the text">&#8617;</a></p>
</li>
</ol>
</div>
@@ -1256,10 +1630,10 @@
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script> <script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script> <script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -9,25 +9,25 @@
<link rel="prev" href="../../installation/"> <link rel="prev" href="../minimizer_selection/">
<link rel="next" href="../encoding/"> <link rel="next" href="../indexing_architecture/">
<link rel="icon" href="../../assets/images/favicon.png"> <link rel="icon" href="../../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6"> <meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
<title>Kmers and super-kmers - obikmer — User Guide</title> <title>Super-kmers - obikmer — User Guide</title>
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css"> <link rel="stylesheet" href="../../../assets/stylesheets/main.484c7ddc.min.css">
@@ -46,7 +46,7 @@
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script> <script>__md_scope=new URL("../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
@@ -64,7 +64,7 @@
<div data-md-component="skip"> <div data-md-component="skip">
<a href="#kmers-and-super-kmers" class="md-skip"> <a href="#super-kmers" class="md-skip">
Skip to content Skip to content
</a> </a>
@@ -80,7 +80,7 @@
<header class="md-header md-header--shadow" data-md-component="header"> <header class="md-header md-header--shadow" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header"> <nav class="md-header__inner md-grid" aria-label="Header">
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -100,7 +100,7 @@
<div class="md-header__topic" data-md-component="header-topic"> <div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Super-kmers
</span> </span>
</div> </div>
@@ -138,7 +138,7 @@
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0"> <nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer"> <label class="md-nav__title" for="__drawer">
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -156,7 +156,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../.." class="md-nav__link"> <a href="../../.." class="md-nav__link">
@@ -183,7 +183,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../installation/" class="md-nav__link"> <a href="../../../installation/" class="md-nav__link">
@@ -260,22 +260,26 @@
<li class="md-nav__item md-nav__item--active">
<input class="md-nav__toggle md-toggle" type="checkbox" id="__toc">
<label class="md-nav__link md-nav__link--active" for="__toc"> <li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" checked>
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -283,88 +287,19 @@
<span class="md-nav__icon md-icon"></span> <span class="md-nav__icon md-icon"></span>
</label> </label>
<a href="./" class="md-nav__link md-nav__link--active"> <nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
<span class="md-ellipsis">
Kmers and super-kmers
</span>
</a>
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#kmers" class="md-nav__link">
<span class="md-ellipsis">
Kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#super-kmers" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
<nav class="md-nav" aria-label="Super-kmers">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#canonical-super-kmers" class="md-nav__link">
<span class="md-ellipsis">
Canonical super-kmers
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -401,14 +336,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../entropy_filter/" class="md-nav__link"> <a href="../kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -456,6 +391,92 @@
<li class="md-nav__item md-nav__item--active">
<input class="md-nav__toggle md-toggle" type="checkbox" id="__toc">
<label class="md-nav__link md-nav__link--active" for="__toc">
<span class="md-ellipsis">
Super-kmers
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<a href="./" class="md-nav__link md-nav__link--active">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item">
<a href="#canonical-super-kmers" class="md-nav__link">
<span class="md-ellipsis">
Canonical super-kmers
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../indexing_architecture/" class="md-nav__link"> <a href="../indexing_architecture/" class="md-nav__link">
@@ -478,6 +499,347 @@
<li class="md-nav__item">
<a href="../entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -539,7 +901,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/superkmer/" class="md-nav__link"> <a href="../../../usage/superkmer/" class="md-nav__link">
@@ -567,7 +929,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/index_command/" class="md-nav__link"> <a href="../../../usage/index_command/" class="md-nav__link">
@@ -595,7 +957,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/merge/" class="md-nav__link"> <a href="../../../usage/merge/" class="md-nav__link">
@@ -623,7 +985,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/filter/" class="md-nav__link"> <a href="../../../usage/filter/" class="md-nav__link">
@@ -651,7 +1013,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/select/" class="md-nav__link"> <a href="../../../usage/select/" class="md-nav__link">
@@ -679,7 +1041,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/query/" class="md-nav__link"> <a href="../../../usage/query/" class="md-nav__link">
@@ -707,7 +1069,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/dump/" class="md-nav__link"> <a href="../../../usage/dump/" class="md-nav__link">
@@ -735,7 +1097,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/annotate/" class="md-nav__link"> <a href="../../../usage/annotate/" class="md-nav__link">
@@ -763,7 +1125,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/phylo/" class="md-nav__link"> <a href="../../../usage/phylo/" class="md-nav__link">
@@ -791,7 +1153,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/unitig/" class="md-nav__link"> <a href="../../../usage/unitig/" class="md-nav__link">
@@ -819,7 +1181,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/estimate/" class="md-nav__link"> <a href="../../../usage/estimate/" class="md-nav__link">
@@ -847,7 +1209,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/convert/" class="md-nav__link"> <a href="../../../usage/convert/" class="md-nav__link">
@@ -875,7 +1237,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/utils/" class="md-nav__link"> <a href="../../../usage/utils/" class="md-nav__link">
@@ -903,7 +1265,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/pack/" class="md-nav__link"> <a href="../../../usage/pack/" class="md-nav__link">
@@ -931,7 +1293,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/predicates/" class="md-nav__link"> <a href="../../../usage/predicates/" class="md-nav__link">
@@ -1013,7 +1375,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../formats/index_layout/" class="md-nav__link"> <a href="../../../formats/index_layout/" class="md-nav__link">
@@ -1034,6 +1396,34 @@
<li class="md-nav__item">
<a href="../../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1048,7 +1438,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../architecture/" class="md-nav__link"> <a href="../../../architecture/" class="md-nav__link">
@@ -1095,29 +1485,6 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix> <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item"> <li class="md-nav__item">
<a href="#kmers" class="md-nav__link">
<span class="md-ellipsis">
Kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#super-kmers" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
<nav class="md-nav" aria-label="Super-kmers">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#canonical-super-kmers" class="md-nav__link"> <a href="#canonical-super-kmers" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
@@ -1126,11 +1493,6 @@
</span> </span>
</a> </a>
</li>
</ul>
</nav>
</li> </li>
</ul> </ul>
@@ -1150,29 +1512,22 @@
<h1 id="kmers-and-super-kmers">Kmers and super-kmers</h1> <h1 id="super-kmers">Super-kmers</h1>
<h2 id="kmers">Kmers</h2>
<p>A <strong>kmer</strong> is a DNA subsequence of fixed length <span class="arithmatex">\(k\)</span>. Two constraints apply to <span class="arithmatex">\(k\)</span>, both enforced when a command starts (an invalid value exits immediately with an error):</p>
<ul>
<li><span class="arithmatex">\(k \in [11, 31]\)</span>: long enough to be specific, short enough to fit in a single 64-bit word at 2 bits/base (<span class="arithmatex">\(k \le 32\)</span> is the hard limit; <span class="arithmatex">\(k &lt; 11\)</span> gives insufficient specificity).</li>
<li><span class="arithmatex">\(k\)</span> <strong>is odd</strong>: an odd-length sequence can never equal its own reverse complement, so the two orientations of any kmer are always distinct. This is required for the canonical form (see <a href="../encoding/">DNA encoding</a>) to be well defined.</li>
</ul>
<h2 id="super-kmers">Super-kmers</h2>
<p>A <strong>super-kmer</strong> is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see <a href="../minimizer_selection/">Minimizer selection</a>). Each kmer in the run overlaps the next by <span class="arithmatex">\(k-1\)</span> nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.</p> <p>A <strong>super-kmer</strong> is a maximal run of consecutive, overlapping kmers from a read that share the same canonical minimizer (see <a href="../minimizer_selection/">Minimizer selection</a>). Each kmer in the run overlaps the next by <span class="arithmatex">\(k-1\)</span> nucleotides. A super-kmer is capped at 256 nucleotides; a longer run is split at that boundary.</p>
<p>For a random minimizer of length <span class="arithmatex">\(m\)</span> over kmers of length <span class="arithmatex">\(k\)</span>, the expected length of a super-kmer is approximately (Golan &amp; Shur 2025; Zheng <em>et al.</em> 2020)<sup id="fnref:Zheng2020-ji"><a class="footnote-ref" href="#fn:Zheng2020-ji">1</a></sup> <sup id="fnref:Golan2025-xf"><a class="footnote-ref" href="#fn:Golan2025-xf">2</a></sup>:</p> <p>For a random minimizer of length <span class="arithmatex">\(m\)</span> over kmers of length <span class="arithmatex">\(k\)</span>, the expected length of a super-kmer is approximately (Zheng <em>et al.</em> 2020; Golan &amp; Shur 2025)<sup id="fnref:Zheng2020-ji"><a class="footnote-ref" href="#fn:Zheng2020-ji">1</a></sup> <sup id="fnref:Golan2025-xf"><a class="footnote-ref" href="#fn:Golan2025-xf">2</a></sup>:</p>
<div class="arithmatex">\[L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1\]</div> <div class="arithmatex">\[L_{\text{nt}} \approx \frac{k-m+2}{2} + k - 1\]</div>
<p>For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span> this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.</p> <p>For <span class="arithmatex">\(k=31\)</span>, <span class="arithmatex">\(m=13\)</span> this is about 40 nucleotides; in practice super-kmers rarely exceed a few dozen nucleotides.</p>
<h3 id="canonical-super-kmers">Canonical super-kmers</h3> <h2 id="canonical-super-kmers">Canonical super-kmers</h2>
<p>A <strong>canonical super-kmer</strong> is the lexicographic minimum of a super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.</p> <p>A <strong>canonical super-kmer</strong> is the canonical form of a super-kmer (see <a href="../encoding/">DNA encoding</a>): the lexicographic minimum of the super-kmer and its reverse complement. When a read and its reverse complement are both encountered, they produce super-kmers that are reverse complements of each other; both reduce to the same canonical super-kmer, so a genomic region is represented once regardless of which strand was read.</p>
<p>Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.</p> <p>Super-kmers are the unit of work used throughout construction and querying: sequences are decomposed into super-kmers first, and every downstream step (partition routing, deduplication, counting) operates on them rather than on individual kmers.</p>
<div class="footnote"> <div class="footnote">
<hr /> <hr />
<ol> <ol>
<li id="fn:Zheng2020-ji"> <li id="fn:Zheng2020-ji">
<p>Zheng, H., Kingsford, C. &amp; Marçais, G. (2020). <a href="https://doi.org/10.1093/bioinformatics/btaa472">Improved design and analysis of practical minimizers</a>. <em>Bioinformatics (Oxford, England)</em>, 36, i119--i127.&#160;<a class="footnote-backref" href="#fnref:Zheng2020-ji" title="Jump back to footnote 1 in the text">&#8617;</a></p> <p>Zheng H, Kingsford C, Marçais G (2020) <a href="https://doi.org/10.1093/bioinformatics/btaa472">Improved design and analysis of practical minimizers</a>. <em>Bioinformatics (Oxford, England)</em>, <strong>36</strong>, i119--i127.&#160;<a class="footnote-backref" href="#fnref:Zheng2020-ji" title="Jump back to footnote 1 in the text">&#8617;</a></p>
</li> </li>
<li id="fn:Golan2025-xf"> <li id="fn:Golan2025-xf">
<p>Golan, S. &amp; Shur, A.M. (2025). <a href="https://doi.org/10.1007/978-3-031-82670-2\_25">Expected density of random minimizers</a>. In: <em>Lecture notes in computer science</em>, Lecture notes in computer science. Springer Nature Switzerland, Cham, pp. 347--360.&#160;<a class="footnote-backref" href="#fnref:Golan2025-xf" title="Jump back to footnote 2 in the text">&#8617;</a></p> <p>Golan S, Shur AM (2025) <a href="https://doi.org/10.1007/978-3-031-82670-2\_25">Expected density of random minimizers</a>. In: <em>Lecture notes in computer science</em> Lecture notes in computer science., pp. 347--360. Springer Nature Switzerland, Cham.&#160;<a class="footnote-backref" href="#fnref:Golan2025-xf" title="Jump back to footnote 2 in the text">&#8617;</a></p>
</li> </li>
</ol> </ol>
</div> </div>
@@ -1225,10 +1580,10 @@
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script id="__config" type="application/json">{"annotate": null, "base": "../../..", "features": [], "search": "../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script> <script src="../../../assets/javascripts/bundle.79ae519e.min.js"></script>
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script> <script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
@@ -9,25 +9,25 @@
<link rel="prev" href="../entropy_filter/"> <link rel="prev" href="../gamma_rate_heterogeneity/">
<link rel="next" href="../indexing_architecture/"> <link rel="next" href="../sampling_design/">
<link rel="icon" href="../../assets/images/favicon.png"> <link rel="icon" href="../../../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6"> <meta name="generator" content="mkdocs-1.6.1, mkdocs-material-9.7.6">
<title>Minimizer selection - obikmer — User Guide</title> <title>Sankoff model - obikmer — User Guide</title>
<link rel="stylesheet" href="../../assets/stylesheets/main.484c7ddc.min.css"> <link rel="stylesheet" href="../../../../assets/stylesheets/main.484c7ddc.min.css">
@@ -46,7 +46,7 @@
<script>__md_scope=new URL("../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script> <script>__md_scope=new URL("../../../..",location),__md_hash=e=>[...e].reduce(((e,_)=>(e<<5)-e+_.charCodeAt(0)),0),__md_get=(e,_=localStorage,t=__md_scope)=>JSON.parse(_.getItem(t.pathname+"."+e)),__md_set=(e,_,t=localStorage,a=__md_scope)=>{try{t.setItem(a.pathname+"."+e,JSON.stringify(_))}catch(e){}}</script>
@@ -64,7 +64,7 @@
<div data-md-component="skip"> <div data-md-component="skip">
<a href="#minimizer-selection" class="md-skip"> <a href="#the-16-state-sankoff-model" class="md-skip">
Skip to content Skip to content
</a> </a>
@@ -80,7 +80,7 @@
<header class="md-header md-header--shadow" data-md-component="header"> <header class="md-header md-header--shadow" data-md-component="header">
<nav class="md-header__inner md-grid" aria-label="Header"> <nav class="md-header__inner md-grid" aria-label="Header">
<a href="../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../../.." title="obikmer — User Guide" class="md-header__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -100,7 +100,7 @@
<div class="md-header__topic" data-md-component="header-topic"> <div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis"> <span class="md-ellipsis">
Minimizer selection Sankoff model
</span> </span>
</div> </div>
@@ -138,7 +138,7 @@
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0"> <nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer"> <label class="md-nav__title" for="__drawer">
<a href="../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo"> <a href="../../../.." title="obikmer — User Guide" class="md-nav__button md-logo" aria-label="obikmer — User Guide" data-md-component="logo">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M12 8a3 3 0 0 0 3-3 3 3 0 0 0-3-3 3 3 0 0 0-3 3 3 3 0 0 0 3 3m0 3.54C9.64 9.35 6.5 8 3 8v11c3.5 0 6.64 1.35 9 3.54 2.36-2.19 5.5-3.54 9-3.54V8c-3.5 0-6.64 1.35-9 3.54"/></svg>
@@ -156,7 +156,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../.." class="md-nav__link"> <a href="../../../.." class="md-nav__link">
@@ -183,7 +183,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../installation/" class="md-nav__link"> <a href="../../../../installation/" class="md-nav__link">
@@ -258,15 +258,26 @@
<li class="md-nav__item">
<a href="../kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -274,12 +285,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -287,7 +306,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../encoding/" class="md-nav__link"> <a href="../../../kmer_indexing/encoding/" class="md-nav__link">
@@ -315,7 +334,119 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../entropy_filter/" class="md-nav__link"> <a href="../../../kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis">
Kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../kmer_indexing/minimizer_selection/" class="md-nav__link">
<span class="md-ellipsis">
Minimizer selection
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../kmer_indexing/indexing_architecture/" class="md-nav__link">
<span class="md-ellipsis">
Partitioning and indexing architecture
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../../kmer_indexing/entropy_filter/" class="md-nav__link">
@@ -336,6 +467,251 @@
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" checked>
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" checked>
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="true">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
@@ -359,7 +735,7 @@
<span class="md-ellipsis"> <span class="md-ellipsis">
Minimizer selection Sankoff model
@@ -377,7 +753,7 @@
<span class="md-ellipsis"> <span class="md-ellipsis">
Minimizer selection Sankoff model
@@ -403,49 +779,10 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix> <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item"> <li class="md-nav__item">
<a href="#definition" class="md-nav__link"> <a href="#combining-into-a-1616-cost-matrix" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Definition Combining into a 16×16 cost matrix
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#hash-based-random-minimizer" class="md-nav__link">
<span class="md-ellipsis">
Hash-based ("random") minimizer
</span>
</a>
<nav class="md-nav" aria-label="Hash-based (&#34;random&#34;) minimizer">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#hash-function" class="md-nav__link">
<span class="md-ellipsis">
Hash function
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#partition-routing-is-independent-of-minimizer-selection" class="md-nav__link">
<span class="md-ellipsis">
Partition routing is independent of minimizer selection
</span> </span>
</a> </a>
@@ -468,14 +805,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../indexing_architecture/" class="md-nav__link"> <a href="../sampling_design/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Partitioning and indexing architecture Sampling design
@@ -497,6 +834,22 @@
</ul>
</nav>
</li>
</ul>
</nav>
</li>
@@ -550,7 +903,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/superkmer/" class="md-nav__link"> <a href="../../../../usage/superkmer/" class="md-nav__link">
@@ -578,7 +931,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/index_command/" class="md-nav__link"> <a href="../../../../usage/index_command/" class="md-nav__link">
@@ -606,7 +959,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/merge/" class="md-nav__link"> <a href="../../../../usage/merge/" class="md-nav__link">
@@ -634,7 +987,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/filter/" class="md-nav__link"> <a href="../../../../usage/filter/" class="md-nav__link">
@@ -662,7 +1015,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/select/" class="md-nav__link"> <a href="../../../../usage/select/" class="md-nav__link">
@@ -690,7 +1043,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/query/" class="md-nav__link"> <a href="../../../../usage/query/" class="md-nav__link">
@@ -718,7 +1071,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/dump/" class="md-nav__link"> <a href="../../../../usage/dump/" class="md-nav__link">
@@ -746,7 +1099,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/annotate/" class="md-nav__link"> <a href="../../../../usage/annotate/" class="md-nav__link">
@@ -774,7 +1127,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/phylo/" class="md-nav__link"> <a href="../../../../usage/phylo/" class="md-nav__link">
@@ -802,7 +1155,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/unitig/" class="md-nav__link"> <a href="../../../../usage/unitig/" class="md-nav__link">
@@ -830,7 +1183,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/estimate/" class="md-nav__link"> <a href="../../../../usage/estimate/" class="md-nav__link">
@@ -858,7 +1211,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/convert/" class="md-nav__link"> <a href="../../../../usage/convert/" class="md-nav__link">
@@ -886,7 +1239,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/utils/" class="md-nav__link"> <a href="../../../../usage/utils/" class="md-nav__link">
@@ -914,7 +1267,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/pack/" class="md-nav__link"> <a href="../../../../usage/pack/" class="md-nav__link">
@@ -942,7 +1295,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../usage/predicates/" class="md-nav__link"> <a href="../../../../usage/predicates/" class="md-nav__link">
@@ -1024,7 +1377,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../formats/index_layout/" class="md-nav__link"> <a href="../../../../formats/index_layout/" class="md-nav__link">
@@ -1045,6 +1398,34 @@
<li class="md-nav__item">
<a href="../../../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1059,7 +1440,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../architecture/" class="md-nav__link"> <a href="../../../../architecture/" class="md-nav__link">
@@ -1106,49 +1487,10 @@
<ul class="md-nav__list" data-md-component="toc" data-md-scrollfix> <ul class="md-nav__list" data-md-component="toc" data-md-scrollfix>
<li class="md-nav__item"> <li class="md-nav__item">
<a href="#definition" class="md-nav__link"> <a href="#combining-into-a-1616-cost-matrix" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Definition Combining into a 16×16 cost matrix
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#hash-based-random-minimizer" class="md-nav__link">
<span class="md-ellipsis">
Hash-based ("random") minimizer
</span>
</a>
<nav class="md-nav" aria-label="Hash-based (&#34;random&#34;) minimizer">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#hash-function" class="md-nav__link">
<span class="md-ellipsis">
Hash function
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#partition-routing-is-independent-of-minimizer-selection" class="md-nav__link">
<span class="md-ellipsis">
Partition routing is independent of minimizer selection
</span> </span>
</a> </a>
@@ -1172,33 +1514,20 @@
<h1 id="minimizer-selection">Minimizer selection</h1> <h1 id="the-16-state-sankoff-model">The 16-state Sankoff model</h1>
<h2 id="definition">Definition</h2> <p>Each family (see <a href="../central_snp_model/">Central-position SNP model</a>) is treated as a character with 16 possible states: one per subset of the 4 possible central bases, including the empty subset (no observed form). A genome pair's calibration reduces to two separately-estimated first-order Markov models, both restricted to genome pairs below a configurable SNP-ratio ceiling (excluding saturated pairs from skewing the calibration):</p>
<p>A <strong>minimizer</strong> of a kmer window is the m-mer (<span class="arithmatex">\(m &lt; k\)</span>) that is smallest, among all <span class="arithmatex">\(k - m + 1\)</span> overlapping m-mers in the window, under a chosen ordering. The minimizer is always taken in canonical form (lexicographic minimum of forward and reverse complement) so that selection is strand-independent.</p>
<p>The minimizer partitions a sequence into super-kmers: maximal runs of overlapping kmers that share the same minimizer (see <a href="../kmers_and_superkmers/">Kmers and super-kmers</a>).</p>
<h2 id="hash-based-random-minimizer">Hash-based ("random") minimizer</h2>
<p><code>obikmer</code> selects minimizers by hash order rather than plain lexicographic order. Ordering m-mers lexicographically on their 2-bit encoding systematically favors AT-rich m-mers (an all-A m-mer always encodes to 0), which causes low-complexity regions to dominate as minimizers and produces unbalanced partitions.</p>
<p>Instead, a well-distributed hash function <span class="arithmatex">\(H\)</span> is applied to the canonical (lexicographically minimal) form of each m-mer, and the m-mer with the smallest <span class="arithmatex">\(H\)</span> value wins. Because <span class="arithmatex">\(H\)</span> is a bijection with good avalanche properties, every distinct m-mer in a window has an equal chance of holding the minimum hash value, independent of its nucleotide composition.</p>
<p>The canonical form used as input to <span class="arithmatex">\(H\)</span> is still the lexicographic minimum of forward/reverse-complement — hashing is applied on top of it, not used to redefine it. Defining canonicity by hash value instead would bias the <em>distribution of hash values themselves</em> toward small values (the minimum of two independent hashes is not uniformly distributed), reintroducing a bias one layer down.</p>
<h3 id="hash-function">Hash function</h3>
<p>The hash function is a 64-bit mixing function (splitmix64-style finalizer) applied to the m-mer XORed with a fixed non-zero seed:</p>
<div class="arithmatex">\[H(x) = \text{mix64}(x \oplus s), \quad s = \lfloor 2^{64}/\varphi \rfloor = \texttt{0x9e3779b97f4a7c15}\]</div>
<div class="highlight"><pre><span></span><code>H(x):
x ← x ⊕ 0x9e3779b97f4a7c15
x ← x ⊕ (x &gt;&gt; 30)
x ← x × 0xbf58476d1ce4e5b9
x ← x ⊕ (x &gt;&gt; 27)
x ← x × 0x94d049bb133111eb
return x ⊕ (x &gt;&gt; 31)
</code></pre></div>
<p>The XOR seed avoids the finalizer's fixed point at 0 (<span class="arithmatex">\(\text{mix64}(0) = 0\)</span>), which would otherwise make an all-A m-mer (canonical value 0) win every window comparison.</p>
<h2 id="partition-routing-is-independent-of-minimizer-selection">Partition routing is independent of minimizer selection</h2>
<p>The hash used to select a minimizer within a window (the minimum of several hash values) and the hash used to route a super-kmer to a storage partition are computed separately:</p>
<ul> <ul>
<li><strong>Selection</strong> uses <span class="arithmatex">\(H\)</span> applied to every candidate m-mer in the window, keeping the minimum.</li> <li><strong>Cardinality transitions</strong> (<span class="arithmatex">\(5 \times 5\)</span>, row-stochastic): the probability that a family observed with <span class="arithmatex">\(i\)</span> forms in one genome (<span class="arithmatex">\(i \in \{0,\dots,4\}\)</span>) is observed with <span class="arithmatex">\(j\)</span> forms in the other.</li>
<li><strong>Partition routing</strong> recomputes <span class="arithmatex">\(H\)</span> on the single selected minimizer only, once its position is fixed. This is a hash of one specific value, not the minimum of several, so it is uniformly distributed and safe to use directly for routing.</li> <li><strong>Composition transitions</strong> (<span class="arithmatex">\(4 \times 4\)</span>, row-stochastic): the probability that an unambiguous, single-copy base observed as one of A/C/G/T in one genome is observed as another base in the other genome.</li>
</ul> </ul>
<p>See <a href="../indexing_architecture/">Partitioning and indexing architecture</a> for how the routing value is turned into a partition index.</p> <h2 id="combining-into-a-1616-cost-matrix">Combining into a 16×16 cost matrix</h2>
<p>The two matrices are <em>not</em> combined as a plain tensor product. For a pair of states <span class="arithmatex">\((A, B)\)</span> — each a subset of <span class="arithmatex">\(\{A,C,G,T\}\)</span> — let <code>shared</code> <span class="arithmatex">\(= A \cap B\)</span>, <code>lost</code> <span class="arithmatex">\(= A \setminus B\)</span>, <code>gained</code> <span class="arithmatex">\(= B \setminus A\)</span>. The log-probability of the transition <span class="arithmatex">\(A \to B\)</span> is:</p>
<div class="arithmatex">\[\log P(A,B) = \underbrace{\log P_{\text{card}}(|A|,|B|)}_{\text{cardinality term, omitted when loss events are treated as free}} \;+\; \underbrace{\sum_{i \,\in\, \text{shared}} \log P_{\text{comp}}(i,i)}_{\text{bases conserved on both sides}} \;-\; \underbrace{\text{best\_pairing\_cost}(\text{lost}, \text{gained})}_{\text{substitutions among the differing bases}}\]</div>
<p><code>best_pairing_cost</code> enumerates every injection pairing elements of <code>lost</code> with elements of <code>gained</code> and keeps the one minimizing <span class="arithmatex">\(-\sum \log P_{\text{comp}}\)</span> over the pairs — the discrete analogue of "prefer a substitution over an independent loss and gain": if a base is lost from one side and a different base is gained on the other, that is scored as a single substitution between them (cheaper, under any sane calibration, than treating them as two unrelated cardinality-changing events) whenever such a pairing is possible.</p>
<p>The resulting <span class="arithmatex">\(16 \times 16\)</span> log-probability matrix is row-normalized in log-space (log-sum-exp, not a direct <code>exp()</code>/divide, for numerical stability), giving a genuine row-stochastic transition matrix <span class="arithmatex">\(P\)</span>. The cost matrix is then <span class="arithmatex">\(\text{cost}(A,B) = -\ln P(A,B)\)</span>, symmetrized by simple averaging:</p>
<div class="arithmatex">\[\text{sym}(A,B) = \frac{\text{cost}(A,B) + \text{cost}(B,A)}{2}\]</div>
<p>This calibrated cost matrix is what every downstream export (for TNT, PhyG, IQ-TREE) reuses, recoded into each tool's own format.</p>
<p>See <a href="../../../../usage/phylo/#sankoff-calibration-and-phylogenetic-exports"><code>phylo</code></a> for how to run the calibration and produce these exports.</p>
@@ -1248,10 +1577,10 @@
<script id="__config" type="application/json">{"annotate": null, "base": "../..", "features": [], "search": "../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script> <script id="__config" type="application/json">{"annotate": null, "base": "../../../..", "features": [], "search": "../../../../assets/javascripts/workers/search.2c215733.min.js", "tags": null, "translations": {"clipboard.copied": "Copied to clipboard", "clipboard.copy": "Copy to clipboard", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.placeholder": "Type to start searching", "search.result.term.missing": "Missing", "select.version": "Select version"}, "version": null}</script>
<script src="../../assets/javascripts/bundle.79ae519e.min.js"></script> <script src="../../../../assets/javascripts/bundle.79ae519e.min.js"></script>
<script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script> <script src="https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js"></script>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1017,6 +1405,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1017,6 +1405,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1017,6 +1405,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1006,6 +1394,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1028,6 +1416,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+452 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -549,6 +937,17 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#which-files-are-read" class="md-nav__link">
<span class="md-ellipsis">
Which files are read
</span>
</a>
</li> </li>
<li class="md-nav__item"> <li class="md-nav__item">
@@ -1028,6 +1427,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1097,6 +1524,17 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#which-files-are-read" class="md-nav__link">
<span class="md-ellipsis">
Which files are read
</span>
</a>
</li> </li>
<li class="md-nav__item"> <li class="md-nav__item">
@@ -1157,6 +1595,10 @@
</tr> </tr>
</tbody> </tbody>
</table> </table>
<h2 id="which-files-are-read">Which files are read</h2>
<p>A file named explicitly on the command line is always read, whatever its extension: the format (FASTA, FASTQ or GenBank) and the compression (gzip, bzip2, xz or zstd) are detected from the content. If the content is not a sequence file, the command reports an error rather than skipping the file.</p>
<p>When an input is a directory, the files inside it are selected by extension: <code>.fasta</code>, <code>.fa</code>, <code>.fna</code> (the extension of NCBI genome downloads), <code>.ffn</code>, <code>.fas</code>, <code>.fsa</code>, <code>.fastq</code>, <code>.fq</code>, <code>.gbff</code>, <code>.gbk</code> and <code>.gb</code>, each optionally followed by <code>.gz</code>.</p>
<p>A small input spread over many partitions leaves most partitions empty. This is valid: the index is built normally and empty partitions simply hold no k-mers.</p>
<h2 id="options">Options</h2> <h2 id="options">Options</h2>
<table> <table>
<thead> <thead>
+481 -13
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -599,6 +987,28 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#resource-usage" class="md-nav__link">
<span class="md-ellipsis">
Resource usage
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#merging-an-index-that-was-already-merged" class="md-nav__link">
<span class="md-ellipsis">
Merging an index that was already merged
</span>
</a>
</li> </li>
</ul> </ul>
@@ -1028,6 +1438,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1119,6 +1557,28 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#resource-usage" class="md-nav__link">
<span class="md-ellipsis">
Resource usage
</span>
</a>
</li>
<li class="md-nav__item">
<a href="#merging-an-index-that-was-already-merged" class="md-nav__link">
<span class="md-ellipsis">
Merging an index that was already merged
</span>
</a>
</li> </li>
</ul> </ul>
@@ -1188,15 +1648,23 @@
<td>Disambiguate duplicate genome labels (<code>.1</code>, <code>.2</code>, …) instead of failing</td> <td>Disambiguate duplicate genome labels (<code>.1</code>, <code>.2</code>, …) instead of failing</td>
</tr> </tr>
<tr> <tr>
<td><code>--budget-fraction</code></td> <td><code>--dense</code></td>
<td><code>0.5</code></td> <td>off</td>
<td>Fraction of available RAM reserved as the memory budget for parallel partition merging</td> <td>Store the merged index's presence/absence matrices in the dense packed format instead of the default sparse one (see <a href="../pack/">pack</a>)</td>
</tr> </tr>
</tbody> </tbody>
</table> </table>
<h2 id="behaviour">Behaviour</h2> <h2 id="behaviour">Behaviour</h2>
<p>The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. <code>--force-presence</code> forces presence/absence regardless of the sources.</p> <p>The output mode is chosen automatically: if every source index stores counts, the merged index stores counts too; otherwise it is presence/absence. <code>--force-presence</code> forces presence/absence regardless of the sources.</p>
<p>By default, merging two indexes that share a genome label fails with an error; <code>--rename-duplicates</code> instead appends a numeric suffix to keep both copies.</p> <p>By default, merging two indexes that share a genome label fails with an error; <code>--rename-duplicates</code> instead appends a numeric suffix to keep both copies.</p>
<p>The merged index is written in the sparse packed format by default, because it is usually several times smaller on disk; <code>--dense</code> keeps the dense format, which is faster when whole genome columns are read at once, for example for distance matrices. The <a href="../pack/">pack</a> page explains the trade-off between the two formats.</p>
<h2 id="resource-usage">Resource usage</h2>
<p>The merge works partition by partition, and processes several partitions in parallel. Within a partition, the source indexes are read two at a time rather than all at once. The memory and the number of files mapped into memory for one partition therefore depend on how many k-mers the partition holds, not on how many indexes are being merged: merging hundreds of genomes maps about as many files per partition as merging three. The partitions do grow as more diverse genomes are added, so the total memory still rises with the size of the merged collection.</p>
<p>Each genome column being filled is held in memory and written to disk in one go once it is complete. A full disk or an exhausted quota is therefore reported as an ordinary error rather than crashing the program. The largest memory cost comes from the k-mers that are new to the merged index, which are assembled in memory, partition by partition, before being written out.</p>
<h2 id="merging-an-index-that-was-already-merged">Merging an index that was already merged</h2>
<p>An index produced by <code>merge</code> can be used as one of the sources of a later merge, which is how a collection is extended without starting again from the original genomes. There is one rule: among the sources, at least one must not have been through a merge. A merged index is stored in a compact packed format (see <a href="../pack/">pack</a>) that is read very efficiently but cannot be extended in place. The merge therefore takes a freshly built index as its starting point and brings the packed index in as a source. A typical case is a merged index of many genomes plus one or a few newly built single-genome indexes, and this works.</p>
<p>If every source has already been merged or packed, the command stops before doing any work and says so. In that case, merge the original single-genome indexes together instead.</p>
<p>The genome columns of the result follow the order of the starting index first, then the other sources, so they may not come out in the order the sources were listed. The genome labels always identify each column unambiguously.</p>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1028,6 +1416,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+947 -277
View File
File diff suppressed because it is too large. Load diff
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1045,6 +1433,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1028,6 +1416,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+470 -20
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -655,6 +1043,17 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#disk-usage" class="md-nav__link">
<span class="md-ellipsis">
Disk usage
</span>
</a>
</li> </li>
</ul> </ul>
@@ -1028,6 +1427,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
@@ -1119,6 +1546,17 @@
</span> </span>
</a> </a>
</li>
<li class="md-nav__item">
<a href="#disk-usage" class="md-nav__link">
<span class="md-ellipsis">
Disk usage
</span>
</a>
</li> </li>
</ul> </ul>
@@ -1139,8 +1577,8 @@
<h1 id="select">select</h1> <h1 id="select">select</h1>
<p>Project and/or aggregate the genome columns of an index into a new (or in-place) index. Where <a href="../filter/"><code>filter</code></a> selects rows (kmers), <code>select</code> operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.</p> <p>Project and/or aggregate the genome columns of an index into a new index. Where <a href="../filter/"><code>filter</code></a> selects rows (kmers), <code>select</code> operates on columns (genomes): grouping several genomes into one aggregated column, reordering columns, or dropping some.</p>
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>SOURCE<span class="w"> </span><span class="o">(</span>--output<span class="w"> </span>OUTPUT<span class="w"> </span><span class="p">|</span><span class="w"> </span>--in-place<span class="o">)</span><span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span> <div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>SOURCE<span class="w"> </span>--output<span class="w"> </span>OUTPUT<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
</code></pre></div> </code></pre></div>
<h2 id="arguments">Arguments</h2> <h2 id="arguments">Arguments</h2>
<table> <table>
@@ -1168,14 +1606,9 @@
</thead> </thead>
<tbody> <tbody>
<tr> <tr>
<td><code>--output</code></td> <td><code>-o, --output</code></td>
<td>—</td> <td>—</td>
<td>Output index directory (mutually exclusive with <code>--in-place</code>)</td> <td>Output index directory (required)</td>
</tr>
<tr>
<td><code>--in-place</code></td>
<td>off</td>
<td>Rewrite the source index in place (mutually exclusive with <code>--output</code>)</td>
</tr> </tr>
<tr> <tr>
<td><code>-f, --force</code></td> <td><code>-f, --force</code></td>
@@ -1212,12 +1645,29 @@
<td><code>0</code></td> <td><code>0</code></td>
<td>Minimum count for a genome to be considered a carrier (logical operators only)</td> <td>Minimum count for a genome to be considered a carrier (logical operators only)</td>
</tr> </tr>
<tr>
<td><code>--dense</code></td>
<td>off</td>
<td>Pack the output's presence matrices in the dense format instead of the default sparse one</td>
</tr>
<tr>
<td><code>--force-copy</code></td>
<td>off</td>
<td>Copy each layer's unchanged kmer-identity files (mphf/unitigs/evidence/fingerprint) instead of hard-linking them</td>
</tr>
</tbody> </tbody>
</table> </table>
<h2 id="aggregation-operators">Aggregation operators</h2> <h2 id="aggregation-operators">Aggregation operators</h2>
<p><code>any</code>, <code>all</code>, <code>none</code> (logical, evaluated against <code>--presence-threshold</code>), <code>sum</code>, <code>min</code>, <code>max</code> (numeric, count index only). If a group's operator is left unspecified, it defaults to <code>any</code> when the source is a presence/absence index and <code>sum</code> when it stores counts.</p> <p><code>any</code>, <code>all</code>, <code>none</code> (logical, evaluated against <code>--presence-threshold</code>), <code>sum</code>, <code>min</code>, <code>max</code> (numeric, count index only). If a group's operator is left unspecified, it defaults to <code>any</code> when the source is a presence/absence index and <code>sum</code> when it stores counts.</p>
<p>A <code>select</code> never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in <code>--select</code>) is a cheap copy.</p> <p>A <code>select</code> never changes the underlying kmer set — only the per-genome data (counts or presence) is rewritten, so an unaggregated pass-through column (a plain genome label in <code>--select</code>) is a cheap copy.</p>
<p>At least one of <code>--output</code>/<code>--in-place</code> is required, and at least one output column must be defined; every name listed in <code>--select</code> must resolve to either a defined group or an existing genome label. See <a href="../predicates/">Genome predicates and taxonomy paths</a> for the predicate syntax used by <code>--group</code>.</p> <p>At least one output column must be defined; every name listed in <code>--select</code> must resolve to either a defined group or an existing genome label. See <a href="../predicates/">Genome predicates and taxonomy paths</a> for the predicate syntax used by <code>--group</code>.</p>
<h2 id="disk-usage">Disk usage</h2>
<p><code>select</code> always writes to a new output directory — there is no in-place mode. Each layer's kmer-identity files (MPHF, unitigs, evidence, fingerprint) never change under a column projection/aggregation, so they are hard-linked into the output rather than copied: no extra disk is used for them, even on a very large index. Linking falls back to a real copy automatically if it fails (e.g. <code>SOURCE</code>/<code>OUTPUT</code> on different filesystems). Use <code>--force-copy</code> to always copy instead — needed when the output must be able to survive independently of the source on disk (a hard link shares the same underlying data, so overwriting one path outside <code>select</code> itself would affect the other).</p>
<p>To replace an index with a selected version of itself, select to a temporary directory and swap it in:</p>
<div class="highlight"><pre><span></span><code>obikmer<span class="w"> </span><span class="k">select</span><span class="w"> </span>INDEX<span class="w"> </span>--output<span class="w"> </span>INDEX.tmp<span class="w"> </span>--group<span class="w"> </span>...<span class="w"> </span>--group-op<span class="w"> </span>...<span class="w"> </span>--select<span class="w"> </span>...
rm<span class="w"> </span>-rf<span class="w"> </span>INDEX
mv<span class="w"> </span>INDEX.tmp<span class="w"> </span>INDEX
</code></pre></div>
+427 -11
View File
@@ -9,7 +9,7 @@
<link rel="prev" href="../../theory/indexing_architecture/"> <link rel="prev" href="../../theory/phylogeny/snp_based/sampling_design/">
<link rel="next" href="../index_command/"> <link rel="next" href="../index_command/">
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1017,6 +1405,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1017,6 +1405,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+426 -10
View File
@@ -256,15 +256,26 @@
<li class="md-nav__item">
<a href="../../theory/kmers_and_superkmers/" class="md-nav__link">
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_1" >
<label class="md-nav__link" for="__nav_3_1" id="__nav_3_1_label" tabindex="0">
<span class="md-ellipsis"> <span class="md-ellipsis">
Kmers and super-kmers Kmer indexing
@@ -272,12 +283,20 @@
</a> <span class="md-nav__icon md-icon"></span>
</li> </label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_1">
<span class="md-nav__icon md-icon"></span>
Kmer indexing
</label>
<ul class="md-nav__list" data-md-scrollfix>
@@ -285,7 +304,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/encoding/" class="md-nav__link"> <a href="../../theory/kmer_indexing/encoding/" class="md-nav__link">
@@ -313,14 +332,14 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/entropy_filter/" class="md-nav__link"> <a href="../../theory/kmer_indexing/kmers/" class="md-nav__link">
<span class="md-ellipsis"> <span class="md-ellipsis">
Low-complexity kmer filter Kmers
@@ -341,7 +360,7 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/minimizer_selection/" class="md-nav__link"> <a href="../../theory/kmer_indexing/minimizer_selection/" class="md-nav__link">
@@ -369,7 +388,35 @@
<li class="md-nav__item"> <li class="md-nav__item">
<a href="../../theory/indexing_architecture/" class="md-nav__link"> <a href="../../theory/kmer_indexing/superkmers/" class="md-nav__link">
<span class="md-ellipsis">
Super-kmers
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/indexing_architecture/" class="md-nav__link">
@@ -390,6 +437,347 @@
<li class="md-nav__item">
<a href="../../theory/kmer_indexing/entropy_filter/" class="md-nav__link">
<span class="md-ellipsis">
Low-complexity kmer filter
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2" >
<label class="md-nav__link" for="__nav_3_2" id="__nav_3_2_label" tabindex="0">
<span class="md-ellipsis">
Phylogeny
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="2" aria-labelledby="__nav_3_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2">
<span class="md-nav__icon md-icon"></span>
Phylogeny
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_1" >
<label class="md-nav__link" for="__nav_3_2_1" id="__nav_3_2_1_label" tabindex="0">
<span class="md-ellipsis">
Kmer-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_1_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_1">
<span class="md-nav__icon md-icon"></span>
Kmer-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/kmer_based/kmer_set_distances/" class="md-nav__link">
<span class="md-ellipsis">
Whole k-mer set distances
</span>
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle " type="checkbox" id="__nav_3_2_2" >
<label class="md-nav__link" for="__nav_3_2_2" id="__nav_3_2_2_label" tabindex="0">
<span class="md-ellipsis">
SNP-based
</span>
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" data-md-level="3" aria-labelledby="__nav_3_2_2_label" aria-expanded="false">
<label class="md-nav__title" for="__nav_3_2_2">
<span class="md-nav__icon md-icon"></span>
SNP-based
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/central_snp_model/" class="md-nav__link">
<span class="md-ellipsis">
Central-position SNP model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/gamma_rate_heterogeneity/" class="md-nav__link">
<span class="md-ellipsis">
Gamma rate heterogeneity
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sankoff_model/" class="md-nav__link">
<span class="md-ellipsis">
Sankoff model
</span>
</a>
</li>
<li class="md-nav__item">
<a href="../../theory/phylogeny/snp_based/sampling_design/" class="md-nav__link">
<span class="md-ellipsis">
Sampling design
</span>
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
</ul> </ul>
</nav> </nav>
@@ -1017,6 +1405,34 @@
<li class="md-nav__item">
<a href="../../formats/vcf/" class="md-nav__link">
<span class="md-ellipsis">
VCF pseudo-alignment
</span>
</a>
</li>
</ul> </ul>
</nav> </nav>
+9 -15
View File
@@ -77,27 +77,21 @@ pub struct DistanceArgs {
#[arg(long, value_name = "ALPHA|auto", value_parser = parse_gamma_shape)] #[arg(long, value_name = "ALPHA|auto", value_parser = parse_gamma_shape)]
pub gamma_shape: Option<GammaShapeArg>, pub gamma_shape: Option<GammaShapeArg>,
/// Save and reuse the sample of a SNP-based distance /// Save and reuse a SNP distance sample (unavailable for now)
/// ///
/// Saves, in this directory, the sample's tally of pairwise differences /// Meant to save, in this directory, the sample of a SNP-based distance,
/// between genomes, so that a later run on the same index can reuse it /// so that a later run with identical options reuses it instead of
/// instead of drawing again. The directory is created if needed. /// drawing again.
/// ///
/// The saved sample is reused only if these options are identical: /// This feature is being rebuilt and is not available for now: giving
/// --n-sites, --free-loss, --no-ambiguity, --quartet, --exclude-genome, /// this option, or --session-force, stops the command with an error.
/// --min-shared-family, --entropy and --entropy-sd. Otherwise the run
/// stops and lists the differences, unless --session-force is given.
///
/// For now only SNP-based distances use sessions; Sankoff analyses and
/// the TNT, PhyG and IQ-TREE exports neither read nor write them.
#[arg(long, value_name = "DIR")] #[arg(long, value_name = "DIR")]
pub session: Option<PathBuf>, pub session: Option<PathBuf>,
/// Replace a saved sample made with other parameters /// With --session: replace a saved sample (unavailable for now)
/// ///
/// With --session, discards the saved sample when its parameters differ /// Meant to discard a saved sample made with other parameters and draw a
/// from this run's, and draws a new one. Without this option, a /// new one. Unavailable for now, like --session.
/// difference is an error.
#[arg(long, requires = "session")] #[arg(long, requires = "session")]
pub session_force: bool, pub session_force: bool,