docs: restructure documentation for kmer indexing and phylogeny theory
Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models. - Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic. - Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation. - Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers. - Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details. - Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics. - Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture. - Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model. - Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections. - Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation. - Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design. - Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure. - Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations. - Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`. - Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013). - Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation. - Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance. - Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990). - Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section. - Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`. - Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
This commit is contained in:
1 parent
d78a6b4501
commit
33d7a59dd9
43 files changed
+25375
-1117
No files matched your search
@@ -14,7 +14,7 @@ Nothing here comes from a reference genome. The file has no real coordinates, an
|
||||
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
|
||||
```
|
||||
|
||||
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, together with an already-built sibling annex (see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
|
||||
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, (the sibling annex it reads is built automatically if missing; see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
|
||||
|
||||
## Layout
|
||||
|
||||
|
||||
@@ -857,3 +857,235 @@
|
||||
doi = "10.1214/aop/1176996461",
|
||||
issn = "0091-1798,2168-894X"
|
||||
}
|
||||
|
||||
@ARTICLE{Karlin1997-lm,
|
||||
title = "{Compositional differences within and between eukaryotic genomes}",
|
||||
author = "Karlin, S and Mrázek, J",
|
||||
journal = "Proceedings of the National Academy of Sciences of the United
|
||||
States of America",
|
||||
publisher = "Proceedings of the National Academy of Sciences",
|
||||
volume = 94,
|
||||
number = 19,
|
||||
pages = "10227--10232",
|
||||
abstract = "Eukaryotic genome similarity relationships are inferred using
|
||||
sequence information derived from large aggregates of genomic
|
||||
sequences. Comparisons within and between species sample
|
||||
sequences are based on the profile of dinucleotide relative
|
||||
abundance values (The profile is ρ* XY = f * XY / f * X f * Y for
|
||||
all XY , where f * X denotes the frequency of the nucleotide X
|
||||
and f * XY denotes the frequency of the dinucleotide XY , both
|
||||
computed from the sequence concatenated with its inverted
|
||||
complement). Previous studies with respect to prokaryotes and
|
||||
this study document that profiles of different DNA sequence
|
||||
samples (sample size ≥50 kb) from the same organism are generally
|
||||
much more similar to each other than they are to profiles from
|
||||
other organisms, and that closely related organisms generally
|
||||
have more similar profiles than do distantly related organisms.
|
||||
On this basis we refer to the collection {ρ* XY } as the genome
|
||||
signature . This paper identifies ρ* XY extremes and compares
|
||||
genome signature differences for a diverse range of eukaryotic
|
||||
species. Interpretations on the mechanisms maintaining these
|
||||
profile differences center on genome-wide replication, repair,
|
||||
DNA structures, and context-dependent mutational biases. It is
|
||||
also observed that mitochondrial genome signature differences
|
||||
between species parallel the corresponding nuclear genome
|
||||
signature differences despite large differences between
|
||||
corresponding mitochondrial and nuclear signatures. The genome
|
||||
signature differences also have implications for contrasts
|
||||
between rodents and other mammals, and between monocot and dicot
|
||||
plants, as well as providing evidence for similarities among
|
||||
fungi and the diversity of protists.",
|
||||
month = sep,
|
||||
year = 1997,
|
||||
url = "http://dx.doi.org/10.1073/pnas.94.19.10227",
|
||||
doi = "10.1073/pnas.94.19.10227",
|
||||
pmc = "PMC23344",
|
||||
pmid = 9294192,
|
||||
issn = "0027-8424,1091-6490",
|
||||
language = "en"
|
||||
}
|
||||
|
||||
@ARTICLE{Karlin1997-vm,
|
||||
title = "{Compositional biases of bacterial genomes and evolutionary
|
||||
implications}",
|
||||
author = "Karlin, S and Mrázek, J and Campbell, A M",
|
||||
journal = "Journal of Bacteriology",
|
||||
publisher = "American Society for Microbiology",
|
||||
volume = 179,
|
||||
number = 12,
|
||||
pages = "3899--3913",
|
||||
abstract = "We compare and contrast genome-wide compositional biases and
|
||||
distributions of short oligonucleotides across 15 diverse
|
||||
prokaryotes that have substantial genomic sequence collections.
|
||||
These include seven complete genomes (Escherichia coli,
|
||||
Haemophilus influenzae, Mycoplasma genitalium, Mycoplasma
|
||||
pneumoniae, Synechocystis sp. strain PCC6803, Methanococcus
|
||||
jannaschii, and Pyrobaculum aerophilum). A key observation
|
||||
concerns the constancy of the dinucleotide relative abundance
|
||||
profiles over multiple 50-kb disjoint contigs within the same
|
||||
genome. (The profile is rhoXY* = fXY*/fX*fY* for all XY, where
|
||||
fX* denotes the frequency of the nucleotide X and fY* denotes the
|
||||
frequency of the dinucleotide XY, both computed from the sequence
|
||||
concatenated with its inverted complementary sequence.) On the
|
||||
basis of this constancy, we refer to the collection [rhoXY*] as
|
||||
the genome signature. We establish that the differences between
|
||||
[rhoXY*] vectors of 50-kb sample contigs of different genomes
|
||||
virtually always exceed the differences between those of the same
|
||||
genomes. Various di- and tetranucleotide biases are identified.
|
||||
In particular, we find that the dinucleotide CpG=CG is
|
||||
underrepresented in many thermophiles (e.g., M. jannaschii,
|
||||
Sulfolobus sp., and M. thermoautotrophicum) but overrepresented
|
||||
in halobacteria. TA is broadly underrepresented in prokaryotes
|
||||
and eukaryotes, but normal counts appear in Sulfolobus and P.
|
||||
aerophilum sequences. More than for any other bacterial genome,
|
||||
palindromic tetranucleotides are underrepresented in H.
|
||||
influenzae. The M. jannaschii sequence is unprecedented in its
|
||||
extreme underrepresentation of CTAG tetranucleotides and in the
|
||||
anomalous distribution of CTAG sites around the genome.
|
||||
Comparative analysis of numbers of long tetranucleotide
|
||||
microsatellites distinguishes H. influenzae. Dinucleotide
|
||||
relative abundance differences between bacterial sequences are
|
||||
compared. For example, in these assessments of differences, the
|
||||
cyanobacteria Synechocystis, Synechococcus, and Anabaena do not
|
||||
form a coherent group and are as far from each other as general
|
||||
gram-negative sequences are from general gram-positive sequences.
|
||||
The difference of M. jannaschii from low-G+C gram-positive
|
||||
proteobacteria is one-half of the difference from gram-negative
|
||||
proteobacteria. Interpretations and hypotheses center on the role
|
||||
of the genome signature in highlighting similarities and
|
||||
dissimilarities across different classes of prokaryotic species,
|
||||
possible mechanisms underlying the genome signature, the form and
|
||||
level of genome compositional flux, the use of the genome
|
||||
signature as a chronometer of molecular phylogeny, and
|
||||
implications with respect to the three putative eubacterial,
|
||||
archaeal, and eukaryote domains of life and to the origin and
|
||||
early evolution of eukaryotes.",
|
||||
month = jun,
|
||||
year = 1997,
|
||||
url = "http://dx.doi.org/10.1128/jb.179.12.3899-3913.1997",
|
||||
doi = "10.1128/jb.179.12.3899-3913.1997",
|
||||
pmc = "PMC179198",
|
||||
pmid = 9190805,
|
||||
issn = "0021-9193,1098-5530",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Karlin1995-lt,
|
||||
title = "{Dinucleotide relative abundance extremes: a genomic signature}",
|
||||
author = "Karlin, S and Burge, C",
|
||||
journal = "Trends in Genetics: TIG",
|
||||
publisher = "Elsevier BV",
|
||||
volume = 11,
|
||||
number = 7,
|
||||
pages = "283--290",
|
||||
abstract = "Early biochemical experiments established that the set of
|
||||
dinucleotide odds ratios or 'general design' is a remarkably
|
||||
stable property of the DNA of an organism, which is essentially
|
||||
the same in protein-coding DNA, bulk genomic DNA, and in
|
||||
different renaturation rate and density gradient fractions of
|
||||
genomic DNA in many organisms. Analysis of currently available
|
||||
genomic sequence data has extended these earlier results, showing
|
||||
that the general designs of disjoint samples of a genome are
|
||||
substantially more similar to each other than to those of
|
||||
sequences from other organisms and that closely related organisms
|
||||
have similar general designs. From this perspective, the set of
|
||||
dinucleotide odds ratio (relative abundance) values constitute a
|
||||
signature of each DNA genome, which can discriminate between
|
||||
sequences from different organisms. Dinucleotide-odds ratio
|
||||
values appear to reflect not only the chemistry of dinucleotide
|
||||
stacking energies and base-step conformational preferences, but
|
||||
also the species-specific properties of DNA modification,
|
||||
replication and repair mechanisms.",
|
||||
month = jul,
|
||||
year = 1995,
|
||||
url = "http://dx.doi.org/10.1016/s0168-9525(00)89076-9",
|
||||
doi = "10.1016/s0168-9525(00)89076-9",
|
||||
pmid = 7482779,
|
||||
issn = "0168-9525,1362-4555",
|
||||
language = "en"
|
||||
}
|
||||
|
||||
@ARTICLE{Karlin1994-on,
|
||||
title = "{Heterogeneity of genomes: measures and values}",
|
||||
author = "Karlin, S and Ladunga, I and Blaisdell, B E",
|
||||
journal = "Proceedings of the National Academy of Sciences of the United
|
||||
States of America",
|
||||
publisher = "Proceedings of the National Academy of Sciences",
|
||||
volume = 91,
|
||||
number = 26,
|
||||
pages = "12837--12841",
|
||||
abstract = "Genomic homogeneity is investigated for a broad base of DNA
|
||||
sequences in terms of dinucleotide relative abundance distances
|
||||
(abbreviated delta-distances) and of oligonucleotide
|
||||
compositional extremes. It is shown that delta-distances between
|
||||
different genomic sequences in the same species are low, only
|
||||
about 2 or 3 times the distance found in random DNA, and are
|
||||
generally smaller than the between-species delta-distances.
|
||||
Extremes in short oligonucleotides include underrepresentation of
|
||||
TpA and overrepresentation of GpC in most temperate bacteriophage
|
||||
sequences; underrepresentation of CTAG in most eubacterial
|
||||
genomes; underrepresentation of GATC in most bacteriophage; CpG
|
||||
suppression in vertebrates, in all animal mitochondrial genomes,
|
||||
and in many thermophilic bacterial sequences; and
|
||||
overrepresentation of GpG/CpC in all animal mitochondrial sets
|
||||
and chloroplast genomes. Interpretations center on DNA structures
|
||||
(dinucleotide stacking energies, DNA curvature and superhelicity,
|
||||
nucleosome organization), context-dependent mutational events,
|
||||
methylation effects, and processes of replication and repair.",
|
||||
month = dec,
|
||||
year = 1994,
|
||||
url = "http://dx.doi.org/10.1073/pnas.91.26.12837",
|
||||
doi = "10.1073/pnas.91.26.12837",
|
||||
pmc = "PMC45535",
|
||||
pmid = 7809131,
|
||||
issn = "0027-8424,1091-6490",
|
||||
language = "en"
|
||||
}
|
||||
@ARTICLE{Bennett2013-cl,
|
||||
title = "{Small, smaller, smallest: the origins and evolution of ancient
|
||||
dual symbioses in a Phloem-feeding insect}",
|
||||
author = "Bennett, Gordon M and Moran, Nancy A",
|
||||
journal = "Genome Biology and Evolution",
|
||||
publisher = "Oxford University Press (OUP)",
|
||||
volume = 5,
|
||||
number = 9,
|
||||
pages = "1675--1688",
|
||||
abstract = "Many insects rely on bacterial symbionts with tiny genomes
|
||||
specialized for provisioning nutrients lacking in host diets.
|
||||
Xylem sap and phloem sap are both deficient as insect diets, but
|
||||
differ dramatically in nutrient content, potentially affecting
|
||||
symbiont genome evolution. For sap-feeding insects, sequenced
|
||||
symbiont genomes are available only for phloem-feeding examples
|
||||
from the suborder Sternorrhyncha and xylem-feeding examples from
|
||||
the suborder Auchenorrhyncha, confounding comparisons. We
|
||||
sequenced genomes of the obligate symbionts, Sulcia muelleri and
|
||||
Nasuia deltocephalinicola, of the phloem-feeding pest insect,
|
||||
Macrosteles quadrilineatus (Auchenorrhyncha: Cicadellidae). Our
|
||||
results reveal that Nasuia-ALF has the smallest bacterial genome
|
||||
yet sequenced (112 kb), and that the Sulcia-ALF genome (190 kb)
|
||||
is smaller than that of Sulcia in other insect lineages.
|
||||
Together, these symbionts retain the capability to synthesize the
|
||||
10 essential amino acids, as observed for several symbiont pairs
|
||||
from xylem-feeding Auchenorrhyncha. Nasuia retains genes enabling
|
||||
synthesis of two amino acids, DNA replication, transcription, and
|
||||
translation. Both symbionts have lost genes underlying ATP
|
||||
synthesis through oxidative phosphorylation, possibly as a
|
||||
consequence of the enriched sugar content of phloem. Shared
|
||||
genomic features, including reassignment of the UGA codon from
|
||||
Stop to tryptophan, and phylogenetic results suggest that
|
||||
Nasuia-ALF is most closely related to Zinderia, the
|
||||
betaproteobacterial symbiont of spittlebugs. Thus,
|
||||
Nasuia/Zinderia and Sulcia likely represent ancient associates
|
||||
that have co-resided in hosts since the divergence of leafhoppers
|
||||
and spittlebugs >200 Ma, and possibly since the origin of the
|
||||
Auchenorrhyncha, >260 Ma.",
|
||||
month = aug,
|
||||
year = 2013,
|
||||
url = "http://dx.doi.org/10.1093/gbe/evt118",
|
||||
keywords = "Nasuia deltocephalinicola; Sulcia muelleri; gene loss; genome
|
||||
evolution; leafhopper; nutritional symbioses",
|
||||
doi = "10.1093/gbe/evt118",
|
||||
pmc = "PMC3787670",
|
||||
pmid = 23918810,
|
||||
issn = "1759-6653",
|
||||
language = "en"
|
||||
}
|
||||
@@ -47,7 +47,9 @@ P(n_K > 0) &= 1-e^{-\lambda} \\
|
||||
\end{aligned}
|
||||
$$
|
||||
|
||||
Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
|
||||
##### Assuming equal genome lengths
|
||||
|
||||
The number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
|
||||
|
||||
|
||||
$$
|
||||
@@ -56,11 +58,22 @@ $$
|
||||
|
||||
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
|
||||
|
||||
$$
|
||||
I_p(r,S-r+1)
|
||||
=
|
||||
\frac{1}{B(r,S-r+1)}
|
||||
\int_0^p t^{r-1}(1-t)^{S-r}\,dt
|
||||
$$
|
||||
|
||||
and
|
||||
|
||||
$$
|
||||
p = 1-e^{-2G/4^k}.
|
||||
$$
|
||||
|
||||
Different genome sizes
|
||||
the regularized incomplete beta function is available in standard statistical libraries.
|
||||
|
||||
##### Different genome sizes
|
||||
|
||||
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
|
||||
|
||||
@@ -82,39 +95,63 @@ Using the mean genome size therefore provides a conservative estimate of the FDR
|
||||
|
||||
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
|
||||
|
||||
### Exact tail probability
|
||||
##### Exact tail probability
|
||||
|
||||
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
|
||||
|
||||
Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
|
||||
Let $P_i(j)$ denote the probability that the k-mer occurs in exactly $j$ of the first $i$ genomes. Initially,
|
||||
|
||||
$$
|
||||
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
|
||||
$$
|
||||
|
||||
When genome $i$ is added, the k-mer is either absent, with probability $1-p_i$, or present, with probability $p_i$. The probability of observing exactly $j$ presences among the first $i$ genomes is therefore
|
||||
|
||||
$$
|
||||
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
|
||||
$$
|
||||
|
||||
This dynamic programming algorithm requires $O(S^2)$ operations and $O(S)$ memory. However, to compute $P(X\geq r)$, only the probabilities for fewer than $r$ presences need to be tracked, since
|
||||
|
||||
$$
|
||||
P(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
|
||||
$$
|
||||
|
||||
This reduces the computational cost to $O(Sr)$ operations and $O(r)$ memory.
|
||||
|
||||
#### Distinguishing random from non-random k-mer sharing
|
||||
|
||||
The probability $P(X\geq r)$ describes the occurrence of a given k-mer in at least $r$ genomes under the random-sequence model. However, the index contains only k-mers observed in at least one genome. Therefore, when considering the k-mers recorded in the index, the relevant probability is the conditional probability:
|
||||
|
||||
$$
|
||||
q_r = P(X\geq r\mid X\geq 1)
|
||||
= \frac{P(X\geq r)}{P(X\geq 1)}.
|
||||
$$
|
||||
|
||||
Let $k_r$ be the number of distinct k-mers observed in at least $r$ genomes, among the $k_1$ distinct k-mers recorded in the index. The observed proportion of k-mers shared by at least $r$ genomes is:
|
||||
|
||||
$$
|
||||
O_r = \frac{k_r}{k_1}.
|
||||
$$
|
||||
|
||||
Under the random-sequence model, the expected proportion is $q_r$. We can therefore quantify the deviation from random expectations using the difference:
|
||||
|
||||
$$
|
||||
\left \{
|
||||
\begin{aligned}
|
||||
d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\
|
||||
d_0&=1
|
||||
D_r &= O_r-q_r \\
|
||||
&= \frac{k_r}{k_1}-q_r.
|
||||
\end{aligned}
|
||||
\right .
|
||||
$$
|
||||
|
||||
before processing the first genome.
|
||||
Positive values of $D_r$ indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.
|
||||
|
||||
The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then
|
||||
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$ is bounded:
|
||||
|
||||
$$
|
||||
P(X\geq r)=1-\sum_{j=0}^{r-1}d_j.
|
||||
-1\leq D_r\leq 1.
|
||||
$$
|
||||
|
||||
Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$.
|
||||
|
||||
When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean
|
||||
|
||||
$$
|
||||
\lambda=\sum_{i=1}^{S}p_i.
|
||||
$$
|
||||
|
||||
The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe].
|
||||
|
||||
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By construction, $D_1=0$, since all k-mers recorded in the index occur in at least one genome. Unlike a ratio-based measure, this difference remains defined even when no k-mers are observed in at least $r$ genomes. It thus provides a bounded measure of the excess or deficit of k-mer sharing relative to the random-sequence model.
|
||||
|
||||
*******
|
||||
|
||||
@@ -150,11 +187,12 @@ Consequently, the appropriate value of $k$ depends on the total amount of sequen
|
||||
|
||||
|
||||
|
||||
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Mycoplasma genitalium* having a size $G=580\,Mb$ [@Fraser1995-jc], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
|
||||
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Nasuia deltocephalinicola* having a size $G=112\,Kb$ [@Bennett2013-cl], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
|
||||
|
||||
$$
|
||||
\begin{aligned}
|
||||
P(C\geq1) &= 1- e^{-\frac{580.000^2}{2 \cdot 4^{11}}} \\
|
||||
P(C\geq1) &= 1- e^{-\frac{112\,000^2}{2 \cdot 4^{11}}} \\
|
||||
&= 1- e^{-1.5\times10^{3}} \\
|
||||
&\simeq 1
|
||||
\end{aligned}
|
||||
$$
|
||||
@@ -182,7 +220,7 @@ The distinction between the simple $4^k$ model and the Karlin--Ost framework is
|
||||
|
||||
These scales have very different implications for different genome sizes.
|
||||
|
||||
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995; @Karlin1997]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
|
||||
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995-lt; @Karlin1994-on]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
|
||||
|
||||
This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.
|
||||
|
||||
@@ -232,7 +270,7 @@ $$
|
||||
|
||||
The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985].
|
||||
|
||||
|
||||
<!--
|
||||
$o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ».
|
||||
|
||||
Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ :
|
||||
@@ -265,7 +303,7 @@ Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nomb
|
||||
- **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant.
|
||||
|
||||
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
|
||||
|
||||
-->
|
||||
|
||||
## Chance sharing across genomes and the FDR of homology
|
||||
|
||||
@@ -359,6 +397,7 @@ Chance alone produces millions of pairwise shared 21-mers at this genome size, a
|
||||
|
||||
---
|
||||
|
||||
<!-- Notes perso (bac à sable), non publiées dans le HTML
|
||||
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
|
||||
|
||||
## 1. FDR d'un ensemble de k-mers sélectionnés
|
||||
@@ -414,3 +453,5 @@ Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Ell
|
||||
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
|
||||
|
||||
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
|
||||
|
||||
-->
|
||||
@@ -34,7 +34,7 @@ $$
|
||||
D = \sum_i \mathbb{1}[a_i \ne b_i],
|
||||
$$
|
||||
|
||||
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded.
|
||||
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded. On an index that stores counts, a k-mer is taken as present when its count reaches the presence threshold (`--presence-threshold`, 1 by default).
|
||||
|
||||
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
|
||||
|
||||
|
||||
@@ -24,4 +24,4 @@ If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (
|
||||
|
||||
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation.
|
||||
See [`phylo`](../../../usage/phylo.md#distance-estimation-and-tree-reconstruction) for how to select a fixed $\alpha$ or request automatic estimation.
|
||||
@@ -31,4 +31,4 @@ How well a pair of genomes is represented in a sample depends on how many famili
|
||||
|
||||
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
|
||||
|
||||
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
|
||||
See [`phylo`](../../../usage/phylo.md#snp-character-selection-and-sampling) for how to bound and bias the sample.
|
||||
+112
-91
@@ -1,6 +1,13 @@
|
||||
# phylo
|
||||
|
||||
Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
|
||||
Compare the genomes stored in an index and build phylogenetic trees from them. Two complementary approaches are available:
|
||||
|
||||
- **Whole-genome distances** compare the k-mer content of the genomes, without any alignment.
|
||||
- **SNP-based distances and analyses** compare the nucleotide variants found at the central position of related k-mers.
|
||||
|
||||
Distance matrices can be used to build Neighbor-Joining or UPGMA trees. SNP-based analyses can also produce a pseudo-alignment and calibrated substitution costs, with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
obikmer phylo INDEX [OPTIONS]
|
||||
@@ -12,28 +19,43 @@ obikmer phylo INDEX [OPTIONS]
|
||||
|---|---|
|
||||
| `INDEX` | Index directory |
|
||||
|
||||
## Distance matrix (`--distance`)
|
||||
## General options
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--distance` | none | Compute and write a distance matrix. See the two tables below for the full list of accepted values. No matrix is computed unless this is given, or `--nj`, `--upgma` or `--shared-kmers` is (they need a matrix and then use `jaccard`) |
|
||||
| `--cpu-max N` | all available cores | Maximum number of CPU cores to use. It caps every thread pool of the process and can only lower the number of cores, never raise it above those available. Common to all subcommands |
|
||||
| `-o, --output PREFIX` | none | Prefix of the output files. Without it, the distance matrix goes to standard output and every other file is written in the current directory under its name without the prefix. See "Output files" below |
|
||||
|
||||
## SNP families
|
||||
|
||||
SNP-based analyses use groups of k-mers that differ only at their central nucleotide. These groups are called families (see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the exact definitions and eligibility rules). A variable family is one character of the analysis, and the variants a genome carries give its state for that character.
|
||||
|
||||
## Distance estimation and tree reconstruction
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--distance` | none | Compute and write a distance matrix. The accepted values are listed just below the table. No matrix is computed unless this is given, or `--nj` or `--upgma` is (they need a matrix and then use `jaccard`) |
|
||||
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
|
||||
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
|
||||
| `--presence-threshold` | `1` | How counts are turned into presence/absence on an index that stores counts: a kmer is present in a genome only when its count reaches this value. Used by `jaccard`, `mash` and `hamming`, and by the shared-kmer counts of `--shared-kmers`. No effect on an index without counts, nor on the distances that use the counts themselves |
|
||||
| `--session DIR`, `--session-force` | none | Save and reuse the sample of a `snp-*` distance. Currently unavailable, see `--session` below |
|
||||
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
|
||||
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index distance, not a `snp-*` value |
|
||||
| `--shared-kmers` | off | Write the shared-kmer count matrix (`<prefix>_shared.csv`). It can be used alone, without any distance, or together with a whole-index distance; it is refused with a `snp-*` value |
|
||||
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
|
||||
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
|
||||
| `-o, --output` | none (stdout) | Output file prefix |
|
||||
|
||||
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
||||
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `jaccard`, `mash` and `hamming` compare which kmers are present and work on either index type; the six others (`bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`) use the counts and need an index that stores them), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
|
||||
|
||||
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||
`snp-*` values read the sibling annex (below), which is built automatically the first time it is needed, and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "SNP character selection and sampling" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
|
||||
|
||||
### Output
|
||||
### Matrix format
|
||||
|
||||
When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
|
||||
|
||||
## `--exclude-genome`, `--min-shared-family`
|
||||
### `--session`: currently unavailable
|
||||
|
||||
`--session DIR` and `--session-force` are meant to save the sample of a `snp-*` `--distance` run in a directory, so that a later run with identical selection parameters reuses it instead of drawing again. This feature is being rebuilt and is not available for now: giving either option stops the command with an error. Until it returns, keep the output files of a run (the alignment, or the distance matrix) rather than relying on re-drawing the same sample.
|
||||
|
||||
## Genome selection and filtering
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
@@ -42,21 +64,79 @@ When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHY
|
||||
|
||||
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
|
||||
|
||||
## Central-position SNP model
|
||||
## SNP character selection and sampling
|
||||
|
||||
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index:
|
||||
SNP-based analyses can use every eligible variable family, or a sample drawn from the index. The options below control which characters are selected, which genomes and states are kept, and how the sample is spread over the index.
|
||||
|
||||
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
|
||||
|
||||
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags.
|
||||
|
||||
Every invocation draws its own fresh random sample: running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately.
|
||||
|
||||
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--n-sites N\|all` | none (mandatory for `--pseudo-alignment` and `--sankoff`) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
|
||||
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
||||
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
||||
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
|
||||
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
|
||||
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
|
||||
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
||||
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
||||
|
||||
### `--quartet`: keeping only families that can resolve a tree
|
||||
|
||||
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
||||
|
||||
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
||||
|
||||
### `--pair-balance`: representing every pair of genomes
|
||||
|
||||
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
|
||||
|
||||
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
|
||||
|
||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||
|
||||
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||
|
||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
||||
|
||||
The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
|
||||
|
||||
### Describing a draw: `--sample-stats`, `--sample-plan`
|
||||
|
||||
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
|
||||
|
||||
`--sample-stats` writes three files:
|
||||
|
||||
| File | Content |
|
||||
|---|---|
|
||||
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
|
||||
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
|
||||
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
|
||||
|
||||
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. A site is informative under the same rule as `--quartet`: without `--free-loss`, absence is a state of its own and can form one of the two pairs; with `--free-loss` it is missing data and cannot. Excluded genomes are listed with zeros in the first file and count nowhere.
|
||||
|
||||
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
|
||||
|
||||
## SNP family preparation and diagnostics
|
||||
|
||||
SNP-based options read precomputed tables describing the families: the sibling annex, and tables derived from it (entropy, quartet, family overlap, pair weights). They are built automatically the first time they are needed and reused afterwards. The options below prepare them in advance, or describe the families.
|
||||
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the `--no-ambiguity` / `--only-diploid` convention given: `--entropy` the entropy annex, `--quartet` the quartet annex, `--pair-balance` the pair-weight and family-overlap annexes, `--family-overlap` or `--min-shared-family` the family-overlap annex. For example `--sibling-annex --quartet --only-diploid` prepares exactly the quartet annex of that convention, and `--sibling-annex` alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
|
||||
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
|
||||
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
|
||||
| `--sibling-hist` | Write `<prefix>_sibling_hist.csv` (columns `size,count`): the global family-size histogram (1-4 members) only, without the per-genome breakdown |
|
||||
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
|
||||
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
|
||||
| `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index |
|
||||
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
|
||||
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
|
||||
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
|
||||
|
||||
### `--sibling-stats`
|
||||
|
||||
@@ -73,50 +153,6 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
||||
|
||||
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
|
||||
|
||||
### `--pseudo-alignment`
|
||||
|
||||
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
|
||||
|
||||
#### VCF output (`--vcf`)
|
||||
|
||||
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
|
||||
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
|
||||
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
|
||||
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
|
||||
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
|
||||
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
|
||||
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
|
||||
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
|
||||
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
|
||||
|
||||
## Sampling at scale: `--n-sites`, `--shannon`, `--entropy`
|
||||
|
||||
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
|
||||
|
||||
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
|
||||
|
||||
Every invocation, with or without `--session`, draws its own fresh random sample by default — running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately. `--session` does not change this: it makes a *specific* sample reusable on request, it does not make sampling itself reproducible from one independent run to the next.
|
||||
|
||||
### `--session`: reusing a sample across separate commands
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
|
||||
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
|
||||
|
||||
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
|
||||
|
||||
A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
|
||||
|
||||
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
|
||||
|
||||
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
|
||||
|
||||
### `--shannon`: measuring how informative a family is
|
||||
|
||||
`<prefix>_entropy.csv` has one row per family visited:
|
||||
@@ -134,43 +170,27 @@ With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`)
|
||||
|
||||
`--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
|
||||
|
||||
### `--quartet`: keeping only families that can resolve a tree
|
||||
|
||||
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
|
||||
|
||||
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
|
||||
|
||||
### Describing a draw: `--sample-stats`, `--sample-plan`
|
||||
|
||||
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
|
||||
|
||||
`--sample-stats` writes three files:
|
||||
|
||||
| File | Content |
|
||||
|---|---|
|
||||
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
|
||||
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
|
||||
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
|
||||
|
||||
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. Excluded genomes are listed with zeros in the first file and count nowhere.
|
||||
|
||||
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
|
||||
### `--shannon-summary`
|
||||
|
||||
`--shannon-summary` writes `<prefix>_shannon_summary.csv` (`entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families`): the number of variable families per entropy (`entropy15` over every carrier, `entropy4` over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (`bin_start` is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers `--no-ambiguity` loses, from the same scan as `--shannon` but without one line per family.
|
||||
|
||||
### `--pair-balance`: representing every pair of genomes
|
||||
## SNP pseudo-alignment
|
||||
|
||||
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
|
||||
A pseudo-alignment has one column per variable family and one state per genome. Unlike a conventional sequence alignment, its columns come directly from groups of related k-mers of the index. The number of sites is set with `--n-sites`, and the options of "SNP character selection and sampling" above control which families are included and how ambiguous or absent states are written.
|
||||
|
||||
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
|
||||
| Option | Description |
|
||||
|---|---|
|
||||
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
|
||||
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
|
||||
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
|
||||
|
||||
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
|
||||
### `--pseudo-alignment`
|
||||
|
||||
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
|
||||
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
|
||||
|
||||
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
|
||||
#### VCF output (`--vcf`)
|
||||
|
||||
The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
|
||||
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
|
||||
|
||||
## Sankoff calibration and phylogenetic exports
|
||||
|
||||
@@ -205,7 +225,7 @@ TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in t
|
||||
|
||||
## Output files
|
||||
|
||||
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix, if one is computed, is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
|
||||
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, the distance matrix goes to standard output and every other file is written in the current directory under its name without the `<prefix>_` part (for example `nj.nwk`, `entropy.csv`, `alignment.fasta`). All matrices use genome labels as row/column headers, in index order.
|
||||
|
||||
### Distance matrix
|
||||
|
||||
@@ -219,13 +239,14 @@ With `-o/--output PREFIX`, the relevant subset of the files below is written. Wi
|
||||
|
||||
CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
|
||||
|
||||
### Central-position SNP model
|
||||
### SNP analyses
|
||||
|
||||
| File | Written by | Format | Content |
|
||||
|---|---|---|---|
|
||||
| `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
|
||||
| `<prefix>_sibling_hist.csv` | `--sibling-hist` | CSV table | global family-size histogram (`size,count`) |
|
||||
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
|
||||
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
|
||||
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see `--shannon` above |
|
||||
| `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers |
|
||||
| `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes |
|
||||
| `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept |
|
||||
|
||||
Reference in new issue
Block a user