docs: restructure documentation for kmer indexing and phylogeny theory

Reorganize the documentation structure to better separate kmer indexing logic from phylogenetic analysis methods. This involves moving several existing pages into new directories under `theory/kmer_indexing` and `theory/phylogeny`, updating navigation links across the site, and adding new content for VCF pseudo-alignment and specific phylogenetic models.

- Move `doc/index.html` to `doc/theory/kmer_indexing/minimizer_selection/index.html` to document minimizer selection logic.
- Move `doc/installation/index.html` to `doc/theory/phylogeny/kmer_based/kmer_set_distances/index.html` for whole k-mer set distance documentation.
- Move `doc/usage/dump/index.html` to `doc/theory/kmer_indexing/superkmers/index.html` to cover canonical super-kmers.
- Move `doc/usage/query/index.html` to `doc/theory/kmer_indexing/entropy_filter/index.html` for low-complexity kmer filtering details.
- Move `doc/theory/encoding/index.html` to `doc/theory/kmer_indexing/encoding/index.html` for DNA and kmer encoding specifics.
- Move `doc/theory/indexing_architecture/index.html` to `doc/theory/kmer_indexing/indexing_architecture/index.html` for partitioning and routing architecture.
- Move `doc/usage/query/index.html` (duplicate move noted in facts, likely referring to the query command usage or similar) to `doc/theory/phylogeny/snp_based/sankoff_model/index.html` for the 16-state Sankoff model.
- Move `doc/theory/minimizer_selection/index.html` to `doc/theory/phylogeny/snp_based/central_snp_model/index.html` for central-position SNP models and substitution corrections.
- Move `doc/theory/indexing_architecture/index.html` (duplicate move noted) to `doc/theory/phylogeny/snp_based/gamma_rate_heterogeneity/index.html` for gamma rate heterogeneity and automatic alpha estimation.
- Move `doc/theory/entropy_filter/index.html` to `doc/theory/phylogeny/snp_based/sampling_design/index.html` for entropy-biased sampling design.
- Move `doc/formats/index_layout/index.html` to `doc/formats/vcf/index.html` to document VCF pseudo-alignment output and structure.
- Add `doc/molecular-ecology-modified.csl` with a new XML style definition for Molecular Ecology Modified citations.
- Update `UserDocMD/formats/vcf.md` to clarify that `--vcf` implies `--pseudo-alignment`, requires `--n-sites`, and generates files alongside `--fasta`.
- Add three new BibTeX entries in `UserDocMD/references.bib` for Karlin (1997, 1995) and Bennett & Moran (2013).
- Update `UserDocMD/theory/kmer_indexing/kmers.md` to use binomial distribution for probability calculations and define k-mer sharing deviation.
- Clarify in `UserDocMD/theory/phylogeny/kmer_based/kmer_set_distances.md` that counts are used instead of presence/absence for $L^1$ distance.
- Update `UserDocMD/theory/phylogeny/snp_based/gamma_rate_heterogeneity.md` to attribute method-of-moments $\alpha$ estimation to obikmer, not Jin & Nei (1990).
- Update `UserDocMD/theory/phylogeny/snp_based/sampling_design.md` to redirect entropy sampling references to the correct phylo section.
- Add new options `--cpu-max`, `--output`, `--pair-balance`, `--sibling-stats`, `--family-overlap`, and `--shannon` to `UserDocMD/usage/phylo.md`.
- Mark SNP-based distance saving and session reuse as unavailable in `src/obikmer/src/cmd/phylo/args/distance.rs`.
This commit is contained in:
Eric Coissac committed 2026-11-19 17:47:59 +01:00
1 parent d78a6b4501
commit 33d7a59dd9
43 files changed
+25375 -1117

No files matched your search

+1 -1
View File
@@ -14,7 +14,7 @@ Nothing here comes from a reference genome. The file has no real coordinates, an
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
```
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, together with an already-built sibling annex (see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, (the sibling annex it reads is built automatically if missing; see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
## Layout
+232
View File
@@ -857,3 +857,235 @@
doi = "10.1214/aop/1176996461",
issn = "0091-1798,2168-894X"
}
@ARTICLE{Karlin1997-lm,
title = "{Compositional differences within and between eukaryotic genomes}",
author = "Karlin, S and Mrázek, J",
journal = "Proceedings of the National Academy of Sciences of the United
States of America",
publisher = "Proceedings of the National Academy of Sciences",
volume = 94,
number = 19,
pages = "10227--10232",
abstract = "Eukaryotic genome similarity relationships are inferred using
sequence information derived from large aggregates of genomic
sequences. Comparisons within and between species sample
sequences are based on the profile of dinucleotide relative
abundance values (The profile is ρ* XY = f * XY / f * X f * Y for
all XY , where f * X denotes the frequency of the nucleotide X
and f * XY denotes the frequency of the dinucleotide XY , both
computed from the sequence concatenated with its inverted
complement). Previous studies with respect to prokaryotes and
this study document that profiles of different DNA sequence
samples (sample size ≥50 kb) from the same organism are generally
much more similar to each other than they are to profiles from
other organisms, and that closely related organisms generally
have more similar profiles than do distantly related organisms.
On this basis we refer to the collection {ρ* XY } as the genome
signature . This paper identifies ρ* XY extremes and compares
genome signature differences for a diverse range of eukaryotic
species. Interpretations on the mechanisms maintaining these
profile differences center on genome-wide replication, repair,
DNA structures, and context-dependent mutational biases. It is
also observed that mitochondrial genome signature differences
between species parallel the corresponding nuclear genome
signature differences despite large differences between
corresponding mitochondrial and nuclear signatures. The genome
signature differences also have implications for contrasts
between rodents and other mammals, and between monocot and dicot
plants, as well as providing evidence for similarities among
fungi and the diversity of protists.",
month = sep,
year = 1997,
url = "http://dx.doi.org/10.1073/pnas.94.19.10227",
doi = "10.1073/pnas.94.19.10227",
pmc = "PMC23344",
pmid = 9294192,
issn = "0027-8424,1091-6490",
language = "en"
}
@ARTICLE{Karlin1997-vm,
title = "{Compositional biases of bacterial genomes and evolutionary
implications}",
author = "Karlin, S and Mrázek, J and Campbell, A M",
journal = "Journal of Bacteriology",
publisher = "American Society for Microbiology",
volume = 179,
number = 12,
pages = "3899--3913",
abstract = "We compare and contrast genome-wide compositional biases and
distributions of short oligonucleotides across 15 diverse
prokaryotes that have substantial genomic sequence collections.
These include seven complete genomes (Escherichia coli,
Haemophilus influenzae, Mycoplasma genitalium, Mycoplasma
pneumoniae, Synechocystis sp. strain PCC6803, Methanococcus
jannaschii, and Pyrobaculum aerophilum). A key observation
concerns the constancy of the dinucleotide relative abundance
profiles over multiple 50-kb disjoint contigs within the same
genome. (The profile is rhoXY* = fXY*/fX*fY* for all XY, where
fX* denotes the frequency of the nucleotide X and fY* denotes the
frequency of the dinucleotide XY, both computed from the sequence
concatenated with its inverted complementary sequence.) On the
basis of this constancy, we refer to the collection [rhoXY*] as
the genome signature. We establish that the differences between
[rhoXY*] vectors of 50-kb sample contigs of different genomes
virtually always exceed the differences between those of the same
genomes. Various di- and tetranucleotide biases are identified.
In particular, we find that the dinucleotide CpG=CG is
underrepresented in many thermophiles (e.g., M. jannaschii,
Sulfolobus sp., and M. thermoautotrophicum) but overrepresented
in halobacteria. TA is broadly underrepresented in prokaryotes
and eukaryotes, but normal counts appear in Sulfolobus and P.
aerophilum sequences. More than for any other bacterial genome,
palindromic tetranucleotides are underrepresented in H.
influenzae. The M. jannaschii sequence is unprecedented in its
extreme underrepresentation of CTAG tetranucleotides and in the
anomalous distribution of CTAG sites around the genome.
Comparative analysis of numbers of long tetranucleotide
microsatellites distinguishes H. influenzae. Dinucleotide
relative abundance differences between bacterial sequences are
compared. For example, in these assessments of differences, the
cyanobacteria Synechocystis, Synechococcus, and Anabaena do not
form a coherent group and are as far from each other as general
gram-negative sequences are from general gram-positive sequences.
The difference of M. jannaschii from low-G+C gram-positive
proteobacteria is one-half of the difference from gram-negative
proteobacteria. Interpretations and hypotheses center on the role
of the genome signature in highlighting similarities and
dissimilarities across different classes of prokaryotic species,
possible mechanisms underlying the genome signature, the form and
level of genome compositional flux, the use of the genome
signature as a chronometer of molecular phylogeny, and
implications with respect to the three putative eubacterial,
archaeal, and eukaryote domains of life and to the origin and
early evolution of eukaryotes.",
month = jun,
year = 1997,
url = "http://dx.doi.org/10.1128/jb.179.12.3899-3913.1997",
doi = "10.1128/jb.179.12.3899-3913.1997",
pmc = "PMC179198",
pmid = 9190805,
issn = "0021-9193,1098-5530",
language = "en"
}
@ARTICLE{Karlin1995-lt,
title = "{Dinucleotide relative abundance extremes: a genomic signature}",
author = "Karlin, S and Burge, C",
journal = "Trends in Genetics: TIG",
publisher = "Elsevier BV",
volume = 11,
number = 7,
pages = "283--290",
abstract = "Early biochemical experiments established that the set of
dinucleotide odds ratios or 'general design' is a remarkably
stable property of the DNA of an organism, which is essentially
the same in protein-coding DNA, bulk genomic DNA, and in
different renaturation rate and density gradient fractions of
genomic DNA in many organisms. Analysis of currently available
genomic sequence data has extended these earlier results, showing
that the general designs of disjoint samples of a genome are
substantially more similar to each other than to those of
sequences from other organisms and that closely related organisms
have similar general designs. From this perspective, the set of
dinucleotide odds ratio (relative abundance) values constitute a
signature of each DNA genome, which can discriminate between
sequences from different organisms. Dinucleotide-odds ratio
values appear to reflect not only the chemistry of dinucleotide
stacking energies and base-step conformational preferences, but
also the species-specific properties of DNA modification,
replication and repair mechanisms.",
month = jul,
year = 1995,
url = "http://dx.doi.org/10.1016/s0168-9525(00)89076-9",
doi = "10.1016/s0168-9525(00)89076-9",
pmid = 7482779,
issn = "0168-9525,1362-4555",
language = "en"
}
@ARTICLE{Karlin1994-on,
title = "{Heterogeneity of genomes: measures and values}",
author = "Karlin, S and Ladunga, I and Blaisdell, B E",
journal = "Proceedings of the National Academy of Sciences of the United
States of America",
publisher = "Proceedings of the National Academy of Sciences",
volume = 91,
number = 26,
pages = "12837--12841",
abstract = "Genomic homogeneity is investigated for a broad base of DNA
sequences in terms of dinucleotide relative abundance distances
(abbreviated delta-distances) and of oligonucleotide
compositional extremes. It is shown that delta-distances between
different genomic sequences in the same species are low, only
about 2 or 3 times the distance found in random DNA, and are
generally smaller than the between-species delta-distances.
Extremes in short oligonucleotides include underrepresentation of
TpA and overrepresentation of GpC in most temperate bacteriophage
sequences; underrepresentation of CTAG in most eubacterial
genomes; underrepresentation of GATC in most bacteriophage; CpG
suppression in vertebrates, in all animal mitochondrial genomes,
and in many thermophilic bacterial sequences; and
overrepresentation of GpG/CpC in all animal mitochondrial sets
and chloroplast genomes. Interpretations center on DNA structures
(dinucleotide stacking energies, DNA curvature and superhelicity,
nucleosome organization), context-dependent mutational events,
methylation effects, and processes of replication and repair.",
month = dec,
year = 1994,
url = "http://dx.doi.org/10.1073/pnas.91.26.12837",
doi = "10.1073/pnas.91.26.12837",
pmc = "PMC45535",
pmid = 7809131,
issn = "0027-8424,1091-6490",
language = "en"
}
@ARTICLE{Bennett2013-cl,
title = "{Small, smaller, smallest: the origins and evolution of ancient
dual symbioses in a Phloem-feeding insect}",
author = "Bennett, Gordon M and Moran, Nancy A",
journal = "Genome Biology and Evolution",
publisher = "Oxford University Press (OUP)",
volume = 5,
number = 9,
pages = "1675--1688",
abstract = "Many insects rely on bacterial symbionts with tiny genomes
specialized for provisioning nutrients lacking in host diets.
Xylem sap and phloem sap are both deficient as insect diets, but
differ dramatically in nutrient content, potentially affecting
symbiont genome evolution. For sap-feeding insects, sequenced
symbiont genomes are available only for phloem-feeding examples
from the suborder Sternorrhyncha and xylem-feeding examples from
the suborder Auchenorrhyncha, confounding comparisons. We
sequenced genomes of the obligate symbionts, Sulcia muelleri and
Nasuia deltocephalinicola, of the phloem-feeding pest insect,
Macrosteles quadrilineatus (Auchenorrhyncha: Cicadellidae). Our
results reveal that Nasuia-ALF has the smallest bacterial genome
yet sequenced (112 kb), and that the Sulcia-ALF genome (190 kb)
is smaller than that of Sulcia in other insect lineages.
Together, these symbionts retain the capability to synthesize the
10 essential amino acids, as observed for several symbiont pairs
from xylem-feeding Auchenorrhyncha. Nasuia retains genes enabling
synthesis of two amino acids, DNA replication, transcription, and
translation. Both symbionts have lost genes underlying ATP
synthesis through oxidative phosphorylation, possibly as a
consequence of the enriched sugar content of phloem. Shared
genomic features, including reassignment of the UGA codon from
Stop to tryptophan, and phylogenetic results suggest that
Nasuia-ALF is most closely related to Zinderia, the
betaproteobacterial symbiont of spittlebugs. Thus,
Nasuia/Zinderia and Sulcia likely represent ancient associates
that have co-resided in hosts since the divergence of leafhoppers
and spittlebugs >200 Ma, and possibly since the origin of the
Auchenorrhyncha, >260 Ma.",
month = aug,
year = 2013,
url = "http://dx.doi.org/10.1093/gbe/evt118",
keywords = "Nasuia deltocephalinicola; Sulcia muelleri; gene loss; genome
evolution; leafhopper; nutritional symbioses",
doi = "10.1093/gbe/evt118",
pmc = "PMC3787670",
pmid = 23918810,
issn = "1759-6653",
language = "en"
}
+67 -26
View File
@@ -47,7 +47,9 @@ P(n_K > 0) &= 1-e^{-\lambda} \\
\end{aligned}
$$
Assuming equal genome lengths, the number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
##### Assuming equal genome lengths
The number of genomes containing at least one occurrence of $K$ among the $S$ sequences follows a binomial distribution. The probability of observing $K$ in at least $r$ genomes is therefore
$$
@@ -56,11 +58,22 @@ $$
This sum is the regularized incomplete beta function $I_p(r,\,S-r+1)$, where
$$
I_p(r,S-r+1)
=
\frac{1}{B(r,S-r+1)}
\int_0^p t^{r-1}(1-t)^{S-r}\,dt
$$
and
$$
p = 1-e^{-2G/4^k}.
$$
Different genome sizes
the regularized incomplete beta function is available in standard statistical libraries.
##### Different genome sizes
When genome sizes differ, the probability of observing a given k-mer also differs between genomes. A simple approximation is to replace the individual genome sizes $G_i$ by their mean
@@ -82,39 +95,63 @@ Using the mean genome size therefore provides a conservative estimate of the FDR
When genome sizes differ substantially, however, this approximation can become overly conservative. In that case, the exact Poisson-binomial distribution can be computed directly from the individual genome-specific probabilities $p_i$.
### Exact tail probability
##### Exact tail probability
With genome-specific probabilities $p_i$, the number of genomes containing the k-mer follows a Poisson-binomial distribution. Writing the probability of exactly $j$ presences as a sum over all subsets of $j$ genomes would be impractical, since there can be up to $2^S$ subsets. The distribution can instead be obtained by adding the genomes one at a time.
Let $d_j$ be the probability that exactly $j$ of the genomes processed so far contain the k-mer. Adding genome $i$ leaves $j$ unchanged if the genome does not contain the k-mer, and moves $j-1$ to $j$ if it does:
Let $P_i(j)$ denote the probability that the k-mer occurs in exactly $j$ of the first $i$ genomes. Initially,
$$
P_0(0)=1,\qquad P_0(j)=0 \quad (j>0).
$$
When genome $i$ is added, the k-mer is either absent, with probability $1-p_i$, or present, with probability $p_i$. The probability of observing exactly $j$ presences among the first $i$ genomes is therefore
$$
P_i(j)=(1-p_i)P_{i-1}(j)+p_iP_{i-1}(j-1).
$$
This dynamic programming algorithm requires $O(S^2)$ operations and $O(S)$ memory. However, to compute $P(X\geq r)$, only the probabilities for fewer than $r$ presences need to be tracked, since
$$
P(X\geq r)=1-\sum_{j=0}^{r-1}P_S(j).
$$
This reduces the computational cost to $O(Sr)$ operations and $O(r)$ memory.
#### Distinguishing random from non-random k-mer sharing
The probability $P(X\geq r)$ describes the occurrence of a given k-mer in at least $r$ genomes under the random-sequence model. However, the index contains only k-mers observed in at least one genome. Therefore, when considering the k-mers recorded in the index, the relevant probability is the conditional probability:
$$
q_r = P(X\geq r\mid X\geq 1)
= \frac{P(X\geq r)}{P(X\geq 1)}.
$$
Let $k_r$ be the number of distinct k-mers observed in at least $r$ genomes, among the $k_1$ distinct k-mers recorded in the index. The observed proportion of k-mers shared by at least $r$ genomes is:
$$
O_r = \frac{k_r}{k_1}.
$$
Under the random-sequence model, the expected proportion is $q_r$. We can therefore quantify the deviation from random expectations using the difference:
$$
\left \{
\begin{aligned}
d'_j &= d_j(1-p_i) + d_{j-1}p_i, \\
d_0&=1
D_r &= O_r-q_r \\
&= \frac{k_r}{k_1}-q_r.
\end{aligned}
\right .
$$
before processing the first genome.
Positive values of $D_r$ indicate that k-mers are shared across genomes more frequently than expected by chance, whereas negative values indicate less frequent sharing. A value of zero indicates agreement with the random-sequence model.
The table $d$ only needs to record how many genomes contain the k-mer, not which ones, reducing the computational cost to $O(S^2)$. The result is independent of the order in which the genomes are processed and of the k-mer itself, so it can be computed once per index. The upper tail is then
Since both $O_r$ and $q_r$ are proportions between zero and one, $D_r$ is bounded:
$$
P(X\geq r)=1-\sum_{j=0}^{r-1}d_j.
-1\leq D_r\leq 1.
$$
Only $d_0,\ldots,d_{r-1}$ are needed, reducing the cost to $O(Sr)$.
When all $p_i$ are small, as is typically the case for large $k$, the Poisson-binomial distribution can also be approximated by a Poisson distribution with mean
$$
\lambda=\sum_{i=1}^{S}p_i.
$$
The approximation error is bounded in terms of $\sum_i p_i^2$ by Le Cam's inequality [@LeCam1960-pe].
More precisely, for a given $q_r$, its bounds are $-q_r$ and $1-q_r$. By construction, $D_1=0$, since all k-mers recorded in the index occur in at least one genome. Unlike a ratio-based measure, this difference remains defined even when no k-mers are observed in at least $r$ genomes. It thus provides a bounded measure of the excess or deficit of k-mer sharing relative to the random-sequence model.
*******
@@ -150,11 +187,12 @@ Consequently, the appropriate value of $k$ depends on the total amount of sequen
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Mycoplasma genitalium* having a size $G=580\,Mb$ [@Fraser1995-jc], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
Considering that bacterial genomes are typically a few megabases long, with the smallesr plublished today *Nasuia deltocephalinicola* having a size $G=112\,Kb$ [@Bennett2013-cl], and the largest $13\,Mb$ in *Sorangium cellulosum* [@Schneiker2007-gs]. For such genomes, the space of 11-mers is already to small to get a reasonable false positive rate:
$$
\begin{aligned}
P(C\geq1) &= 1- e^{-\frac{580.000^2}{2 \cdot 4^{11}}} \\
P(C\geq1) &= 1- e^{-\frac{112\,000^2}{2 \cdot 4^{11}}} \\
&= 1- e^{-1.5\times10^{3}} \\
&\simeq 1
\end{aligned}
$$
@@ -182,7 +220,7 @@ The distinction between the simple $4^k$ model and the Karlin--Ost framework is
These scales have very different implications for different genome sizes.
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995; @Karlin1997]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
The random-sequence calculation is only a null model. Real genomes exhibit strong and reproducible oligonucleotide composition biases. Karlin and colleagues showed that relative abundances of short oligonucleotides, in particular dinucleotides, form characteristic genomic signatures that remain relatively stable within genomes and differ between organisms [@Karlin1995-lt; @Karlin1994-on]. More generally, short k-mer frequencies can therefore capture compositional properties of genomes rather than provide highly specific sequence identifiers.
This distinction is important for whole-genome k-mer comparison. Increasing $k$ increases the theoretical diversity of the k-mer space exponentially and generally increases the proportion of k-mers that occur at unique genomic positions. Conversely, shorter k-mers occur more frequently and are more strongly influenced by genomic composition and repeated sequence. Empirical studies of k-mer spectra show this transition clearly: for example, k-mers of length $k\leq11$ can behave primarily as genomic-composition signatures, whereas longer k-mers progressively provide greater sequence specificity.
@@ -232,7 +270,7 @@ $$
The factor $(1-\lambda^{*})$ multiplying $S_1(N,k)$ is the key correction over the naive pairwise-collision count: it de-clumps overlapping candidate matches, which are not independent events, into an effective Poisson rate. This theorem is stated for $s\geq2$ independent sequences (e.g. comparing two distinct genomes); the intra-sequence self-repeat case (a single genome compared to itself) is treated separately, via the reverse-complement match length $R(N)$, in [@Karlin1985].
<!--
$o(1)$ (« petit o de 1 ») désigne une quantité qui **tend vers 0 quand $N\to\infty$**, mais dont on ne précise pas la vitesse exacte ni la formule exacte. C'est une façon d'écrire « il reste un terme d'erreur, mais il devient négligeable ».
Concrètement dans $S_1(N,k) = N^{r}\binom{s}{r}\gamma\,\lambda^{k}\,(1+o(1))$ :
@@ -265,7 +303,7 @@ Le rapport est exactement $S_1 / P(R=2\mid w) = 4^{k}$ — c'est-à-dire le nomb
- **Score de significativité a posteriori pour un k-mer effectivement observé dans l'index** (« ce hit précis, présent dans $r$ génomes sur $S$, est-il compatible avec le hasard ? ») → l'approche Poisson-binomiale est exactement le bon outil, et même préférable à Karlin-Ost pour cet usage : elle gère nativement des $G_i$ inégaux sans les subtilités de convergence du théorème 2.2, et elle donne directement une p-value par hit exploitable pour filtrer les résultats d'un index présence/absence — ce qui correspond bien à l'usage qu'a en tête ton correspondant.
Une seule réserve sur le texte proposé tel quel : $p_i\simeq1-\exp(-G_i/4^k)$ souffre du même biais que celui identifié plus tôt dans notre discussion — un mot de longueur $k$ n'est pas indépendant à chaque position (chevauchement), donc ce $p_i$ est un peu optimiste (légèrement trop haut) pour les mêmes raisons que $E[C]$ l'était. Négligeable pour $k\geq21$, plus discutable à $k=11$.
-->
## Chance sharing across genomes and the FDR of homology
@@ -359,6 +397,7 @@ Chance alone produces millions of pairwise shared 21-mers at this genome size, a
---
<!-- Notes perso (bac à sable), non publiées dans le HTML
Je n'ai rien modifié. Voici la formalisation des deux points, à valider avant de l'écrire dans `kmers.md`.
## 1. FDR d'un ensemble de k-mers sélectionnés
@@ -414,3 +453,5 @@ Cette somme porte sur $\binom{k+3}{3}$ compositions, soit 5 984 pour k = 31. Ell
**Limites à signaler dans le texte.** Markov 0 ignore les dinucléotides, comme les CpG ou les répétitions, et le $M$ du modèle nul reste celui de l'espace total.
Veux-tu que j'écrive ces deux sections, avec les corrections de numérique évoquées plus haut (`expm1`, somme directe de la queue) ?
-->
@@ -34,7 +34,7 @@ $$
D = \sum_i \mathbb{1}[a_i \ne b_i],
$$
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded.
where $a_i,b_i\in\{0,1\}$ indicate the presence of k-mer $i$ in the two genomes. It counts the number of k-mers for which the two incidence vectors differ, without normalization by the size of the k-mer space. As an $L^1$ distance, Hamming distance is a metric. Unlike abundance-based measures, it cannot be recovered from k-mer counts alone once the presence/absence representation has been discarded. On an index that stores counts, a k-mer is taken as present when its count reaches the presence threshold (`--presence-threshold`, 1 by default).
**Mash** is not an independent set dissimilarity but a nonlinear transformation of the Jaccard similarity [@Ondov2016-dq]. If
@@ -24,4 +24,4 @@ If the measured variance $V$ doesn't exceed the Poisson floor $\hat\mu/\bar L$ (
**This is not Jin & Nei's (1990) original estimator.** Their publication defines the $+\Gamma$ distance formula itself and, absent an estimate, recommends the fixed default $\alpha = 1$ — it does not propose a way to estimate $\alpha$ from data. The method-of-moments estimator above is a standard consequence of the Poisson–Gamma relationship between substitution counts and gamma-distributed rate variation, applied per-partition; it is `obikmer`'s own addition, not part of the cited correction.
See [`phylo`](../../../usage/phylo.md#distance-matrix---distance) for how to select a fixed $\alpha$ or request automatic estimation.
See [`phylo`](../../../usage/phylo.md#distance-estimation-and-tree-reconstruction) for how to select a fixed $\alpha$ or request automatic estimation.
@@ -31,4 +31,4 @@ How well a pair of genomes is represented in a sample depends on how many famili
By default, sampling draws families uniformly. Weighting instead by informativeness biases the draw toward a target entropy $\mu$ with a Gaussian curve of width $\sigma$: a family's probability of being kept scales with $\exp\!\left(-\dfrac{(\text{entropy15} - \mu)^2}{2\sigma^2}\right)$ — a soft preference, not a hard cutoff, so no family is categorically excluded purely for having low or high entropy.
See [`phylo`](../../../usage/phylo.md#sampling-at-scale---n-sites---shannon---entropy) for how to bound and bias the sample, and for session/checkpointing behavior.
See [`phylo`](../../../usage/phylo.md#snp-character-selection-and-sampling) for how to bound and bias the sample.
+112 -91
View File
@@ -1,6 +1,13 @@
# phylo
Compute pairwise distances between the genomes stored in an index, optionally build trees (NJ/UPGMA) from them, and optionally calibrate a 16-state parsimony model for a central-position SNP character with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
Compare the genomes stored in an index and build phylogenetic trees from them. Two complementary approaches are available:
- **Whole-genome distances** compare the k-mer content of the genomes, without any alignment.
- **SNP-based distances and analyses** compare the nucleotide variants found at the central position of related k-mers.
Distance matrices can be used to build Neighbor-Joining or UPGMA trees. SNP-based analyses can also produce a pseudo-alignment and calibrated substitution costs, with exports for external phylogenetic tools (TNT, PhyG, IQ-TREE).
## Usage
```bash
obikmer phylo INDEX [OPTIONS]
@@ -12,28 +19,43 @@ obikmer phylo INDEX [OPTIONS]
|---|---|
| `INDEX` | Index directory |
## Distance matrix (`--distance`)
## General options
| Option | Default | Description |
|---|---|---|
| `--distance` | none | Compute and write a distance matrix. See the two tables below for the full list of accepted values. No matrix is computed unless this is given, or `--nj`, `--upgma` or `--shared-kmers` is (they need a matrix and then use `jaccard`) |
| `--cpu-max N` | all available cores | Maximum number of CPU cores to use. It caps every thread pool of the process and can only lower the number of cores, never raise it above those available. Common to all subcommands |
| `-o, --output PREFIX` | none | Prefix of the output files. Without it, the distance matrix goes to standard output and every other file is written in the current directory under its name without the prefix. See "Output files" below |
## SNP families
SNP-based analyses use groups of k-mers that differ only at their central nucleotide. These groups are called families (see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the exact definitions and eligibility rules). A variable family is one character of the analysis, and the variants a genome carries give its state for that character.
## Distance estimation and tree reconstruction
| Option | Default | Description |
|---|---|---|
| `--distance` | none | Compute and write a distance matrix. The accepted values are listed just below the table. No matrix is computed unless this is given, or `--nj` or `--upgma` is (they need a matrix and then use `jaccard`) |
| `--gamma-shape ALPHA\|auto` | none | Rate-heterogeneity correction, for `snp-*` values that support it (see [Gamma rate heterogeneity](../theory/phylogeny/snp_based/gamma_rate_heterogeneity.md)). Either a fixed $\alpha$ or `auto`/`estimate` to fit it from the data. No effect on the other values; rejected if given together with a value that doesn't support it |
| `--presence-threshold` | `1` | Minimum count for a kmer to be considered present, for `jaccard`/`mash` on a count index |
| `--presence-threshold` | `1` | How counts are turned into presence/absence on an index that stores counts: a kmer is present in a genome only when its count reaches this value. Used by `jaccard`, `mash` and `hamming`, and by the shared-kmer counts of `--shared-kmers`. No effect on an index without counts, nor on the distances that use the counts themselves |
| `--session DIR`, `--session-force` | none | Save and reuse the sample of a `snp-*` distance. Currently unavailable, see `--session` below |
| `--csv` | off | Write the matrix as plain CSV instead of the default relaxed-PHYLIP format |
| `--shared-kmers` | off | Also write the shared-kmer count matrix. Only valid with a whole-index distance, not a `snp-*` value |
| `--shared-kmers` | off | Write the shared-kmer count matrix (`<prefix>_shared.csv`). It can be used alone, without any distance, or together with a whole-index distance; it is refused with a `snp-*` value |
| `--nj` | off | Compute and write a Neighbor-Joining tree (Newick) |
| `--upgma` | off | Compute and write a UPGMA tree (Newick) |
| `-o, --output` | none (stdout) | Output file prefix |
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `hamming` requires a presence/absence index, the others work on either index type), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
Every value routes to one of two independent computations: **whole-index distances** (`jaccard`, `mash`, `hamming`, `bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean` — see [Whole k-mer set distances](../theory/phylogeny/kmer_based/kmer_set_distances.md) for the definitions; `jaccard`, `mash` and `hamming` compare which kmers are present and work on either index type; the six others (`bray-curtis`, `relfreq-bray-curtis`, `euclidean`, `relfreq-euclidean`, `hellinger`, `hellinger-euclidean`) use the counts and need an index that stores them), or **`snp-*` corrections** (`snp-raw`, `snp-jc`, `snp-k2p`, `snp-k81`, `snp-f81`, `snp-t92`, `snp-tn93`, `snp-tv` — see the [central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the definitions).
`snp-*` values require the sibling annex (`--sibling-annex`, below) and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "Sampling at scale" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
`snp-*` values read the sibling annex (below), which is built automatically the first time it is needed, and are, by default, computed exhaustively over every non-monomorphic family in the index; add `--n-sites N` to bound the computation to approximately `N` families instead (see "SNP character selection and sampling" below — the same option `--pseudo-alignment`/`--sankoff` use, but optional here).
### Output
### Matrix format
When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHYLIP format (`n` on the first line, then one `label<TAB>value...` row per genome). With `--csv`, the format is instead a header row `genome,<label1>,<label2>,...` followed by one `<label>,<value1>,<value2>,...` row per genome, 6 decimals. Both formats are symmetric with a zero diagonal, except where noted below.
## `--exclude-genome`, `--min-shared-family`
### `--session`: currently unavailable
`--session DIR` and `--session-force` are meant to save the sample of a `snp-*` `--distance` run in a directory, so that a later run with identical selection parameters reuses it instead of drawing again. This feature is being rebuilt and is not available for now: giving either option stops the command with an error. Until it returns, keep the output files of a run (the alignment, or the distance matrix) rather than relying on re-drawing the same sample.
## Genome selection and filtering
| Option | Description |
|---|---|
@@ -42,21 +64,79 @@ When a matrix is computed and there is no `-o`, it goes to stdout in relaxed-PHY
Neighbor-Joining and UPGMA trees (`--nj`/`--upgma`) are always built from every genome in the index, regardless of `--exclude-genome`/`--min-shared-family`.
## Central-position SNP model
## SNP character selection and sampling
See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md) for the family/eligibility definitions. Requires the sibling annex, built once per index:
SNP-based analyses can use every eligible variable family, or a sample drawn from the index. The options below control which characters are selected, which genomes and states are kept, and how the sample is spread over the index.
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags.
Every invocation draws its own fresh random sample: running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately.
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
| Option | Default | Description |
|---|---|---|
| `--n-sites N\|all` | none (mandatory for `--pseudo-alignment` and `--sankoff`) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
### `--quartet`: keeping only families that can resolve a tree
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
### `--pair-balance`: representing every pair of genomes
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
### Describing a draw: `--sample-stats`, `--sample-plan`
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
`--sample-stats` writes three files:
| File | Content |
|---|---|
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. A site is informative under the same rule as `--quartet`: without `--free-loss`, absence is a state of its own and can form one of the two pairs; with `--free-loss` it is missing data and cannot. Excluded genomes are listed with zeros in the first file and count nowhere.
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
## SNP family preparation and diagnostics
SNP-based options read precomputed tables describing the families: the sibling annex, and tables derived from it (entropy, quartet, family overlap, pair weights). They are built automatically the first time they are needed and reused afterwards. The options below prepare them in advance, or describe the families.
| Option | Description |
|---|---|
| `--sibling-annex` | Build (or rebuild) the sibling-count/minorant annex. Rebuilding it makes the annexes derived from it obsolete, so they are dropped; then only the derived annexes that the other options of the same command read are rebuilt, each under the `--no-ambiguity` / `--only-diploid` convention given: `--entropy` the entropy annex, `--quartet` the quartet annex, `--pair-balance` the pair-weight and family-overlap annexes, `--family-overlap` or `--min-shared-family` the family-overlap annex. For example `--sibling-annex --quartet --only-diploid` prepares exactly the quartet annex of that convention, and `--sibling-annex` alone prepares none. An annex that is not prepared is built later, on demand, by the computation that reads it. Every option in this section needs the sibling annex, and builds it by itself, under an exclusive lock on the index, if it is missing (as it is after `migrate-mphf`); running `--sibling-annex` first does the same work once, which is the safe way to start several `phylo` runs on the same index at the same time |
| `--sibling-stats` | Write `<prefix>_siblings.csv`: the family-size distribution, per genome and globally |
| `--sibling-hist` | Print the global family-size histogram (1-4 members) only |
| `--sibling-hist` | Write `<prefix>_sibling_hist.csv` (columns `size,count`): the global family-size histogram (1-4 members) only, without the per-genome breakdown |
| `--family-overlap` | Write `<prefix>_family_overlap.csv`: for every genome pair, how many variable families both genomes carry a call for |
| `--shannon` | Write `<prefix>_entropy.csv`: per-family Shannon entropy, one row per family, full unsampled scan |
| `--shannon-summary` | Write `<prefix>_shannon_summary.csv`: the same scan as `--shannon`, reduced to a distribution (see below) — a few hundred thousand lines at most, whatever the size of the index |
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
### `--sibling-stats`
@@ -73,50 +153,6 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
`--min-shared-family N` uses the mean of each genome's own row (excluding the diagonal) against this same matrix as its exclusion statistic. There is no universal value for `N` — inspect `--family-overlap`'s own output to find where the real gap sits in a given genome collection before choosing a threshold.
### `--pseudo-alignment`
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
#### VCF output (`--vcf`)
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
| Option | Default | Description |
|---|---|---|
| `--n-sites N\|all` | none (mandatory here) | Number of families to draw, or `all` to use every variable family that passes the other filters, with no sampling. A number is a target, not a guarantee. |
| `--free-loss` | off | Treat a genome carrying none of a family's observed members as missing data (`?`) instead of `-` |
| `--no-ambiguity` | off | Treat a genome carrying more than one member of a family as missing data (`?`) instead of an IUPAC ambiguity code |
| `--only-diploid` | off | Milder than `--no-ambiguity` (the two cannot be combined): keep the genomes carrying one or two members of a family (a base, or a two-base IUPAC code such as `R`) and treat those carrying three or four as missing data (`?`) |
| `--quartet` | off | Keep only the families that can separate two pairs of genomes from each other: at least two distinct states, each carried by at least two genomes. Absence counts as a state unless `--free-loss` is given. Which genomes count as carriers follows `--no-ambiguity` |
| `--pair-balance` | off | Favour the families covering the pairs of genomes that share the fewest families across the index, so that every pair is better represented in the sample. Which genomes count as carriers follows `--no-ambiguity` |
| `--sample-stats` | off | Write three CSV files describing what the draw contains (see "Describing a draw" below) |
| `--sample-plan` | off | Write `<prefix>_sample_plan.csv`: how the draw was spread over the layers of the index |
| `--entropy MU` | off (`1.0` if only `--entropy-sd` is given) | Center of the entropy band to favor when sampling |
| `--entropy-sd SIGMA` | off (`0.5` if only `--entropy` is given) | Width of that band |
## Sampling at scale: `--n-sites`, `--shannon`, `--entropy`
`--n-sites`, `--free-loss`, `--no-ambiguity`, `--quartet`, `--pair-balance`, `--entropy`/`--entropy-sd` are shared by `--pseudo-alignment`, `--sankoff` (and everything it implies: `--tnt`/`--phyg`/`--iqtree`), and a `snp-*` `--distance` value — one draw feeds all of them in a single invocation. `--n-sites` is mandatory for `--pseudo-alignment`/`--sankoff`, where it takes either a number of sites or `all`; for a `snp-*` `--distance` value it is optional, and omitting it is the same as `all` (every non-monomorphic family in the index, not an approximation).
Combining `--sankoff` (or `--tnt`/`--phyg`/`--iqtree`) with a `snp-*` `--distance` value in the same command reuses that one draw for both — the distance and the Sankoff calibration/alignment are guaranteed to be computed from the *identical* set of sampled sites, never two independent samples, so the two outputs are directly comparable. This only holds within a single command; running them as two separate `obikmer phylo` invocations draws two independent samples even with the same flags — unless `--session` is used (below).
Every invocation, with or without `--session`, draws its own fresh random sample by default — running the same command twice gives two different (but equally valid) samples, which is useful for measuring sampling variance and is kept that way deliberately. `--session` does not change this: it makes a *specific* sample reusable on request, it does not make sampling itself reproducible from one independent run to the next.
### `--session`: reusing a sample across separate commands
| Option | Default | Description |
|---|---|---|
| `--session DIR` | none | Persist the sample (and, for `--sankoff`, its calibration/alignment) in `DIR` so a later, separate `obikmer phylo` invocation with the exact same selection parameters restores it instead of resampling |
| `--session-force` | off | With `--session DIR`: overwrite its saved parameters and cached sample instead of erroring out when this run's parameters don't match. No effect without `--session` |
`DIR` is created if it doesn't exist. If it already holds a sample built with different `--n-sites`/`--free-loss`/`--no-ambiguity`/`--quartet`/`--exclude-genome`/`--min-shared-family`/`--entropy`/`--entropy-sd` values than this run, the command exits with an error rather than silently using either the old or the new values — pass `--session-force` to discard the old sample and rebuild under the new parameters, or point `--session` at a different directory to keep both.
A `--sankoff`-family run and a `snp-*` `--distance` run share the same cached sample when pointed at the same `--session DIR` — build it once with either, reuse it from the other, in either order, across separate commands.
If a run using `--session` is interrupted (crash, kill, `Ctrl-C`), the next run against the same `--session DIR` resumes from the last automatic checkpoint (roughly every 8 partitions'-worth of sampling progress) instead of starting over. The resumed run's sample is **not** guaranteed to be identical to what an uninterrupted run would have produced past that checkpoint — each process draws its own independent random sequence, same as any two separate invocations do — but nothing already checkpointed is lost, and no work needs redoing beyond that point.
With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`), every variable family (family size ≥ 2) that passes the other filters is used. With `--n-sites N`, roughly `N` families are kept instead (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for how the draw stays representative of the whole index). If fewer than `N` candidate families remain, `--n-sites` has no effect.
### `--shannon`: measuring how informative a family is
`<prefix>_entropy.csv` has one row per family visited:
@@ -134,43 +170,27 @@ With `--n-sites all` (or, for a `snp-*` `--distance` value, without `--n-sites`)
`--shannon` always scans every non-monomorphic family of the whole index; `--n-sites`, `--entropy` and `--entropy-sd` have no effect on it. The output therefore grows with the index, with one row per family, and can reach hundreds of millions of rows (tens of gigabytes) on a large index. Analyse it with a tool that reads the file as a stream (for example DuckDB or Miller) rather than loading it whole. It is nevertheless the right basis for choosing `--entropy`/`--entropy-sd` values, since it gives the complete entropy distribution, including how it varies with `n_genomes_present`, rather than an estimate from a sample.
### `--quartet`: keeping only families that can resolve a tree
A family carried by fewer than four genomes, or whose genomes all share one state except a single variant, fits every tree topology equally well and so tells a tree nothing: to separate two pairs of genomes from each other (`AB|CD`) it takes at least two distinct states, each carried by at least two genomes. `--quartet` restricts the draw to the families that show this pattern. The states are those of the alignment. Without `--free-loss`, the absence of a family is a state of its own, so two genomes that carry a base and two genomes that lack the family do form two pairs. With `--free-loss`, absence is missing data and never forms a pair. A genome carrying several members of the family counts as a carrier of its own combination of bases, unless `--no-ambiguity` is given, in which case it is left out, as in the alignment.
The restriction is applied before the draw, from a persisted annex (one bit per variable family, built the first time `--quartet` is used with a given `--no-ambiguity` / `--only-diploid` convention and with or without `--free-loss`, in one pass shared with the other annexes, or ahead of time by `--sibling-annex --quartet`), then checked exactly on each family drawn, taking `--exclude-genome` into account. The quota given to `--n-sites` is therefore spent only on usable families.
### Describing a draw: `--sample-stats`, `--sample-plan`
Both need a draw to describe (`--pseudo-alignment`, `--sankoff` or a `snp-*` `--distance`) and are computed from the very sites the draw kept, in the same pass, so they cost almost nothing and always agree with the alignment.
`--sample-stats` writes three files:
| File | Content |
|---|---|
| `<prefix>_sample_genomes.csv` | per genome (`genome,excluded,sites_with_call,share_of_sites,informative_sites`): the sampled sites it has a call in, their share of all sampled sites, and the informative sites (two states each carried by at least two genomes) in which its own state is one of those carried by at least two genomes. A genome with almost no calls cannot be placed whatever the tree method |
| `<prefix>_sample_carriers.csv` | per number of carriers (`n_carriers,sites,informative_sites`): how many sampled sites have that many genomes with a call, and how many of them are informative. Shows how much of the sample can resolve a tree at all |
| `<prefix>_sample_overlap.csv` | the sample's own genome × genome matrix, same layout as `--family-overlap`: sampled sites both genomes have a call in. Compare it with the index's matrix to see which pairs the draw under- or over-represents |
A call is any state other than the empty one: absence is never a call, with or without `--free-loss`. Excluded genomes are listed with zeros in the first file and count nowhere.
`--sample-plan` writes `<prefix>_sample_plan.csv`, one line per layer of the index in the order the draw visited them (`partition,layer,eligible,mass,quota,resolved,kept`): the variable families eligible when the draw starts, their total weight (their number, or with `--pair-balance` the sum of their pair weights), the layer's share of the target, the families drawn and resolved, and the ones kept after masking. A low `kept` against `resolved` means masking removes most of what is drawn in that layer; a `quota` summing to less than the target means some layers had nothing eligible. The target and the total kept are logged.
### `--shannon-summary`
`--shannon-summary` writes `<prefix>_shannon_summary.csv` (`entropy,n_genomes_present,n_genomes_unambiguous,bin_start,n_families`): the number of variable families per entropy (`entropy15` over every carrier, `entropy4` over the single-member genomes only — families with none are left out of its lines), number of genomes carrying the family, number of those without ambiguity, and 0.05-bit entropy bin (`bin_start` is its lower bound). It answers how entropy varies with the number of carriers, or how many carriers `--no-ambiguity` loses, from the same scan as `--shannon` but without one line per family.
### `--pair-balance`: representing every pair of genomes
## SNP pseudo-alignment
Two genomes that are distantly related share few families, so a uniform draw covers such a pair with few sites, far fewer than a close pair: the pairs that matter for the deep branches of a tree are the worst represented. `--pair-balance` compensates. Each family is weighted by the mean, over the pairs of genomes carrying it, of 1 divided by the number of families that pair shares across the whole index (the matrix `--family-overlap` writes): a family carried by a rarely-sharing pair weighs more. It is the mean, not the sum, so a family carried by most genomes — in skimming data often a high-copy region — is not favoured merely for covering many pairs.
A pseudo-alignment has one column per variable family and one state per genome. Unlike a conventional sequence alignment, its columns come directly from groups of related k-mers of the index. The number of sites is set with `--n-sites`, and the options of "SNP character selection and sampling" above control which families are included and how ambiguous or absent states are written.
Each layer's share of the draw is proportional to the total weight of its families, and a family's chance of being drawn is proportional to its own weight. The weights are persisted (one value per variable family, built the first time `--pair-balance` is used with a given `--no-ambiguity` / `--only-diploid` convention, after the family-overlap matrix it depends on, or ahead of time by `--sibling-annex --pair-balance`). They combine with `--entropy` and `--quartet`: the weight multiplies the entropy kernel, and `--quartet` restricts the families first.
| Option | Description |
|---|---|
| `--pseudo-alignment` | Write `<prefix>_alignment.fasta`: a SNP-only pseudo-alignment. Requires `--n-sites N\|all` |
| `--fasta` | Choose FASTA as the pseudo-alignment format (the default when neither `--fasta` nor `--vcf` is given). Implies `--pseudo-alignment` |
| `--vcf` | Also (or instead) write `<prefix>_alignment.vcf`, the same sites as a VCF. Implies `--pseudo-alignment`; combinable with `--fasta` |
### `--entropy MU` / `--entropy-sd SIGMA`: biasing the sample toward informative families
### `--pseudo-alignment`
By default, `--n-sites N` draws families uniformly. With `--entropy`/`--entropy-sd`, each family's chance of being kept is instead weighted by how close its own entropy (`entropy15`) is to `MU`, using a Gaussian curve of width `SIGMA` — no hard cutoff (see [Sampling design](../theory/phylogeny/snp_based/sampling_design.md) for the weighting formula). The filter activates as soon as either flag is given; the other defaults to `1.0`/`0.5`. Combine with `--n-sites N` (expect somewhat fewer than `N` families kept in practice) or use alone (a soft filter over the whole index, no size target).
`<prefix>_alignment.fasta` — one record per non-excluded genome, one column per variable family (family size ≥ 2). Each site is IUPAC-coded from the genome's presence mask at that family: a single observed form → the plain base; several forms → the matching IUPAC ambiguity code; no form → `-`.
The first `phylo` run on a given index that uses `--entropy`/`--entropy-sd` pays a one-time extra cost (every candidate family's entropy is computed once and saved alongside the index); later runs, even with different `MU`/`SIGMA`, reuse that saved data.
#### VCF output (`--vcf`)
The entropy that weights the draw follows `--no-ambiguity` and `--only-diploid`, so that it describes the same genomes as the alignment. By default, every genome carrying the family counts, and a genome carrying several members of it is a state of its own (`entropy15`). With `--no-ambiguity`, only the genomes carrying exactly one member count, since the others become missing data (`entropy4`). With `--only-diploid`, the genomes carrying one or two members count and those carrying three or four become missing data. The three entropies are saved separately: switching convention builds the missing one once, without discarding the others, and `--sibling-annex --entropy` builds the one of the convention given.
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
## Sankoff calibration and phylogenetic exports
@@ -205,7 +225,7 @@ TNT and PhyG both write trees with bare numeric leaf labels (`1`, `2`, ..., in t
## Output files
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, only the distance matrix, if one is computed, is produced, on stdout. All matrices use genome labels as row/column headers, in index order.
With `-o/--output PREFIX`, the relevant subset of the files below is written. Without `-o`, the distance matrix goes to standard output and every other file is written in the current directory under its name without the `<prefix>_` part (for example `nj.nwk`, `entropy.csv`, `alignment.fasta`). All matrices use genome labels as row/column headers, in index order.
### Distance matrix
@@ -219,13 +239,14 @@ With `-o/--output PREFIX`, the relevant subset of the files below is written. Wi
CSV matrix layout (`_dist.csv`, `_shared.csv`, `_family_overlap.csv`): header `genome,<label1>,<label2>,...`, one data row per genome, `<label>,<value1>,<value2>,...`.
### Central-position SNP model
### SNP analyses
| File | Written by | Format | Content |
|---|---|---|---|
| `<prefix>_siblings.csv` | `--sibling-stats` | CSV table | family-size distribution, per genome and global |
| `<prefix>_sibling_hist.csv` | `--sibling-hist` | CSV table | global family-size histogram (`size,count`) |
| `<prefix>_family_overlap.csv` | `--family-overlap` | CSV matrix | variable families both genomes of a pair carry a call for |
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see "Sampling at scale" above |
| `<prefix>_entropy.csv` | `--shannon` | CSV table | per-family Shannon entropy, see `--shannon` above |
| `<prefix>_shannon_summary.csv` | `--shannon-summary` | CSV table | families per entropy bin, number of carriers and number of single-member carriers |
| `<prefix>_sample_genomes.csv`, `<prefix>_sample_carriers.csv`, `<prefix>_sample_overlap.csv` | `--sample-stats` | CSV | what the draw contains: per genome, per number of carriers, per pair of genomes |
| `<prefix>_sample_plan.csv` | `--sample-plan` | CSV table | per layer: eligible families, weight, quota, drawn, kept |