Update the VCF export documentation to reflect that `-` represents an overlapping deletion (extra allele `*`) and introduce the `<ABS>` symbolic allele for absent families. Add a new dedicated documentation file for the VCF pseudo-alignment format, remove obsolete descriptions from phylo.md, and fix benchmark scripts to handle errors and directory creation correctly. - Update `obikmer` dependency from 1.4.1 to 1.4.2 in Cargo.toml and Cargo.lock - Add `UserDocMD/formats/vcf.md` describing VCF pseudo-alignment structure, alleles, and genotypes - Remove obsolete VCF descriptions from `UserDocMD/usage/phylo.md` and update ploidy/free-loss notes - Update `DevDocMD/theory/evolutionary_distances.md` to clarify `-` as overlapping deletion and `<ABS>` for others - Fix `benchmark/index_one_count.sh` to create index directory and check read command errors - Fix `benchmark/index_one_presence.sh` to create index directory and check simulation errors - Update `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` to use `<ABS>` for absent families instead of `*`
10 KiB
VCF pseudo-alignment
phylo --vcf writes the sites of a SNP pseudo-alignment as a VCF file, <prefix>_alignment.vcf. This page describes exactly what that file contains, because the k-mer families it is built from are not ordinary genomic variants and a few conventions follow from that.
Why a VCF, and what a site is
The pseudo-alignment (see Central-position SNP model) has one column per family: the up to four k-mers that share all their bases except the central one. A genome that carries one of these k-mers shows a particular base at the centre, and the columns of the alignment are those central bases, genome by genome. The FASTA output keeps this as a matrix of letters. The VCF output lists the same columns as sites with alleles and genotypes, which is the form that variant-oriented tools (bcftools, population-genetics software, genotype viewers) expect.
Nothing here comes from a reference genome. The file has no real coordinates, and the sites are not located on a chromosome: the positions below are labels that make the file valid and reproducible, not map positions.
Producing the file
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
--vcf implies --pseudo-alignment, so --n-sites N|all is required, together with an already-built sibling annex (see phylo). --fasta and --vcf can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (--free-loss, --no-ambiguity, --only-diploid, --quartet, --exclude-genome, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
Layout
An illustrative excerpt (columns are tab-separated in the file):
##fileformat=VCFv4.5
##source=obikmer
##contig=<ID=obikmer_0_0>
##contig=<ID=obikmer_0_1>
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype (diploid)">
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT idx0 idx1 idx2 idx3
obikmer_0_0 440 CGATTNGTTAA A G . . . GT 0/0 1/1 0/0 0/0
obikmer_0_0 1511 CACGANTGTAG A G,T . . . GT 0/0 0/0 1/1 2/2
obikmer_0_1 9 TTACGNAGCTA C * . . . GT 0/0 0/0 1/1 0/0
There is one sample column per genome that takes part in the draw, named after the genome label, in the index's own order. A genome removed with --exclude-genome or --min-shared-family has no column at all. QUAL, FILTER and INFO are always .: obikmer applies no variant filter and attaches no annotation, and leaving them empty keeps a file of millions of sites compact.
Coordinates: CHROM and POS
An index is organised in partitions, and each partition holds one or more layers of k-mers (see Index construction and on-disk layout). A family always belongs to exactly one layer of one partition, and the families of a layer are visited in a fixed order. The VCF uses that structure as coordinates:
CHROMisobikmer_<partition>_<layer>, one virtual contig per layer of each partition in which at least one site was drawn;POSis the rank of the family inside that layer, starting at 1.
For a given index and a given sibling annex, a family therefore always receives the same CHROM and POS, whatever the draw, the options or the number of sites requested. Two runs on the same index can be compared position by position. This stability is lost if the sibling annex is rebuilt, or if the index is rebuilt, merged or filtered, because the families are then enumerated again. Positions are never comparable between two different indexes; use ID for that. Even rebuilding an index from the very same genomes may enumerate the families in another order: on a small test index built twice from identical input, the family CGATTNGTTAA had POS 440 the first time and 371 the second, with the same ID.
The file is sorted by partition, then layer, then position, which is also the order of the ##contig lines, so tools that require a sorted file accept it directly. Positions are sparse: only the drawn families are written.
Identifier: ID
The ID column holds a label for the family that does not depend on the index. It is built from one of the family's k-mers by replacing its central base with N (for example CGATTNGTTAA for k = 11), after choosing, among the family's possible orientations, the one that sorts first. All the members of a family, whichever strand they were read on, give the same label.
This is what makes two files comparable even when they come from different indexes: the same family has the same ID in both, while its CHROM and POS will generally differ. It is also unique within a file, since two different families never share a label.
Alleles: REF and ALT
The alleles are the bases carried at the centre of the family's k-mers, as in the FASTA. Two points are worth knowing.
First, the base is the central base of the member's canonical form (the smaller of the k-mer and its reverse complement). That is not necessarily the strand of the ID label, so the letters should not be matched to the ID sequence by position; they are labels for the allele, not a reading of that sequence.
Second, REF carries no meaning of reference: it is simply the most frequent allele among the genomes of the draw, each allele copy counted once, with the alphabetical order A < C < G < T deciding ties. ALT lists the remaining observed bases in the order A, C, G, T, separated by commas. There is no option to choose a reference genome.
Absence: the <ABS> allele
A genome may carry none of the family's k-mers. Without --free-loss this absence is treated as a real state (the family was lost, or is absent from that genome), which the FASTA shows as -. In the VCF it becomes one more allele, the symbolic allele <ABS>, listed last in ALT and declared in the header by ##ALT=<ID=ABS,Description="Family absent from the genome">. A genome without the family receives the genotype of <ABS>; for a site with ALT equal to G,T,<ABS> that is 3/3. A site that differs between genomes only by presence versus absence has ALT equal to <ABS>.
The VCF specification gives the plain character * a meaning of its own (an allele missing because of an overlapping deletion described by another record), which is not the situation here, so it is deliberately not used. <ABS> is a free-form symbolic identifier, outside the reserved structural-variant names (DEL, INS, DUP, INV, CNV) and outside <*>, which belongs to reference blocks. Tools that only look at sequence alleles treat the site as having an alternative that is not a base.
With --free-loss, absence is declared missing data instead: <ABS> never appears, and the genomes concerned get a missing genotype.
Genotypes
Every genome gets one GT value per site. The ploidy of these values is a convention chosen from the options, and the header says which (Genotype (diploid) or Genotype (haploid)).
| Cell in the alignment | Diploid (default, --only-diploid) |
Haploid (--no-ambiguity) |
|---|---|---|
one base X |
X/X, for example 0/0 or 1/1 |
X, for example 0 |
two bases (IUPAC R Y S W K M) |
the heterozygote, for example 0/1 |
does not occur |
three or four bases (B D H V N) |
./. (not expressible) |
does not occur |
absent family (<ABS>) |
<ABS>/<ABS>, for example 3/3 |
the index of <ABS> |
missing (?) |
./. |
. |
The two allele indices of a diploid genotype are always written in increasing order (0/1, never 1/0), and the separator is / because no phasing information exists.
Why diploid by default? A genome can carry several forms of the same family, which the alignment records as an IUPAC ambiguity code. A diploid genotype can express up to two forms, and --only-diploid removes the genomes that would need more (three or four forms) before the sites are selected, so that no ./. comes from them. --no-ambiguity goes further and treats every such genome as missing, which makes all genotypes haploid.
This is a convention, not a measurement. The file records which forms a genome carries, not how many copies of each: A/A/G and A/G/G both give the code R and the genotype 0/1. Likewise, a genome with two forms may be heterozygous at one locus, or it may carry two similar regions of its genome (paralogues, repeats); the file cannot tell these apart. Do not read the genotypes as calls from sequencing reads.
Which sites are written
Only the families that survive the draw and its masking appear. Within those:
- a site needs at least one genome with a base or (in diploid mode) a two-base code, because
REFis a base; a site where every genome is missing, absent or has an ambiguity code too wide for the mode is left out of the VCF while remaining in the FASTA; - the log line at the end of the run gives the number of sites written out of the number drawn, so the difference is visible.
A site is selected because its genomes show at least two distinct states, and absence counts as a state. A site that differs between genomes only by presence versus absence is therefore a legitimate site, written with ALT equal to <ABS>.
Using the file with other tools
- The contigs are virtual and have no declared length, and the file refers to no reference. Tools that need one (annotation, liftover) cannot use it; tools that only read sites and genotypes (
bcftools view,query,stats, population-genetics software, PLINK-style converters) can. - If a tool rejects multiallelic sites or the
<ABS>allele,bcftools view -m2 -M2keeps biallelic sites only, andbcftools view -e 'ALT="<ABS>"'removes the sites involving absence. The file as written was read bybcftoolsand indexed bytabixwithout complaint on a small test index, in both diploid and haploid modes.bcftools statscounts the sites with<ABS>among the "others" (symbolic alleles) rather than as SNPs. - If a tool expects haploid data, run
phylowith--no-ambiguityrather than converting the genotypes afterwards.
See also
phylo: all the options that shape the draw.- Sampling design: how the sites are drawn.
- Central-position SNP model: what a family is.