docs: clarify VCF pseudo-alignment format and update obikmer dependency
Update the VCF export documentation to reflect that `-` represents an overlapping deletion (extra allele `*`) and introduce the `<ABS>` symbolic allele for absent families. Add a new dedicated documentation file for the VCF pseudo-alignment format, remove obsolete descriptions from phylo.md, and fix benchmark scripts to handle errors and directory creation correctly. - Update `obikmer` dependency from 1.4.1 to 1.4.2 in Cargo.toml and Cargo.lock - Add `UserDocMD/formats/vcf.md` describing VCF pseudo-alignment structure, alleles, and genotypes - Remove obsolete VCF descriptions from `UserDocMD/usage/phylo.md` and update ploidy/free-loss notes - Update `DevDocMD/theory/evolutionary_distances.md` to clarify `-` as overlapping deletion and `<ABS>` for others - Fix `benchmark/index_one_count.sh` to create index directory and check read command errors - Fix `benchmark/index_one_presence.sh` to create index directory and check simulation errors - Update `src/obikphylo/src/siblings/algorithms/alignment_vcf.rs` to use `<ABS>` for absent families instead of `*`
This commit is contained in:
1 parent
5368c98875
commit
0134c69cf4
9 files changed
+156
-34
No files matched your search
@@ -0,0 +1,107 @@
|
||||
# VCF pseudo-alignment
|
||||
|
||||
[`phylo --vcf`](../usage/phylo.md) writes the sites of a SNP pseudo-alignment as a VCF file, `<prefix>_alignment.vcf`. This page describes exactly what that file contains, because the k-mer families it is built from are not ordinary genomic variants and a few conventions follow from that.
|
||||
|
||||
## Why a VCF, and what a site is
|
||||
|
||||
The pseudo-alignment (see [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md)) has one column per *family*: the up to four k-mers that share all their bases except the central one. A genome that carries one of these k-mers shows a particular base at the centre, and the columns of the alignment are those central bases, genome by genome. The FASTA output keeps this as a matrix of letters. The VCF output lists the same columns as sites with alleles and genotypes, which is the form that variant-oriented tools (`bcftools`, population-genetics software, genotype viewers) expect.
|
||||
|
||||
Nothing here comes from a reference genome. The file has no real coordinates, and the sites are not located on a chromosome: the positions below are labels that make the file valid and reproducible, not map positions.
|
||||
|
||||
## Producing the file
|
||||
|
||||
```
|
||||
obikmer phylo --vcf --n-sites 5000000 -o run1 my_index
|
||||
```
|
||||
|
||||
`--vcf` implies `--pseudo-alignment`, so `--n-sites N|all` is required, together with an already-built sibling annex (see [`phylo`](../usage/phylo.md)). `--fasta` and `--vcf` can be given together to get both files from the same draw; with neither, only the FASTA is written. Every option that shapes the draw (`--free-loss`, `--no-ambiguity`, `--only-diploid`, `--quartet`, `--exclude-genome`, ...) applies to the VCF exactly as to the FASTA, since both describe the same sites.
|
||||
|
||||
## Layout
|
||||
|
||||
An illustrative excerpt (columns are tab-separated in the file):
|
||||
|
||||
```
|
||||
##fileformat=VCFv4.5
|
||||
##source=obikmer
|
||||
##contig=<ID=obikmer_0_0>
|
||||
##contig=<ID=obikmer_0_1>
|
||||
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype (diploid)">
|
||||
#CHROM POS ID REF ALT QUAL FILTER INFO FORMAT idx0 idx1 idx2 idx3
|
||||
obikmer_0_0 440 CGATTNGTTAA A G . . . GT 0/0 1/1 0/0 0/0
|
||||
obikmer_0_0 1511 CACGANTGTAG A G,T . . . GT 0/0 0/0 1/1 2/2
|
||||
obikmer_0_1 9 TTACGNAGCTA C * . . . GT 0/0 0/0 1/1 0/0
|
||||
```
|
||||
|
||||
There is one sample column per genome that takes part in the draw, named after the genome label, in the index's own order. A genome removed with `--exclude-genome` or `--min-shared-family` has no column at all. `QUAL`, `FILTER` and `INFO` are always `.`: obikmer applies no variant filter and attaches no annotation, and leaving them empty keeps a file of millions of sites compact.
|
||||
|
||||
## Coordinates: `CHROM` and `POS`
|
||||
|
||||
An index is organised in partitions, and each partition holds one or more layers of k-mers (see [Index construction and on-disk layout](index_layout.md)). A family always belongs to exactly one layer of one partition, and the families of a layer are visited in a fixed order. The VCF uses that structure as coordinates:
|
||||
|
||||
- `CHROM` is `obikmer_<partition>_<layer>`, one virtual contig per layer of each partition in which at least one site was drawn;
|
||||
- `POS` is the rank of the family inside that layer, starting at 1.
|
||||
|
||||
For a given index and a given sibling annex, a family therefore always receives the same `CHROM` and `POS`, whatever the draw, the options or the number of sites requested. Two runs on the same index can be compared position by position. This stability is lost if the sibling annex is rebuilt, or if the index is rebuilt, merged or filtered, because the families are then enumerated again. Positions are never comparable between two different indexes; use `ID` for that. Even rebuilding an index from the very same genomes may enumerate the families in another order: on a small test index built twice from identical input, the family `CGATTNGTTAA` had `POS` 440 the first time and 371 the second, with the same `ID`.
|
||||
|
||||
The file is sorted by partition, then layer, then position, which is also the order of the `##contig` lines, so tools that require a sorted file accept it directly. Positions are sparse: only the drawn families are written.
|
||||
|
||||
## Identifier: `ID`
|
||||
|
||||
The `ID` column holds a label for the family that does not depend on the index. It is built from one of the family's k-mers by replacing its central base with `N` (for example `CGATTNGTTAA` for k = 11), after choosing, among the family's possible orientations, the one that sorts first. All the members of a family, whichever strand they were read on, give the same label.
|
||||
|
||||
This is what makes two files comparable even when they come from different indexes: the same family has the same `ID` in both, while its `CHROM` and `POS` will generally differ. It is also unique within a file, since two different families never share a label.
|
||||
|
||||
## Alleles: `REF` and `ALT`
|
||||
|
||||
The alleles are the bases carried at the centre of the family's k-mers, as in the FASTA. Two points are worth knowing.
|
||||
|
||||
First, the base is the central base of the member's *canonical* form (the smaller of the k-mer and its reverse complement). That is not necessarily the strand of the `ID` label, so the letters should not be matched to the `ID` sequence by position; they are labels for the allele, not a reading of that sequence.
|
||||
|
||||
Second, `REF` carries no meaning of reference: it is simply the most frequent allele among the genomes of the draw, each allele copy counted once, with the alphabetical order A < C < G < T deciding ties. `ALT` lists the remaining observed bases in the order A, C, G, T, separated by commas. There is no option to choose a reference genome.
|
||||
|
||||
### Absence: the `<ABS>` allele
|
||||
|
||||
A genome may carry none of the family's k-mers. Without `--free-loss` this absence is treated as a real state (the family was lost, or is absent from that genome), which the FASTA shows as `-`. In the VCF it becomes one more allele, the symbolic allele `<ABS>`, listed last in `ALT` and declared in the header by `##ALT=<ID=ABS,Description="Family absent from the genome">`. A genome without the family receives the genotype of `<ABS>`; for a site with `ALT` equal to `G,T,<ABS>` that is `3/3`. A site that differs between genomes only by presence versus absence has `ALT` equal to `<ABS>`.
|
||||
|
||||
The VCF specification gives the plain character `*` a meaning of its own (an allele missing because of an overlapping deletion described by another record), which is not the situation here, so it is deliberately not used. `<ABS>` is a free-form symbolic identifier, outside the reserved structural-variant names (`DEL`, `INS`, `DUP`, `INV`, `CNV`) and outside `<*>`, which belongs to reference blocks. Tools that only look at sequence alleles treat the site as having an alternative that is not a base.
|
||||
|
||||
With `--free-loss`, absence is declared missing data instead: `<ABS>` never appears, and the genomes concerned get a missing genotype.
|
||||
|
||||
## Genotypes
|
||||
|
||||
Every genome gets one `GT` value per site. The ploidy of these values is a convention chosen from the options, and the header says which (`Genotype (diploid)` or `Genotype (haploid)`).
|
||||
|
||||
| Cell in the alignment | Diploid (default, `--only-diploid`) | Haploid (`--no-ambiguity`) |
|
||||
|---|---|---|
|
||||
| one base `X` | `X/X`, for example `0/0` or `1/1` | `X`, for example `0` |
|
||||
| two bases (IUPAC `R Y S W K M`) | the heterozygote, for example `0/1` | does not occur |
|
||||
| three or four bases (`B D H V N`) | `./.` (not expressible) | does not occur |
|
||||
| absent family (`<ABS>`) | `<ABS>/<ABS>`, for example `3/3` | the index of `<ABS>` |
|
||||
| missing (`?`) | `./.` | `.` |
|
||||
|
||||
The two allele indices of a diploid genotype are always written in increasing order (`0/1`, never `1/0`), and the separator is `/` because no phasing information exists.
|
||||
|
||||
Why diploid by default? A genome can carry several forms of the same family, which the alignment records as an IUPAC ambiguity code. A diploid genotype can express up to two forms, and `--only-diploid` removes the genomes that would need more (three or four forms) before the sites are selected, so that no `./.` comes from them. `--no-ambiguity` goes further and treats every such genome as missing, which makes all genotypes haploid.
|
||||
|
||||
This is a convention, not a measurement. The file records which forms a genome carries, not how many copies of each: `A/A/G` and `A/G/G` both give the code `R` and the genotype `0/1`. Likewise, a genome with two forms may be heterozygous at one locus, or it may carry two similar regions of its genome (paralogues, repeats); the file cannot tell these apart. Do not read the genotypes as calls from sequencing reads.
|
||||
|
||||
## Which sites are written
|
||||
|
||||
Only the families that survive the draw and its masking appear. Within those:
|
||||
|
||||
- a site needs at least one genome with a base or (in diploid mode) a two-base code, because `REF` is a base; a site where every genome is missing, absent or has an ambiguity code too wide for the mode is left out of the VCF while remaining in the FASTA;
|
||||
- the log line at the end of the run gives the number of sites written out of the number drawn, so the difference is visible.
|
||||
|
||||
A site is selected because its genomes show at least two distinct states, and absence counts as a state. A site that differs between genomes only by presence versus absence is therefore a legitimate site, written with `ALT` equal to `<ABS>`.
|
||||
|
||||
## Using the file with other tools
|
||||
|
||||
- The contigs are virtual and have no declared length, and the file refers to no reference. Tools that need one (annotation, liftover) cannot use it; tools that only read sites and genotypes (`bcftools view`, `query`, `stats`, population-genetics software, PLINK-style converters) can.
|
||||
- If a tool rejects multiallelic sites or the `<ABS>` allele, `bcftools view -m2 -M2` keeps biallelic sites only, and `bcftools view -e 'ALT="<ABS>"'` removes the sites involving absence. The file as written was read by `bcftools` and indexed by `tabix` without complaint on a small test index, in both diploid and haploid modes. `bcftools stats` counts the sites with `<ABS>` among the "others" (symbolic alleles) rather than as SNPs.
|
||||
- If a tool expects haploid data, run `phylo` with `--no-ambiguity` rather than converting the genotypes afterwards.
|
||||
|
||||
## See also
|
||||
|
||||
- [`phylo`](../usage/phylo.md): all the options that shape the draw.
|
||||
- [Sampling design](../theory/phylogeny/snp_based/sampling_design.md): how the sites are drawn.
|
||||
- [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model.md): what a family is.
|
||||
@@ -79,15 +79,7 @@ See [Central-position SNP model](../theory/phylogeny/snp_based/central_snp_model
|
||||
|
||||
#### VCF output (`--vcf`)
|
||||
|
||||
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). A VCF needs genomic coordinates, which a family of k-mers does not have, so each site gets virtual ones that are stable for a given index: the contig is `obikmer_<partition>_<layer>` (one per scanned layer of each partition) and `POS` is the family's rank inside that layer, starting at 1. The same family therefore always lands at the same position as long as the index and its sibling annex are not rebuilt, and the file is sorted by construction.
|
||||
|
||||
The `ID` column is a label that does not depend on the index at all: the family's k-mer with its central base replaced by `N`, taken in a canonical orientation shared by every member of the family. Two VCF files produced from different indexes can therefore be matched on `ID`.
|
||||
|
||||
`REF` is the most frequent allele among the genomes at that site, each allele copy counted once (alphabetical order on ties); `ALT` lists the other observed bases in the order A, C, G, T. The bases are those of the family member's canonical form, exactly as in the FASTA, so they are not necessarily on the same strand as the `ID` k-mer.
|
||||
|
||||
The ploidy of the genotypes follows `--no-ambiguity`; `--only-diploid` keeps the diploid genotypes and simply removes, before the sites are selected, the genomes whose cell would have needed three or four alleles (so no `./.` comes from them). Without it, a genome can carry several forms of the same family, which the FASTA shows as an IUPAC ambiguity code, and genotypes are diploid: a single base `X` is written `X/X` (for example `0/0`), a code standing for two bases is the heterozygote (`0/1`), with the allele indices always in ascending order. A code standing for three or four bases (or `N`) cannot be expressed as a diploid genotype and is written `./.`. With `--no-ambiguity`, such genomes are already treated as missing, so genotypes are haploid: a single index (`0`, `1`...), and `.` for a missing genome. The diploid choice is a convention, not a measurement: the file records which forms the genome carries, not how many copies of each, and a genome with two forms may equally well come from a heterozygous individual or from two similar regions of the same genome.
|
||||
|
||||
Without `--free-loss`, the absence of a family in a genome is a real state (a loss), written `-` in the FASTA. In the VCF it is the extra allele `*`, listed after the bases, so a genome lacking the family gets the genotype of `*` (`1/1` if `*` is the only alternative, `3/3` if `ALT` is `G,T,*`) and a site that differs only by presence/absence has `ALT` equal to `*`. With `--free-loss`, absence is treated as missing data instead and such genomes get a missing genotype. A site where no genome has a single base or a two-base code has no `REF` and is left out; the log line gives the number of sites written out of the number drawn.
|
||||
The same drawn sites can be written as `<prefix>_alignment.vcf`, with `--fasta` and `--vcf` freely combined (FASTA alone remains the default). The coordinates, identifiers, alleles and genotypes of this file follow conventions of their own, described on the [VCF pseudo-alignment](../formats/vcf.md) page. The genotype ploidy follows `--no-ambiguity` (haploid) and `--only-diploid` (diploid, with the genomes carrying three or four forms of a family removed beforehand).
|
||||
|
||||
| Option | Default | Description |
|
||||
|---|---|---|
|
||||
|
||||
Reference in new issue
Block a user