<ahref="./library.html"class="sidebar-item-text sidebar-link"><spanclass="chapter-number">5</span> <spanclass="chapter-title">The GO <em>OBITools</em> library</span></a>
<li><ahref="#specifying-the-input-files-to-obitools-commands"id="toc-specifying-the-input-files-to-obitools-commands"class="nav-link active"data-scroll-target="#specifying-the-input-files-to-obitools-commands"><spanclass="toc-section-number">4.1</span> Specifying the input files to <em>OBITools</em> commands</a></li>
<li><ahref="#options-common-to-most-of-the-obitools-commands"id="toc-options-common-to-most-of-the-obitools-commands"class="nav-link"data-scroll-target="#options-common-to-most-of-the-obitools-commands"><spanclass="toc-section-number">4.2</span> Options common to most of the <em>OBITools</em> commands</a>
<li><ahref="#the-fasta-and-fastq-annotations-format"id="toc-the-fasta-and-fastq-annotations-format"class="nav-link"data-scroll-target="#the-fasta-and-fastq-annotations-format"><spanclass="toc-section-number">4.2.3</span> The Fasta and Fastq annotations format</a></li>
<li><ahref="#variables-usable-in-the-expression"id="toc-variables-usable-in-the-expression"class="nav-link"data-scroll-target="#variables-usable-in-the-expression"><spanclass="toc-section-number">4.3.1</span> Variables usable in the expression</a></li>
<li><ahref="#function-defined-in-the-language"id="toc-function-defined-in-the-language"class="nav-link"data-scroll-target="#function-defined-in-the-language"><spanclass="toc-section-number">4.3.2</span> Function defined in the language</a></li>
<li><ahref="#accessing-to-the-sequence-annotations"id="toc-accessing-to-the-sequence-annotations"class="nav-link"data-scroll-target="#accessing-to-the-sequence-annotations"><spanclass="toc-section-number">4.3.3</span> Accessing to the sequence annotations</a></li>
<li><ahref="#file-format-conversions"id="toc-file-format-conversions"class="nav-link"data-scroll-target="#file-format-conversions"><spanclass="toc-section-number">4.5</span> File format conversions</a>
<li><ahref="#computations-on-sequences"id="toc-computations-on-sequences"class="nav-link"data-scroll-target="#computations-on-sequences"><spanclass="toc-section-number">4.7</span> Computations on sequences</a>
<li><ahref="#sequence-sampling-and-filtering"id="toc-sequence-sampling-and-filtering"class="nav-link"data-scroll-target="#sequence-sampling-and-filtering"><spanclass="toc-section-number">4.8</span> Sequence sampling and filtering</a>
<h2data-number="4.1"class="anchored"data-anchor-id="specifying-the-input-files-to-obitools-commands"><spanclass="header-section-number">4.1</span> Specifying the input files to <em>OBITools</em> commands</h2>
<h2data-number="4.2"class="anchored"data-anchor-id="options-common-to-most-of-the-obitools-commands"><spanclass="header-section-number">4.2</span> Options common to most of the <em>OBITools</em> commands</h2>
<p>Five sequence formats are accepted for input files. <em>Fasta</em> (<ahref="formats.html#sec-fasta"><span>Section 2.1.2</span></a>) and <em>Fastq</em> (<ahref="formats.html#sec-fastq"><span>Section 2.1.3</span></a>) are the main ones, EMBL and Genbank allow the use of flat files produced by these two international databases. The last one, ecoPCR, is maintained for compatibility with previous <em>OBITools</em> and allows to read <em>ecoPCR</em> outputs as sequence files.</p>
<p>Several encoding schemes have been proposed for quality scores in <em>Fastq</em> format. Currently, <em>OBITools</em> considers Sanger encoding as the standard. For reasons of compatibility with older datasets produced with <em>Solexa</em> sequencers, it is possible, by using the following option, to force the use of the corresponding quality encoding scheme when reading these older files.</p>
<p>Only two output sequence formats are supported by OBITools, Fasta and Fastq. Fastq is used when output sequences are associated with quality information. Otherwise, Fasta is the default format. However, it is possible to force the output format by using one of the following two options. Forcing the use of Fasta results in the loss of quality information. Conversely, when the Fastq format is forced with sequences that have no quality data, dummy qualities set to 40 for each nucleotide are added.</p>
<ul>
<li><code>--fasta-output</code> Read data following the ecoPCR output format.</li>
<li><code>--fastq-output</code> Read data following the EMBL flatfile format.</li>
</ul>
<p>OBITools allows multiple input files to be specified for a single command.</p>
<li><code>--no-order</code> When several input files are provided, indicates that there is no order among them. (default: false). Using such option can increase a lot the processing of the data.</li>
<h3data-number="4.2.3"class="anchored"data-anchor-id="the-fasta-and-fastq-annotations-format"><spanclass="header-section-number">4.2.3</span> The Fasta and Fastq annotations format</h3>
<p>OBITools extend the <ahref="#the-fasta-sequence-format">Fasta</a> and <ahref="#the-fastq-sequence-format">Fastq</a> formats by introducing a format for the title lines of these formats allowing to annotate every sequence. While the previous version of OBITools used an <em>ad-hoc</em> format for these annotation, this new version introduce the usage of the standard JSON format to store them.</p>
<p>On input, OBITools automatically recognize the format of the annotations, but two options allows to force the parsing following one of them. You should normally not need to use these options.</p>
<ul>
<li><p><code>--input-OBI-header</code> FASTA/FASTQ title line annotations follow OBI format. (default: false)</p></li>
<li><p><code>--input-json-header</code> FASTA/FASTQ title line annotations follow json format. (default: false)</p></li>
</ul>
<p>On output, by default annotation are formatted using the new JSON format. For compatibility with previous version of OBITools and with external scripts and software, it is possible to force the usage of the previous OBITools format.</p>
<ul>
<li><p><code>--output-OBI-header|-O</code> output FASTA/FASTQ title line annotations follow OBI format. (default: false)</p></li>
<li><p><code>--output-json-header</code> output FASTA/FASTQ title line annotations follow json format. (default: false)</p></li>
<h4data-number="4.2.3.1"class="anchored"data-anchor-id="system-related-options"><spanclass="header-section-number">4.2.3.1</span> System related options</h4>
<p>Several OBITools (<em>e.g.</em> obigrep, obiannotate) allow the user to specify some simple expressions to compute values or define predicates. This expressions are parsed and evaluated using the <ahref="https://pkg.go.dev/github.com/PaesslerAG/gval"title="Gval (Go eVALuate) for evaluating arbitrary expressions Go-like expressions.">gval</a> go package, which allows for evaluating go-Like expression.</p>
<h3data-number="4.3.1"class="anchored"data-anchor-id="variables-usable-in-the-expression"><spanclass="header-section-number">4.3.1</span> Variables usable in the expression</h3>
<h3data-number="4.3.2"class="anchored"data-anchor-id="function-defined-in-the-language"><spanclass="header-section-number">4.3.2</span> Function defined in the language</h3>
<li><code>len(x)</code>is a generic function allowing to retreive the size of a object. It returns the length of a sequences, the number of element in a map like <code>annotations</code>, the number of elements in an array. The reurned value is an <code>int</code>.</li>
<h4class="unnumbered anchored"data-anchor-id="string-related-functions">String related functions</h4>
<ul>
<li><code>printf(format,...)</code> allows to combine several values to build a string. <code>format</code> follows the classical C <code>printf</code> syntax. The function returns a <code>string</code>.</li>
<li><code>subspc(x)</code> substitutes every space in the <code>x</code> string by the underscore (<code>_</code>) character. The function returns a <code>string</code>.</li>
<h3data-number="4.3.3"class="anchored"data-anchor-id="accessing-to-the-sequence-annotations"><spanclass="header-section-number">4.3.3</span> Accessing to the sequence annotations</h3>
<p>The <code>annotations</code> variable is a map object containing all the annotations associated to the currently processed sequence. Index of the map are the attribute names. It exists to possibillities to retreive an annotation. It is possible to use the classical <code>[]</code> indexing operator, putting the attribute name quoted by double quotes between them.</p>
<divclass="sourceCode"id="cb1"><preclass="sourceCode go code-with-copy"><codeclass="sourceCode go"><spanid="cb1-1"><ahref="#cb1-1"aria-hidden="true"tabindex="-1"></a>annotations<spanclass="op">[</span><spanclass="st">"direction"</span><spanclass="op">]</span></span></code><buttontitle="Copy to Clipboard"class="code-copy-button"><iclass="bi"></i></button></pre></div>
<p>The above code retreives the <code>direction</code> annotation. A second notation using the dot (<code>.</code>) is often more convenient.</p>
<divclass="sourceCode"id="cb2"><preclass="sourceCode go code-with-copy"><codeclass="sourceCode go"><spanid="cb2-1"><ahref="#cb2-1"aria-hidden="true"tabindex="-1"></a>annotations<spanclass="op">.</span>direction</span></code><buttontitle="Copy to Clipboard"class="code-copy-button"><iclass="bi"></i></button></pre></div>
<p>Special attributes of the sequence are accessible only by dedicated methods of the <code>sequence</code> object.</p>
<h2data-number="4.5"class="anchored"data-anchor-id="file-format-conversions"><spanclass="header-section-number">4.5</span> File format conversions</h2>
<h2data-number="4.7"class="anchored"data-anchor-id="computations-on-sequences"><spanclass="header-section-number">4.7</span> Computations on sequences</h2>
<p><code>obipairing</code> is introducing a new alignment algorithm compared to the <code>illuminapairedend</code> command of the <code>OBITools V2</code>. Nethertheless this new algorithm has been design to produce the same results than the previous, except in very few cases.</p>
<p>The new algorithm is a two-step procedure. First, a FASTN-type algorithm <spanclass="citation"data-cites="Lipman1985-hw">(<ahref="references.html#ref-Lipman1985-hw"role="doc-biblioref">Lipman and Pearson 1985</a>)</span> identifies the best offset between the two matched readings. This identifies the region of overlap.</p>
<p>In the second step, the matching regions of the two reads are extracted along with a flanking sequence of <spanclass="math inline">\(\Delta\)</span> base pairs. The two subsequences are then aligned using a “one side free end-gap” dynamic programming algorithm. This latter step is only called if at least one mismatch is detected by the FASTP step.</p>
<p>Unless the similarity between the two reads at their overlap region is very low, the addition of the flanking regions in the second step of the alignment ensures the same alignment as if the dynamic programming alignment was performed on the full reads.</p>
<p>In the dynamic programming step, the match and mismatch scores take into account the quality scores of the two aligned nucleotides. By taking these into account, the probability of a true match can be calculated for each aligned base pair.</p>
<p>If we consider a nucleotide read with a quality score <spanclass="math inline">\(Q\)</span>, the probability of misreading this base (<spanclass="math inline">\(P_E\)</span>) is : <spanclass="math display">\[
P_E = 10^{-\frac{Q}{10}}
\]</span></p>
<p>Thus, when a given nucleotide <spanclass="math inline">\(X\)</span> is observed with the quality score <spanclass="math inline">\(Q\)</span>. The probability that <spanclass="math inline">\(X\)</span> is really an <spanclass="math inline">\(X\)</span> is :</p>
<p><spanclass="math display">\[
P(X=X) = 1 - P_E
\]</span></p>
<p>Otherwise, <spanclass="math inline">\(X\)</span> is actually one of the three other possible nucleotides (<spanclass="math inline">\(X_{E1}\)</span>, <spanclass="math inline">\(X_{E2}\)</span> or <spanclass="math inline">\(X_{E3}\)</span>). If we suppose that the three reading error have the same probability :</p>
<p>At each position in an alignment where the two nucleotides <spanclass="math inline">\(X_1\)</span> and <spanclass="math inline">\(X_2\)</span> face each other (not a gapped position), the probability of a true match varies depending on whether <spanclass="math inline">\(X_1=X_2\)</span>, an observed match, or <spanclass="math inline">\(X_1 \neq X_2\)</span>, an observed mismatch.</p>
<p><strong>Probability of a true match when <spanclass="math inline">\(X_1=X_2\)</span></strong></p>
<p>That probability can be divided in two parts. First <spanclass="math inline">\(X_1\)</span> and <spanclass="math inline">\(X_2\)</span> have been correctly read. The corresponding probability is :</p>
<p>Secondly, a match can occure if the true nucleotides read as <spanclass="math inline">\(X_1\)</span> and <spanclass="math inline">\(X_2\)</span> are not <spanclass="math inline">\(X_1\)</span> and <spanclass="math inline">\(X_2\)</span> but identical.</p>
P(X_1==X_{Ex}) \cap P(X_2==X_{Ex}) & = \frac{P_{E1} P_{E2}}{3}
\end{aligned}
\]</span></p>
<p>The probability of a true match between <spanclass="math inline">\(X_1\)</span> and <spanclass="math inline">\(X_2\)</span> when <spanclass="math inline">\(X_1 = X_2\)</span> an observed match :</p>
<p><strong>Probability of a true match when <spanclass="math inline">\(X_1 \neq X_2\)</span></strong></p>
<p>That probability can be divided in three parts.</p>
<oltype="a">
<li><spanclass="math inline">\(X_1\)</span> has been correctly read and <spanclass="math inline">\(X_2\)</span> is a sequencing error and is actually equal to <spanclass="math inline">\(X_1\)</span>. <spanclass="math display">\[
P_a = (1-P_{E1})\frac{P_{E2}}{3}
\]</span></li>
<li><spanclass="math inline">\(X_2\)</span> has been correctly read and <spanclass="math inline">\(X_1\)</span> is a sequencing error and is actually equal to <spanclass="math inline">\(X_2\)</span>. <spanclass="math display">\[
P_b = (1-P_{E2})\frac{P_{E1}}{3}
\]</span></li>
<li><spanclass="math inline">\(X_1\)</span> and <spanclass="math inline">\(X_2\)</span> corresponds to sequencing error but are actually the same base <spanclass="math inline">\(X_{Ex}\)</span><spanclass="math display">\[
<p>The second considered model is a pure random model where every base is equiprobable, hence having a probability of occurrence of a nucleotide equals <spanclass="math inline">\(0.25\)</span>. Under that hypothesis</p>
<p><spanclass="math display">\[
P(MATCH | \text{Random model}) = 0.25
\]</span></p>
<p><strong>The score is a log ration of likelyhood</strong></p>
<p>Score is define as the logarithm of the ratio between the likelyhood of the observations considering the sequencer error model over tha likelyhood u</p>
<p></p><figcaptionclass="figure-caption">Evolution of the match and mismatch scores when the quality of base is 20 while the second range from 10 to 40.</figcaption><p></p>
<h2data-number="4.8"class="anchored"data-anchor-id="sequence-sampling-and-filtering"><spanclass="header-section-number">4.8</span> Sequence sampling and filtering</h2>
<p><code>obicount</code> counts the number of sequence records, the sum of the <code>count</code> attributes, and the sum of the length of all the sequences.</p>
<p><em>Example:</em></p>
<divclass="sourceCode"id="cb3"><preclass="sourceCode bash code-with-copy"><codeclass="sourceCode bash"><spanid="cb3-1"><ahref="#cb3-1"aria-hidden="true"tabindex="-1"></a><spanclass="ex">obicount</span> seq.fasta </span></code><buttontitle="Copy to Clipboard"class="code-copy-button"><iclass="bi"></i></button></pre></div>
<p>Prints the number of sequence records contained in the <code>seq.fasta</code> file and the sum of their <code>count</code> attributes.</p>
<p><em>Options specific to the command</em></p>
<ul>
<li><code>--reads|-r</code> Prints read counts.</li>
<li><code>--symbols|-s</code> Prints symbol counts.</li>
<li><code>--variants|-v</code> Prints variant counts.</li>
Lipman, D J, and W R Pearson. 1985. <span>“<spanclass="nocase">Rapid and sensitive protein similarity searches</span>.”</span><em>Science</em> 227 (4693): 1435–41. <ahref="http://www.ncbi.nlm.nih.gov/pubmed/2983426">http://www.ncbi.nlm.nih.gov/pubmed/2983426</a>.
<spanclass="nav-page-text"><spanclass="chapter-number">5</span> <spanclass="chapter-title">The GO <em>OBITools</em> library</span></span><iclass="bi bi-arrow-right-short"></i>
