obitools/obitools4
2
0 Fork 0
mirror of https://github.com/metabarcoding/obitools4.git synced 2026-11-09 09:24:41 +00:00
Code Issues Packages Projects Releases Wiki Activity

Complete the documentation and add a Release note file

Browse Source
This commit is contained in:
coissac committed 2023-02-18 19:54:21 +01:00
1 parent 56722895e4
commit a0ab14c205
14 files changed
+768 -214
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

Release-notes.md
+29
View File
@@ -0,0 +1,29 @@
# OBITools release notes
## February $18^th$, 2023. Release 4.0.0
It is the first version of the *OBITools* version 4. I decided to tag then following two weeks
of intensive data analysis with them allowing to discover many small bugs present in the previous
non-official version. Obviously other bugs are certainly persent in the code, and you are welcome
to use the git ticket system to mention them. But they seems to produce now reliable results.
### Corrected bugs
- On some computers the end of the output file was lost, leading to the loose of sequences and
to the production of incorrect file because of the last sequence record, sometime truncated in
its middle. This was only occurring when more than a single CPU was used. It was affecting every obitools.
- The `obiparing` software had a bug in the right aligment procedure. This led to the non alignment
of very sort barcode during the paring of the forward and reverse reads.
- The `obipairing` tools had a non deterministic comportment when aligning a paor very low quality reads.
This induced that the result of the same low quality read pair was not the same from run to run.
### New functionality
- Adding of a `--compress|-Z` option to every obitools allowing to produce `gz` compressed output. OBITools
were already able to deal with gziped input files transparently. They can now produce their résults in the same format.
- Adding of a `--append|-A` option to the `obidistribute` tool. It allows to append the result of an
`obidistribute` execution to preexisting files.
- Adding of a `--directory|-d` option to the `obidistribute` tool. It allows to declare a secondary
classification key over the one defined by the '--category|-c` option. This extra key leads to produce
directories in which files produced according to the primary criterion are stored.
- Adding of the functions `subspc`, `printf`, `int`, `numeric`, and `bool` to the expression language.
cmd/obitools/obidistribute/main.go
+1
View File
@@ -16,6 +16,7 @@ func main() {
_, args, _ := optionParser(os.Args)
fs, _ := obiconvert.ReadBioSequences(args...)
obidistribute.DistributeSequence(fs)
obiiter.WaitForLastPipe()
doc/_book/commands.html
+167 -53
View File
@@ -20,6 +20,69 @@ ul.task-list li input[type="checkbox"] {
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
pre > code.sourceCode { white-space: pre; position: relative; }
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
pre > code.sourceCode > span:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
pre > code.sourceCode { white-space: pre-wrap; }
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
{ counter-reset: source-line 0; }
pre.numberSource code > span
{ position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
{ content: counter(source-line);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
color: #aaaaaa;
}
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
div.sourceCode
{ }
@media screen {
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
code span.al { color: #ff0000; font-weight: bold; } /* Alert */
code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code span.at { color: #7d9029; } /* Attribute */
code span.bn { color: #40a070; } /* BaseN */
code span.bu { color: #008000; } /* BuiltIn */
code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code span.ch { color: #4070a0; } /* Char */
code span.cn { color: #880000; } /* Constant */
code span.co { color: #60a0b0; font-style: italic; } /* Comment */
code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code span.do { color: #ba2121; font-style: italic; } /* Documentation */
code span.dt { color: #902000; } /* DataType */
code span.dv { color: #40a070; } /* DecVal */
code span.er { color: #ff0000; font-weight: bold; } /* Error */
code span.ex { } /* Extension */
code span.fl { color: #40a070; } /* Float */
code span.fu { color: #06287e; } /* Function */
code span.im { color: #008000; font-weight: bold; } /* Import */
code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
code span.kw { color: #007020; font-weight: bold; } /* Keyword */
code span.op { color: #666666; } /* Operator */
code span.ot { color: #007020; } /* Other */
code span.pp { color: #bc7a00; } /* Preprocessor */
code span.sc { color: #4070a0; } /* SpecialChar */
code span.ss { color: #bb6688; } /* SpecialString */
code span.st { color: #4070a0; } /* String */
code span.va { color: #19177c; } /* Variable */
code span.vs { color: #4070a0; } /* VerbatimString */
code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
div.csl-bib-body { }
div.csl-entry {
clear: both;
@@ -175,16 +238,36 @@ div.csl-indent {
<li><a href="#function-defined-in-the-language" id="toc-function-defined-in-the-language" class="nav-link" data-scroll-target="#function-defined-in-the-language"><span class="toc-section-number">4.3.2</span> Function defined in the language</a></li>
<li><a href="#accessing-to-the-sequence-annotations" id="toc-accessing-to-the-sequence-annotations" class="nav-link" data-scroll-target="#accessing-to-the-sequence-annotations"><span class="toc-section-number">4.3.3</span> Accessing to the sequence annotations</a></li>
</ul></li>
<li><a href="#metabarcode-design-and-quality-assessment" id="toc-metabarcode-design-and-quality-assessment" class="nav-link" data-scroll-target="#metabarcode-design-and-quality-assessment"><span class="toc-section-number">4.4</span> Metabarcode design and quality assessment</a></li>
<li><a href="#file-format-conversions" id="toc-file-format-conversions" class="nav-link" data-scroll-target="#file-format-conversions"><span class="toc-section-number">4.5</span> File format conversions</a></li>
<li><a href="#sequence-annotations" id="toc-sequence-annotations" class="nav-link" data-scroll-target="#sequence-annotations"><span class="toc-section-number">4.6</span> Sequence annotations</a></li>
<li><a href="#metabarcode-design-and-quality-assessment" id="toc-metabarcode-design-and-quality-assessment" class="nav-link" data-scroll-target="#metabarcode-design-and-quality-assessment"><span class="toc-section-number">4.4</span> Metabarcode design and quality assessment</a>
<ul class="collapse">
<li><a href="#obipcr" id="toc-obipcr" class="nav-link" data-scroll-target="#obipcr"><span class="toc-section-number">4.4.1</span> <code>obipcr</code></a></li>
</ul></li>
<li><a href="#file-format-conversions" id="toc-file-format-conversions" class="nav-link" data-scroll-target="#file-format-conversions"><span class="toc-section-number">4.5</span> File format conversions</a>
<ul class="collapse">
<li><a href="#obiconvert" id="toc-obiconvert" class="nav-link" data-scroll-target="#obiconvert"><span class="toc-section-number">4.5.1</span> <code>obiconvert</code></a></li>
</ul></li>
<li><a href="#sequence-annotations" id="toc-sequence-annotations" class="nav-link" data-scroll-target="#sequence-annotations"><span class="toc-section-number">4.6</span> Sequence annotations</a>
<ul class="collapse">
<li><a href="#obiannotate" id="toc-obiannotate" class="nav-link" data-scroll-target="#obiannotate"><span class="toc-section-number">4.6.1</span> <code>obiannotate</code></a></li>
<li><a href="#obitag" id="toc-obitag" class="nav-link" data-scroll-target="#obitag"><span class="toc-section-number">4.6.2</span> <code>obitag</code></a></li>
</ul></li>
<li><a href="#computations-on-sequences" id="toc-computations-on-sequences" class="nav-link" data-scroll-target="#computations-on-sequences"><span class="toc-section-number">4.7</span> Computations on sequences</a>
<ul class="collapse">
<li><a href="#obipairing" id="toc-obipairing" class="nav-link" data-scroll-target="#obipairing"><span class="toc-section-number">4.7.1</span> <code>obipairing</code></a></li>
<li><a href="#obimultiplex" id="toc-obimultiplex" class="nav-link" data-scroll-target="#obimultiplex"><span class="toc-section-number">4.7.2</span> <code>obimultiplex</code></a></li>
<li><a href="#obicomplement" id="toc-obicomplement" class="nav-link" data-scroll-target="#obicomplement"><span class="toc-section-number">4.7.3</span> <code>obicomplement</code></a></li>
<li><a href="#obiclean" id="toc-obiclean" class="nav-link" data-scroll-target="#obiclean"><span class="toc-section-number">4.7.4</span> <code>obiclean</code></a></li>
<li><a href="#obiuniq" id="toc-obiuniq" class="nav-link" data-scroll-target="#obiuniq"><span class="toc-section-number">4.7.5</span> <code>obiuniq</code></a></li>
</ul></li>
<li><a href="#sequence-sampling-and-filtering" id="toc-sequence-sampling-and-filtering" class="nav-link" data-scroll-target="#sequence-sampling-and-filtering"><span class="toc-section-number">4.8</span> Sequence sampling and filtering</a>
<ul class="collapse">
<li><a href="#utilities" id="toc-utilities" class="nav-link" data-scroll-target="#utilities"><span class="toc-section-number">4.8.1</span> Utilities</a></li>
<li><a href="#obigrep" id="toc-obigrep" class="nav-link" data-scroll-target="#obigrep"><span class="toc-section-number">4.8.1</span> <code>obigrep</code></a></li>
</ul></li>
<li><a href="#utilities" id="toc-utilities" class="nav-link" data-scroll-target="#utilities"><span class="toc-section-number">4.9</span> Utilities</a>
<ul class="collapse">
<li><a href="#obicount" id="toc-obicount" class="nav-link" data-scroll-target="#obicount"><span class="toc-section-number">4.9.1</span> <code>obicount</code></a></li>
<li><a href="#obidistribute" id="toc-obidistribute" class="nav-link" data-scroll-target="#obidistribute"><span class="toc-section-number">4.9.2</span> <code>obidistribute</code></a></li>
<li><a href="#obifind" id="toc-obifind" class="nav-link" data-scroll-target="#obifind"><span class="toc-section-number">4.9.3</span> <code>obifind</code></a></li>
</ul></li>
</ul>
</nav>
@@ -268,40 +351,52 @@ div.csl-indent {
<p>Several OBITools (<em>e.g.</em> obigrep, obiannotate) allow the user to specify some simple expressions to compute values or define predicates. This expressions are parsed and evaluated using the <a href="https://pkg.go.dev/github.com/PaesslerAG/gval" title="Gval (Go eVALuate) for evaluating arbitrary expressions Go-like expressions.">gval</a> go package, which allows for evaluating go-Like expression.</p>
<section id="variables-usable-in-the-expression" class="level3" data-number="4.3.1">
<h3 data-number="4.3.1" class="anchored" data-anchor-id="variables-usable-in-the-expression"><span class="header-section-number">4.3.1</span> Variables usable in the expression</h3>
<section id="sequence" class="level4" data-number="4.3.1.1">
<h4 data-number="4.3.1.1" class="anchored" data-anchor-id="sequence"><span class="header-section-number">4.3.1.1</span> sequence</h4>
<p>sequence is the sequence object on which the expression is evaluated</p>
</section>
<section id="annotation" class="level4" data-number="4.3.1.2">
<h4 data-number="4.3.1.2" class="anchored" data-anchor-id="annotation"><span class="header-section-number">4.3.1.2</span> annotation</h4>
</section>
<ul>
<li><code>sequence</code> is the sequence object on which the expression is evaluated.</li>
<li><code>annotations</code>is a map object containing every annotations associated to the currently processed sequence.</li>
</ul>
</section>
<section id="function-defined-in-the-language" class="level3" data-number="4.3.2">
<h3 data-number="4.3.2" class="anchored" data-anchor-id="function-defined-in-the-language"><span class="header-section-number">4.3.2</span> Function defined in the language</h3>
<section id="len" class="level4" data-number="4.3.2.1">
<h4 data-number="4.3.2.1" class="anchored" data-anchor-id="len"><span class="header-section-number">4.3.2.1</span> len</h4>
<section id="instrospection-functions" class="level4 unnumbered">
<h4 class="unnumbered anchored" data-anchor-id="instrospection-functions">Instrospection functions</h4>
<ul>
<li><code>len(x)</code>is a generic function allowing to retreive the size of a object. It returns the length of a sequences, the number of element in a map like <code>annotations</code>, the number of elements in an array. The reurned value is an <code>int</code>.</li>
</ul>
</section>
<section id="ismap" class="level4" data-number="4.3.2.2">
<h4 data-number="4.3.2.2" class="anchored" data-anchor-id="ismap"><span class="header-section-number">4.3.2.2</span> ismap</h4>
<section id="cast-functions" class="level4 unnumbered">
<h4 class="unnumbered anchored" data-anchor-id="cast-functions">Cast functions</h4>
<ul>
<li><code>int(x)</code> converts if possible the <code>x</code> value to an integer value. The function returns an <code>int</code>.</li>
<li><code>numeric(x)</code> converts if possible the <code>x</code> value to a float value. The function returns a <code>float</code>.</li>
<li><code>bool(x)</code> converts if possible the <code>x</code> value to a boolean value. The function returns a <code>bool</code>.</li>
</ul>
</section>
<section id="hasattribute" class="level4" data-number="4.3.2.3">
<h4 data-number="4.3.2.3" class="anchored" data-anchor-id="hasattribute"><span class="header-section-number">4.3.2.3</span> hasattribute</h4>
</section>
<section id="min" class="level4" data-number="4.3.2.4">
<h4 data-number="4.3.2.4" class="anchored" data-anchor-id="min"><span class="header-section-number">4.3.2.4</span> min</h4>
</section>
<section id="max" class="level4" data-number="4.3.2.5">
<h4 data-number="4.3.2.5" class="anchored" data-anchor-id="max"><span class="header-section-number">4.3.2.5</span> max</h4>
<section id="string-related-functions" class="level4 unnumbered">
<h4 class="unnumbered anchored" data-anchor-id="string-related-functions">String related functions</h4>
<ul>
<li><code>printf(format,...)</code> allows to combine several values to build a string. <code>format</code> follows the classical C <code>printf</code> syntax. The function returns a <code>string</code>.</li>
<li><code>subspc(x)</code> substitutes every space in the <code>x</code> string by the underscore (<code>_</code>) character. The function returns a <code>string</code>.</li>
</ul>
</section>
</section>
<section id="accessing-to-the-sequence-annotations" class="level3" data-number="4.3.3">
<h3 data-number="4.3.3" class="anchored" data-anchor-id="accessing-to-the-sequence-annotations"><span class="header-section-number">4.3.3</span> Accessing to the sequence annotations</h3>
<p>The <code>annotations</code> variable is a map object containing all the annotations associated to the currently processed sequence. Index of the map are the attribute names. It exists to possibillities to retreive an annotation. It is possible to use the classical <code>[]</code> indexing operator, putting the attribute name quoted by double quotes between them.</p>
<div class="sourceCode" id="cb1"><pre class="sourceCode go code-with-copy"><code class="sourceCode go"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>annotations<span class="op">[</span><span class="st">"direction"</span><span class="op">]</span></span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<p>The above code retreives the <code>direction</code> annotation. A second notation using the dot (<code>.</code>) is often more convenient.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode go code-with-copy"><code class="sourceCode go"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>annotations<span class="op">.</span>direction</span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<p>Special attributes of the sequence are accessible only by dedicated methods of the <code>sequence</code> object.</p>
<ul>
<li>The sequence identifier : <code>Id()</code></li>
<li>THe sequence definition : <code>Definition()</code></li>
</ul>
</section>
</section>
<section id="metabarcode-design-and-quality-assessment" class="level2" data-number="4.4">
<h2 data-number="4.4" class="anchored" data-anchor-id="metabarcode-design-and-quality-assessment"><span class="header-section-number">4.4</span> Metabarcode design and quality assessment</h2>
<section id="obipcr" class="level4" data-number="4.4.0.1">
<h4 data-number="4.4.0.1" class="anchored" data-anchor-id="obipcr"><span class="header-section-number">4.4.0.1</span> <code>obipcr</code></h4>
<section id="obipcr" class="level3" data-number="4.4.1">
<h3 data-number="4.4.1" class="anchored" data-anchor-id="obipcr"><span class="header-section-number">4.4.1</span> <code>obipcr</code></h3>
<blockquote class="blockquote">
<p>Replace the <code>ecoPCR</code> original <em>OBITools</em></p>
</blockquote>
@@ -309,14 +404,17 @@ div.csl-indent {
</section>
<section id="file-format-conversions" class="level2" data-number="4.5">
<h2 data-number="4.5" class="anchored" data-anchor-id="file-format-conversions"><span class="header-section-number">4.5</span> File format conversions</h2>
<section id="obiconvert" class="level4" data-number="4.5.0.1">
<h4 data-number="4.5.0.1" class="anchored" data-anchor-id="obiconvert"><span class="header-section-number">4.5.0.1</span> <code>obiconvert</code></h4>
<section id="obiconvert" class="level3" data-number="4.5.1">
<h3 data-number="4.5.1" class="anchored" data-anchor-id="obiconvert"><span class="header-section-number">4.5.1</span> <code>obiconvert</code></h3>
</section>
</section>
<section id="sequence-annotations" class="level2" data-number="4.6">
<h2 data-number="4.6" class="anchored" data-anchor-id="sequence-annotations"><span class="header-section-number">4.6</span> Sequence annotations</h2>
<section id="obitag" class="level4" data-number="4.6.0.1">
<h4 data-number="4.6.0.1" class="anchored" data-anchor-id="obitag"><span class="header-section-number">4.6.0.1</span> <code>obitag</code></h4>
<section id="obiannotate" class="level3" data-number="4.6.1">
<h3 data-number="4.6.1" class="anchored" data-anchor-id="obiannotate"><span class="header-section-number">4.6.1</span> <code>obiannotate</code></h3>
</section>
<section id="obitag" class="level3" data-number="4.6.2">
<h3 data-number="4.6.2" class="anchored" data-anchor-id="obitag"><span class="header-section-number">4.6.2</span> <code>obitag</code></h3>
</section>
</section>
<section id="computations-on-sequences" class="level2" data-number="4.7">
@@ -326,15 +424,15 @@ div.csl-indent {
<blockquote class="blockquote">
<p>Replace the <code>illuminapairedends</code> original <em>OBITools</em></p>
</blockquote>
<section id="alignment-procedure" class="level4" data-number="4.7.1.1">
<h4 data-number="4.7.1.1" class="anchored" data-anchor-id="alignment-procedure"><span class="header-section-number">4.7.1.1</span> Alignment procedure</h4>
<section id="alignment-procedure" class="level4 unnumbered">
<h4 class="unnumbered anchored" data-anchor-id="alignment-procedure">Alignment procedure</h4>
<p><code>obipairing</code> is introducing a new alignment algorithm compared to the <code>illuminapairedend</code> command of the <code>OBITools V2</code>. Nethertheless this new algorithm has been design to produce the same results than the previous, except in very few cases.</p>
<p>The new algorithm is a two-step procedure. First, a FASTN-type algorithm <span class="citation" data-cites="Lipman1985-hw">(<a href="references.html#ref-Lipman1985-hw" role="doc-biblioref">Lipman and Pearson 1985</a>)</span> identifies the best offset between the two matched readings. This identifies the region of overlap.</p>
<p>In the second step, the matching regions of the two reads are extracted along with a flanking sequence of <span class="math inline">\(\Delta\)</span> base pairs. The two subsequences are then aligned using a “one side free end-gap” dynamic programming algorithm. This latter step is only called if at least one mismatch is detected by the FASTP step.</p>
<p>Unless the similarity between the two reads at their overlap region is very low, the addition of the flanking regions in the second step of the alignment ensures the same alignment as if the dynamic programming alignment was performed on the full reads.</p>
</section>
<section id="the-scoring-system" class="level4" data-number="4.7.1.2">
<h4 data-number="4.7.1.2" class="anchored" data-anchor-id="the-scoring-system"><span class="header-section-number">4.7.1.2</span> The scoring system</h4>
<section id="the-scoring-system" class="level4 unnumbered">
<h4 class="unnumbered anchored" data-anchor-id="the-scoring-system">The scoring system</h4>
<p>In the dynamic programming step, the match and mismatch scores take into account the quality scores of the two aligned nucleotides. By taking these into account, the probability of a true match can be calculated for each aligned base pair.</p>
<p>If we consider a nucleotide read with a quality score <span class="math inline">\(Q\)</span>, the probability of misreading this base (<span class="math inline">\(P_E\)</span>) is : <span class="math display">\[
P_E = 10^{-\frac{Q}{10}}
@@ -388,6 +486,12 @@ P(MATCH | X_1 \neq X_2) = (1-P_{E1})\frac{P_{E2}}{3} + (1-P_{E2})\frac{P_{E1}}
\end{aligned}
\]</span></p>
<p><strong>Probability of a match under the random model</strong></p>
<p>The second considered model is a pure random model where every base is equiprobable, hence having a probability of occurrence of a nucleotide equals <span class="math inline">\(0.25\)</span>. Under that hypothesis</p>
<p><span class="math display">\[
P(MATCH | \text{Random model}) = 0.25
\]</span></p>
<p><strong>The score is a log ration of likelyhood</strong></p>
<p>Score is define as the logarithm of the ratio between the likelyhood of the observations considering the sequencer error model over tha likelyhood u</p>
<div class="cell">
<div class="cell-output-display">
<div class="quarto-figure quarto-figure-center">
@@ -399,38 +503,49 @@ P(MATCH | X_1 \neq X_2) = (1-P_{E1})\frac{P_{E2}}{3} + (1-P_{E2})\frac{P_{E1}}
</div>
</div>
</section>
<section id="obimultiplex" class="level4" data-number="4.7.1.3">
<h4 data-number="4.7.1.3" class="anchored" data-anchor-id="obimultiplex"><span class="header-section-number">4.7.1.3</span> <code>obimultiplex</code></h4>
</section>
<section id="obimultiplex" class="level3" data-number="4.7.2">
<h3 data-number="4.7.2" class="anchored" data-anchor-id="obimultiplex"><span class="header-section-number">4.7.2</span> <code>obimultiplex</code></h3>
<blockquote class="blockquote">
<p>Replace the <code>ngsfilter</code> original <em>OBITools</em></p>
</blockquote>
</section>
<section id="obicomplement" class="level4" data-number="4.7.1.4">
<h4 data-number="4.7.1.4" class="anchored" data-anchor-id="obicomplement"><span class="header-section-number">4.7.1.4</span> <code>obicomplement</code></h4>
<section id="obicomplement" class="level3" data-number="4.7.3">
<h3 data-number="4.7.3" class="anchored" data-anchor-id="obicomplement"><span class="header-section-number">4.7.3</span> <code>obicomplement</code></h3>
</section>
<section id="obiclean" class="level4" data-number="4.7.1.5">
<h4 data-number="4.7.1.5" class="anchored" data-anchor-id="obiclean"><span class="header-section-number">4.7.1.5</span> <code>obiclean</code></h4>
</section>
<section id="obiuniq" class="level4" data-number="4.7.1.6">
<h4 data-number="4.7.1.6" class="anchored" data-anchor-id="obiuniq"><span class="header-section-number">4.7.1.6</span> <code>obiuniq</code></h4>
<section id="obiclean" class="level3" data-number="4.7.4">
<h3 data-number="4.7.4" class="anchored" data-anchor-id="obiclean"><span class="header-section-number">4.7.4</span> <code>obiclean</code></h3>
</section>
<section id="obiuniq" class="level3" data-number="4.7.5">
<h3 data-number="4.7.5" class="anchored" data-anchor-id="obiuniq"><span class="header-section-number">4.7.5</span> <code>obiuniq</code></h3>
</section>
</section>
<section id="sequence-sampling-and-filtering" class="level2" data-number="4.8">
<h2 data-number="4.8" class="anchored" data-anchor-id="sequence-sampling-and-filtering"><span class="header-section-number">4.8</span> Sequence sampling and filtering</h2>
<section id="obigrep" class="level4" data-number="4.8.0.1">
<h4 data-number="4.8.0.1" class="anchored" data-anchor-id="obigrep"><span class="header-section-number">4.8.0.1</span> <code>obigrep</code></h4>
<section id="obigrep" class="level3" data-number="4.8.1">
<h3 data-number="4.8.1" class="anchored" data-anchor-id="obigrep"><span class="header-section-number">4.8.1</span> <code>obigrep</code></h3>
</section>
<section id="utilities" class="level3" data-number="4.8.1">
<h3 data-number="4.8.1" class="anchored" data-anchor-id="utilities"><span class="header-section-number">4.8.1</span> Utilities</h3>
<section id="obicount" class="level4" data-number="4.8.1.1">
<h4 data-number="4.8.1.1" class="anchored" data-anchor-id="obicount"><span class="header-section-number">4.8.1.1</span> <code>obicount</code></h4>
</section>
<section id="obidistribute" class="level4" data-number="4.8.1.2">
<h4 data-number="4.8.1.2" class="anchored" data-anchor-id="obidistribute"><span class="header-section-number">4.8.1.2</span> <code>obidistribute</code></h4>
<section id="utilities" class="level2" data-number="4.9">
<h2 data-number="4.9" class="anchored" data-anchor-id="utilities"><span class="header-section-number">4.9</span> Utilities</h2>
<section id="obicount" class="level3" data-number="4.9.1">
<h3 data-number="4.9.1" class="anchored" data-anchor-id="obicount"><span class="header-section-number">4.9.1</span> <code>obicount</code></h3>
<p><code>obicount</code> counts the number of sequence records, the sum of the <code>count</code> attributes, and the sum of the length of all the sequences.</p>
<p><em>Example:</em></p>
<div class="sourceCode" id="cb3"><pre class="sourceCode bash code-with-copy"><code class="sourceCode bash"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a><span class="ex">obicount</span> seq.fasta </span></code><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></pre></div>
<p>Prints the number of sequence records contained in the <code>seq.fasta</code> file and the sum of their <code>count</code> attributes.</p>
<p><em>Options specific to the command</em></p>
<ul>
<li><code>--reads|-r</code> Prints read counts.</li>
<li><code>--symbols|-s</code> Prints symbol counts.</li>
<li><code>--variants|-v</code> Prints variant counts.</li>
</ul>
</section>
<section id="obifind" class="level4" data-number="4.8.1.3">
<h4 data-number="4.8.1.3" class="anchored" data-anchor-id="obifind"><span class="header-section-number">4.8.1.3</span> <code>obifind</code></h4>
<section id="obidistribute" class="level3" data-number="4.9.2">
<h3 data-number="4.9.2" class="anchored" data-anchor-id="obidistribute"><span class="header-section-number">4.9.2</span> <code>obidistribute</code></h3>
</section>
<section id="obifind" class="level3" data-number="4.9.3">
<h3 data-number="4.9.3" class="anchored" data-anchor-id="obifind"><span class="header-section-number">4.9.3</span> <code>obifind</code></h3>
<blockquote class="blockquote">
<p>Replace the <code>ecofind</code> original <em>OBITools.</em></p>
</blockquote>
@@ -443,7 +558,6 @@ Lipman, D J, and W R Pearson. 1985. <span>“<span class="nocase">Rapid and sens
</div>
</section>
</section>
</section>
</main> <!-- /main -->
<script id="quarto-html-after-body" type="application/javascript">
doc/_book/search.json
+20 -6
View File
@@ -60,42 +60,42 @@
"href": "commands.html#obitools-expression-language",
"title": "4  The OBITools V4 commands",
"section": "4.3 OBITools expression language",
"text": "4.3 OBITools expression language\nSeveral OBITools (e.g. obigrep, obiannotate) allow the user to specify some simple expressions to compute values or define predicates. This expressions are parsed and evaluated using the gval go package, which allows for evaluating go-Like expression.\n\n4.3.1 Variables usable in the expression\n\n4.3.1.1 sequence\nsequence is the sequence object on which the expression is evaluated\n\n\n4.3.1.2 annotation\n\n\n\n4.3.2 Function defined in the language\n\n4.3.2.1 len\n\n\n4.3.2.2 ismap\n\n\n4.3.2.3 hasattribute\n\n\n4.3.2.4 min\n\n\n4.3.2.5 max\n\n\n\n4.3.3 Accessing to the sequence annotations"
"text": "4.3 OBITools expression language\nSeveral OBITools (e.g. obigrep, obiannotate) allow the user to specify some simple expressions to compute values or define predicates. This expressions are parsed and evaluated using the gval go package, which allows for evaluating go-Like expression.\n\n4.3.1 Variables usable in the expression\n\nsequence is the sequence object on which the expression is evaluated.\nannotationsis a map object containing every annotations associated to the currently processed sequence.\n\n\n\n4.3.2 Function defined in the language\n\nInstrospection functions\n\nlen(x)is a generic function allowing to retreive the size of a object. It returns the length of a sequences, the number of element in a map like annotations, the number of elements in an array. The reurned value is an int.\n\n\n\nCast functions\n\nint(x) converts if possible the x value to an integer value. The function returns an int.\nnumeric(x) converts if possible the x value to a float value. The function returns a float.\nbool(x) converts if possible the x value to a boolean value. The function returns a bool.\n\n\n\nString related functions\n\nprintf(format,...) allows to combine several values to build a string. format follows the classical C printf syntax. The function returns a string.\nsubspc(x) substitutes every space in the x string by the underscore (_) character. The function returns a string.\n\n\n\n\n4.3.3 Accessing to the sequence annotations\nThe annotations variable is a map object containing all the annotations associated to the currently processed sequence. Index of the map are the attribute names. It exists to possibillities to retreive an annotation. It is possible to use the classical [] indexing operator, putting the attribute name quoted by double quotes between them.\nannotations[\"direction\"]\nThe above code retreives the direction annotation. A second notation using the dot (.) is often more convenient.\nannotations.direction\nSpecial attributes of the sequence are accessible only by dedicated methods of the sequence object.\n\nThe sequence identifier : Id()\nTHe sequence definition : Definition()"
},
{
"objectID": "commands.html#metabarcode-design-and-quality-assessment",
"href": "commands.html#metabarcode-design-and-quality-assessment",
"title": "4  The OBITools V4 commands",
"section": "4.4 Metabarcode design and quality assessment",
"text": "4.4 Metabarcode design and quality assessment\n\n4.4.0.1 obipcr\n\nReplace the ecoPCR original OBITools"
"text": "4.4 Metabarcode design and quality assessment\n\n4.4.1 obipcr\n\nReplace the ecoPCR original OBITools"
},
{
"objectID": "commands.html#file-format-conversions",
"href": "commands.html#file-format-conversions",
"title": "4  The OBITools V4 commands",
"section": "4.5 File format conversions",
"text": "4.5 File format conversions\n\n4.5.0.1 obiconvert"
"text": "4.5 File format conversions\n\n4.5.1 obiconvert"
},
{
"objectID": "commands.html#sequence-annotations",
"href": "commands.html#sequence-annotations",
"title": "4  The OBITools V4 commands",
"section": "4.6 Sequence annotations",
"text": "4.6 Sequence annotations\n\n4.6.0.1 obitag"
"text": "4.6 Sequence annotations\n\n4.6.1 obiannotate\n\n\n4.6.2 obitag"
},
{
"objectID": "commands.html#computations-on-sequences",
"href": "commands.html#computations-on-sequences",
"title": "4  The OBITools V4 commands",
"section": "4.7 Computations on sequences",
"text": "4.7 Computations on sequences\n\n4.7.1 obipairing\n\nReplace the illuminapairedends original OBITools\n\n\n4.7.1.1 Alignment procedure\nobipairing is introducing a new alignment algorithm compared to the illuminapairedend command of the OBITools V2. Nethertheless this new algorithm has been design to produce the same results than the previous, except in very few cases.\nThe new algorithm is a two-step procedure. First, a FASTN-type algorithm (Lipman and Pearson 1985) identifies the best offset between the two matched readings. This identifies the region of overlap.\nIn the second step, the matching regions of the two reads are extracted along with a flanking sequence of \\(\\Delta\\) base pairs. The two subsequences are then aligned using a “one side free end-gap” dynamic programming algorithm. This latter step is only called if at least one mismatch is detected by the FASTP step.\nUnless the similarity between the two reads at their overlap region is very low, the addition of the flanking regions in the second step of the alignment ensures the same alignment as if the dynamic programming alignment was performed on the full reads.\n\n\n4.7.1.2 The scoring system\nIn the dynamic programming step, the match and mismatch scores take into account the quality scores of the two aligned nucleotides. By taking these into account, the probability of a true match can be calculated for each aligned base pair.\nIf we consider a nucleotide read with a quality score \\(Q\\), the probability of misreading this base (\\(P_E\\)) is : \\[\nP_E = 10^{-\\frac{Q}{10}}\n\\]\nThus, when a given nucleotide \\(X\\) is observed with the quality score \\(Q\\). The probability that \\(X\\) is really an \\(X\\) is :\n\\[\nP(X=X) = 1 - P_E\n\\]\nOtherwise, \\(X\\) is actually one of the three other possible nucleotides (\\(X_{E1}\\), \\(X_{E2}\\) or \\(X_{E3}\\)). If we suppose that the three reading error have the same probability :\n\\[\nP(X=X_{E1}) = P(X=X_{E3}) = P(X=X_{E3}) = \\frac{P_E}{3}\n\\]\nAt each position in an alignment where the two nucleotides \\(X_1\\) and \\(X_2\\) face each other (not a gapped position), the probability of a true match varies depending on whether \\(X_1=X_2\\), an observed match, or \\(X_1 \\neq X_2\\), an observed mismatch.\nProbability of a true match when \\(X_1=X_2\\)\nThat probability can be divided in two parts. First \\(X_1\\) and \\(X_2\\) have been correctly read. The corresponding probability is :\n\\[\n\\begin{aligned}\nP_{TM} &= (1- PE_1)(1-PE_2)\\\\\n &=(1 - 10^{-\\frac{Q_1}{10} } )(1 - 10^{-\\frac{Q_2}{10}} )\n\\end{aligned}\n\\]\nSecondly, a match can occure if the true nucleotides read as \\(X_1\\) and \\(X_2\\) are not \\(X_1\\) and \\(X_2\\) but identical.\n\\[\n\\begin{aligned}\nP(X_1==X_{E1}) \\cap P(X_2==X_{E1}) &= \\frac{P_{E1} P_{E2}}{9} \\\\\nP(X_1==X_{Ex}) \\cap P(X_2==X_{Ex}) & = \\frac{P_{E1} P_{E2}}{3}\n\\end{aligned}\n\\]\nThe probability of a true match between \\(X_1\\) and \\(X_2\\) when \\(X_1 = X_2\\) an observed match :\n\\[\n\\begin{aligned}\nP(MATCH | X_1 = X_2) = (1- PE_1)(1-PE_2) + \\frac{P_{E1} P_{E2}}{3}\n\\end{aligned}\n\\]\nProbability of a true match when \\(X_1 \\neq X_2\\)\nThat probability can be divided in three parts.\n\n\\(X_1\\) has been correctly read and \\(X_2\\) is a sequencing error and is actually equal to \\(X_1\\). \\[\nP_a = (1-P_{E1})\\frac{P_{E2}}{3}\n\\]\n\\(X_2\\) has been correctly read and \\(X_1\\) is a sequencing error and is actually equal to \\(X_2\\). \\[\nP_b = (1-P_{E2})\\frac{P_{E1}}{3}\n\\]\n\\(X_1\\) and \\(X_2\\) corresponds to sequencing error but are actually the same base \\(X_{Ex}\\) \\[\nP_c = 2\\frac{P_{E1} P_{E2}}{9}\n\\]\n\nConsequently : \\[\n\\begin{aligned}\nP(MATCH | X_1 \\neq X_2) = (1-P_{E1})\\frac{P_{E2}}{3} + (1-P_{E2})\\frac{P_{E1}}{3} + 2\\frac{P_{E1} P_{E2}}{9}\n\\end{aligned}\n\\]\nProbability of a match under the random model\n\n\n\n\n\nEvolution of the match and mismatch scores when the quality of base is 20 while the second range from 10 to 40.\n\n\n\n\n\n\n4.7.1.3 obimultiplex\n\nReplace the ngsfilter original OBITools\n\n\n\n4.7.1.4 obicomplement\n\n\n4.7.1.5 obiclean\n\n\n4.7.1.6 obiuniq"
"text": "4.7 Computations on sequences\n\n4.7.1 obipairing\n\nReplace the illuminapairedends original OBITools\n\n\nAlignment procedure\nobipairing is introducing a new alignment algorithm compared to the illuminapairedend command of the OBITools V2. Nethertheless this new algorithm has been design to produce the same results than the previous, except in very few cases.\nThe new algorithm is a two-step procedure. First, a FASTN-type algorithm (Lipman and Pearson 1985) identifies the best offset between the two matched readings. This identifies the region of overlap.\nIn the second step, the matching regions of the two reads are extracted along with a flanking sequence of \\(\\Delta\\) base pairs. The two subsequences are then aligned using a “one side free end-gap” dynamic programming algorithm. This latter step is only called if at least one mismatch is detected by the FASTP step.\nUnless the similarity between the two reads at their overlap region is very low, the addition of the flanking regions in the second step of the alignment ensures the same alignment as if the dynamic programming alignment was performed on the full reads.\n\n\nThe scoring system\nIn the dynamic programming step, the match and mismatch scores take into account the quality scores of the two aligned nucleotides. By taking these into account, the probability of a true match can be calculated for each aligned base pair.\nIf we consider a nucleotide read with a quality score \\(Q\\), the probability of misreading this base (\\(P_E\\)) is : \\[\nP_E = 10^{-\\frac{Q}{10}}\n\\]\nThus, when a given nucleotide \\(X\\) is observed with the quality score \\(Q\\). The probability that \\(X\\) is really an \\(X\\) is :\n\\[\nP(X=X) = 1 - P_E\n\\]\nOtherwise, \\(X\\) is actually one of the three other possible nucleotides (\\(X_{E1}\\), \\(X_{E2}\\) or \\(X_{E3}\\)). If we suppose that the three reading error have the same probability :\n\\[\nP(X=X_{E1}) = P(X=X_{E3}) = P(X=X_{E3}) = \\frac{P_E}{3}\n\\]\nAt each position in an alignment where the two nucleotides \\(X_1\\) and \\(X_2\\) face each other (not a gapped position), the probability of a true match varies depending on whether \\(X_1=X_2\\), an observed match, or \\(X_1 \\neq X_2\\), an observed mismatch.\nProbability of a true match when \\(X_1=X_2\\)\nThat probability can be divided in two parts. First \\(X_1\\) and \\(X_2\\) have been correctly read. The corresponding probability is :\n\\[\n\\begin{aligned}\nP_{TM} &= (1- PE_1)(1-PE_2)\\\\\n &=(1 - 10^{-\\frac{Q_1}{10} } )(1 - 10^{-\\frac{Q_2}{10}} )\n\\end{aligned}\n\\]\nSecondly, a match can occure if the true nucleotides read as \\(X_1\\) and \\(X_2\\) are not \\(X_1\\) and \\(X_2\\) but identical.\n\\[\n\\begin{aligned}\nP(X_1==X_{E1}) \\cap P(X_2==X_{E1}) &= \\frac{P_{E1} P_{E2}}{9} \\\\\nP(X_1==X_{Ex}) \\cap P(X_2==X_{Ex}) & = \\frac{P_{E1} P_{E2}}{3}\n\\end{aligned}\n\\]\nThe probability of a true match between \\(X_1\\) and \\(X_2\\) when \\(X_1 = X_2\\) an observed match :\n\\[\n\\begin{aligned}\nP(MATCH | X_1 = X_2) = (1- PE_1)(1-PE_2) + \\frac{P_{E1} P_{E2}}{3}\n\\end{aligned}\n\\]\nProbability of a true match when \\(X_1 \\neq X_2\\)\nThat probability can be divided in three parts.\n\n\\(X_1\\) has been correctly read and \\(X_2\\) is a sequencing error and is actually equal to \\(X_1\\). \\[\nP_a = (1-P_{E1})\\frac{P_{E2}}{3}\n\\]\n\\(X_2\\) has been correctly read and \\(X_1\\) is a sequencing error and is actually equal to \\(X_2\\). \\[\nP_b = (1-P_{E2})\\frac{P_{E1}}{3}\n\\]\n\\(X_1\\) and \\(X_2\\) corresponds to sequencing error but are actually the same base \\(X_{Ex}\\) \\[\nP_c = 2\\frac{P_{E1} P_{E2}}{9}\n\\]\n\nConsequently : \\[\n\\begin{aligned}\nP(MATCH | X_1 \\neq X_2) = (1-P_{E1})\\frac{P_{E2}}{3} + (1-P_{E2})\\frac{P_{E1}}{3} + 2\\frac{P_{E1} P_{E2}}{9}\n\\end{aligned}\n\\]\nProbability of a match under the random model\nThe second considered model is a pure random model where every base is equiprobable, hence having a probability of occurrence of a nucleotide equals \\(0.25\\). Under that hypothesis\n\\[\nP(MATCH | \\text{Random model}) = 0.25\n\\]\nThe score is a log ration of likelyhood\nScore is define as the logarithm of the ratio between the likelyhood of the observations considering the sequencer error model over tha likelyhood u\n\n\n\n\n\nEvolution of the match and mismatch scores when the quality of base is 20 while the second range from 10 to 40.\n\n\n\n\n\n\n\n4.7.2 obimultiplex\n\nReplace the ngsfilter original OBITools\n\n\n\n4.7.3 obicomplement\n\n\n4.7.4 obiclean\n\n\n4.7.5 obiuniq"
},
{
"objectID": "commands.html#sequence-sampling-and-filtering",
"href": "commands.html#sequence-sampling-and-filtering",
"title": "4  The OBITools V4 commands",
"section": "4.8 Sequence sampling and filtering",
"text": "4.8 Sequence sampling and filtering\n\n4.8.0.1 obigrep\n\n\n4.8.1 Utilities\n\n4.8.1.1 obicount\n\n\n4.8.1.2 obidistribute\n\n\n4.8.1.3 obifind\n\nReplace the ecofind original OBITools.\n\n\n\n\n\nLipman, D J, and W R Pearson. 1985. “Rapid and sensitive protein similarity searches.” Science 227 (4693): 1435–41. http://www.ncbi.nlm.nih.gov/pubmed/2983426."
"text": "4.8 Sequence sampling and filtering\n\n4.8.1 obigrep"
},
{
"objectID": "library.html#biosequence",
@@ -124,5 +124,19 @@
"title": "References",
"section": "",
"text": "Andersen, Kenneth, Karen Lise Bird, Morten Rasmussen, James Haile,\nHenrik Breuning-Madsen, Kurt H Kjaer, Ludovic Orlando, M Thomas P\nGilbert, and Eske Willerslev. 2012. “Meta-barcoding of ëdirtı́DNA from soil reflects vertebrate\nbiodiversity.” Molecular Ecology 21 (8): 1966–79.\n\n\nBaldwin, Darren S, Matthew J Colloff, Gavin N Rees, Anthony A Chariton,\nGarth O Watson, Leon N Court, Diana M Hartley, et al. 2013. “Impacts of inundation and drought on eukaryote\nbiodiversity in semi-arid floodplain soils.” Molecular\nEcology 22 (6): 1746–58. https://doi.org/10.1111/mec.12190.\n\n\nBoyer, Frédéric, Céline Mercier, Aurélie Bonin, Yvan Le Bras, Pierre\nTaberlet, and Eric Coissac. 2016. “obitools:\na unix-inspired software package for DNA metabarcoding.”\nMolecular Ecology Resources 16 (1): 176–82. https://doi.org/10.1111/1755-0998.12428.\n\n\nCaporaso, J Gregory, Justin Kuczynski, Jesse Stombaugh, Kyle Bittinger,\nFrederic D Bushman, Elizabeth K Costello, Noah Fierer, et al. 2010.\n“QIIME allows analysis of high-throughput\ncommunity sequencing data.” Nature Methods 7 (5):\n335–36. https://doi.org/10.1038/nmeth.f.303.\n\n\nChariton, Anthony A, Anthony C Roach, Stuart L Simpson, and Graeme E\nBatley. 2010. “Influence of the choice of\nphysical and chemistry variables on interpreting patterns of sediment\ncontaminants and their relationships with estuarine macrobenthic\ncommunities.” Marine and Freshwater Research. https://doi.org/10.1071/mf09263.\n\n\nCock, Peter JA, Christopher J Fields, Naohisa Goto, Michael L Heuer, and\nPeter M Rice. 2010. “The Sanger FASTQ File Format for Sequences\nwith Quality Scores, and the Solexa/Illumina FASTQ Variants.”\nNucleic Acids Research 38 (6): 1767–71.\n\n\nDeagle, Bruce E, Roger Kirkwood, and Simon N Jarman. 2009. “Analysis of Australian fur seal diet by pyrosequencing\nprey DNA in faeces.” Molecular Ecology 18 (9):\n2022–38. https://doi.org/10.1111/j.1365-294X.2009.04158.x.\n\n\nKowalczyk, Rafał, Pierre Taberlet, Eric Coissac, Alice Valentini,\nChristian Miquel, Tomasz Kamiński, and Jan M Wójcik. 2011. “Influence of management practices on large herbivore\ndiet—Case of European bison in Białowieża Primeval Forest (Poland).”\nForest Ecology and Management 261 (4): 821–28. https://doi.org/10.1016/j.foreco.2010.11.026.\n\n\nLipman, D J, and W R Pearson. 1985. “Rapid\nand sensitive protein similarity searches.”\nScience 227 (4693): 1435–41. http://www.ncbi.nlm.nih.gov/pubmed/2983426.\n\n\nParducci, Laura, Tina Jørgensen, Mari Mette Tollefsrud, Ellen Elverland,\nTorbjørn Alm, Sonia L Fontana, K D Bennett, et al. 2012. “Glacial survival of boreal trees in northern\nScandinavia.” Science 335 (6072): 1083–86. https://doi.org/10.1126/science.1216043.\n\n\nRiaz, Tiayyba, Wasim Shehzad, Alain Viari, François Pompanon, Pierre\nTaberlet, and Eric Coissac. 2011. “ecoPrimers: inference of new DNA barcode markers from\nwhole genome sequence analysis.” Nucleic Acids\nResearch 39 (21): e145. https://doi.org/10.1093/nar/gkr732.\n\n\nSchloss, Patrick D, Sarah L Westcott, Thomas Ryabin, Justine R Hall,\nMartin Hartmann, Emily B Hollister, Ryan A Lesniewski, et al. 2009.\n“Introducing mothur: open-source,\nplatform-independent, community-supported software for describing and\ncomparing microbial communities.” Applied and\nEnvironmental Microbiology 75 (23): 7537–41. https://doi.org/10.1128/AEM.01541-09.\n\n\nSeguritan, V, and F Rohwer. 2001. “FastGroup:\na program to dereplicate libraries of 16S rDNA sequences.”\nBMC Bioinformatics 2 (October): 9. https://doi.org/10.1186/1471-2105-2-9.\n\n\nShehzad, Wasim, Tiayyba Riaz, Muhammad A Nawaz, Christian Miquel, Carole\nPoillot, Safdar A Shah, Francois Pompanon, Eric Coissac, and Pierre\nTaberlet. 2012. “Carnivore diet analysis\nbased on next-generation sequencing: Application to the leopard cat\n(Prionailurus bengalensis) in Pakistan.” Molecular\nEcology 21 (8): 1951–65. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1365-294X.2011.05424.x.\n\n\nSogin, Mitchell L, Hilary G Morrison, Julie A Huber, David Mark Welch,\nSusan M Huse, Phillip R Neal, Jesus M Arrieta, and Gerhard J Herndl.\n2006. “Microbial diversity in the deep sea\nand the underexplored \"rare biosphere\".” Proceedings\nof the National Academy of Sciences of the United States of America\n103 (32): 12115–20. https://doi.org/10.1073/pnas.0605127103.\n\n\nSønstebø, J H, L Gielly, A K Brysting, R Elven, M Edwards, J Haile, E\nWillerslev, et al. 2010. “Using\nnext-generation sequencing for molecular reconstruction of past Arctic\nvegetation and climate.” Molecular Ecology\nResources 10 (6): 1009–18. https://doi.org/10.1111/j.1755-0998.2010.02855.x.\n\n\nTaberlet, Pierre, Eric Coissac, Mehrdad Hajibabaei, and Loren H\nRieseberg. 2012. “Environmental DNA.”\nMolecular Ecology 21 (8): 1789–93. https://doi.org/10.1111/j.1365-294X.2012.05542.x.\n\n\nThomsen, Philip FLine truncated
},
{
"objectID": "utilities.html",
"href": "utilities.html",
"title": "5  Utilities",
"section": "",
"text": "5.0.0.1 obicount\n\n\n5.0.0.2 obidistribute\n\n\n5.0.0.3 obifind\n\nReplace the ecofind original OBITools."
},
{
"objectID": "commands.html#utilities",
"href": "commands.html#utilities",
"title": "4  The OBITools V4 commands",
"section": "4.9 Utilities",
"text": "4.9 Utilities\n\n4.9.1 obicount\nobicount counts the number of sequence records, the sum of the count attributes, and the sum of the length of all the sequences.\nExample:\nobicount seq.fasta \nPrints the number of sequence records contained in the seq.fasta file and the sum of their count attributes.\nOptions specific to the command\n\n--reads|-r Prints read counts.\n--symbols|-s Prints symbol counts.\n--variants|-v Prints variant counts.\n\n\n\n4.9.2 obidistribute\n\n\n4.9.3 obifind\n\nReplace the ecofind original OBITools.\n\n\n\n\n\nLipman, D J, and W R Pearson. 1985. “Rapid and sensitive protein similarity searches.” Science 227 (4693): 1435–41. http://www.ncbi.nlm.nih.gov/pubmed/2983426."
}
]
doc/_book/tutorial_files/figure-epub/unnamed-chunk-10-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

doc/_book/tutorial_files/figure-epub/unnamed-chunk-9-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

doc/_book/tutorial_files/figure-pdf/unnamed-chunk-10-1.pdf
BIN
View File
Binary file not shown.
doc/_book/tutorial_files/figure-pdf/unnamed-chunk-9-1.pdf
BIN
View File
Binary file not shown.
doc/_book/utilities.html
+333
View File
@@ -0,0 +1,333 @@
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"><head>
<meta charset="utf-8">
<meta name="generator" content="quarto-1.2.256">
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
<title>OBITools V4 - 5&nbsp; Utilities</title>
<style>
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
ul.task-list li input[type="checkbox"] {
width: 0.8em;
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
</style>
<script src="site_libs/quarto-nav/quarto-nav.js"></script>
<script src="site_libs/quarto-nav/headroom.min.js"></script>
<script src="site_libs/clipboard/clipboard.min.js"></script>
<script src="site_libs/quarto-search/autocomplete.umd.js"></script>
<script src="site_libs/quarto-search/fuse.min.js"></script>
<script src="site_libs/quarto-search/quarto-search.js"></script>
<meta name="quarto:offset" content="./">
<link href="./library.html" rel="next">
<link href="./commands.html" rel="prev">
<script src="site_libs/quarto-html/quarto.js"></script>
<script src="site_libs/quarto-html/popper.min.js"></script>
<script src="site_libs/quarto-html/tippy.umd.min.js"></script>
<script src="site_libs/quarto-html/anchor.min.js"></script>
<link href="site_libs/quarto-html/tippy.css" rel="stylesheet">
<link href="site_libs/quarto-html/quarto-syntax-highlighting.css" rel="stylesheet" id="quarto-text-highlighting-styles">
<script src="site_libs/bootstrap/bootstrap.min.js"></script>
<link href="site_libs/bootstrap/bootstrap-icons.css" rel="stylesheet">
<link href="site_libs/bootstrap/bootstrap.min.css" rel="stylesheet" id="quarto-bootstrap" data-mode="light">
<script id="quarto-search-options" type="application/json">{
"location": "sidebar",
"copy-button": false,
"collapse-after": 3,
"panel-placement": "start",
"type": "textbox",
"limit": 20,
"language": {
"search-no-results-text": "No results",
"search-matching-documents-text": "matching documents",
"search-copy-link-title": "Copy link to search",
"search-hide-matches-text": "Hide additional matches",
"search-more-match-text": "more match in this document",
"search-more-matches-text": "more matches in this document",
"search-clear-button-title": "Clear",
"search-detached-cancel-button-title": "Cancel",
"search-submit-button-title": "Submit"
}
}</script>
</head>
<body class="nav-sidebar floating">
<div id="quarto-search-results"></div>
<header id="quarto-header" class="headroom fixed-top">
<nav class="quarto-secondary-nav" data-bs-toggle="collapse" data-bs-target="#quarto-sidebar" aria-controls="quarto-sidebar" aria-expanded="false" aria-label="Toggle sidebar navigation" onclick="if (window.quartoToggleHeadroom) { window.quartoToggleHeadroom(); }">
<div class="container-fluid d-flex justify-content-between">
<h1 class="quarto-secondary-nav-title"><span class="chapter-number">5</span>&nbsp; <span class="chapter-title">Utilities</span></h1>
<button type="button" class="quarto-btn-toggle btn" aria-label="Show secondary navigation">
<i class="bi bi-chevron-right"></i>
</button>
</div>
</nav>
</header>
<!-- content -->
<div id="quarto-content" class="quarto-container page-columns page-rows-contents page-layout-article">
<!-- sidebar -->
<nav id="quarto-sidebar" class="sidebar collapse sidebar-navigation floating overflow-auto">
<div class="pt-lg-2 mt-2 text-left sidebar-header">
<div class="sidebar-title mb-0 py-0">
<a href="./">OBITools V4</a>
</div>
</div>
<div class="mt-2 flex-shrink-0 align-items-center">
<div class="sidebar-search">
<div id="quarto-search" class="" title="Search"></div>
</div>
</div>
<div class="sidebar-menu-container">
<ul class="list-unstyled mt-1">
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./index.html" class="sidebar-item-text sidebar-link">Preface</a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./intro.html" class="sidebar-item-text sidebar-link"><span class="chapter-number">1</span>&nbsp; <span class="chapter-title">The OBITools</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./formats.html" class="sidebar-item-text sidebar-link"><span class="chapter-number">2</span>&nbsp; <span class="chapter-title">File formats usable with <em>OBITools</em></span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./tutorial.html" class="sidebar-item-text sidebar-link"><span class="chapter-number">3</span>&nbsp; <span class="chapter-title">OBITools V4 Tutorial</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./commands.html" class="sidebar-item-text sidebar-link"><span class="chapter-number">4</span>&nbsp; <span class="chapter-title">The <em>OBITools V4</em> commands</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./utilities.html" class="sidebar-item-text sidebar-link active"><span class="chapter-number">5</span>&nbsp; <span class="chapter-title">Utilities</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./library.html" class="sidebar-item-text sidebar-link"><span class="chapter-number">6</span>&nbsp; <span class="chapter-title">The GO <em>OBITools</em> library</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./annexes.html" class="sidebar-item-text sidebar-link"><span class="chapter-number">7</span>&nbsp; <span class="chapter-title">Annexes</span></a>
</div>
</li>
<li class="sidebar-item">
<div class="sidebar-item-container">
<a href="./references.html" class="sidebar-item-text sidebar-link">References</a>
</div>
</li>
</ul>
</div>
</nav>
<!-- margin-sidebar -->
<div id="quarto-margin-sidebar" class="sidebar margin-sidebar">
</div>
<!-- main -->
<main class="content" id="quarto-document-content">
<header id="title-block-header" class="quarto-title-block default">
<div class="quarto-title">
<h1 class="title d-none d-lg-block"><span class="chapter-number">5</span>&nbsp; <span class="chapter-title">Utilities</span></h1>
</div>
<div class="quarto-title-meta">
</div>
</header>
<section id="obicount" class="level4" data-number="5.0.0.1">
<h4 data-number="5.0.0.1" class="anchored" data-anchor-id="obicount"><span class="header-section-number">5.0.0.1</span> <code>obicount</code></h4>
</section>
<section id="obidistribute" class="level4" data-number="5.0.0.2">
<h4 data-number="5.0.0.2" class="anchored" data-anchor-id="obidistribute"><span class="header-section-number">5.0.0.2</span> <code>obidistribute</code></h4>
</section>
<section id="obifind" class="level4" data-number="5.0.0.3">
<h4 data-number="5.0.0.3" class="anchored" data-anchor-id="obifind"><span class="header-section-number">5.0.0.3</span> <code>obifind</code></h4>
<blockquote class="blockquote">
<p>Replace the <code>ecofind</code> original <em>OBITools.</em></p>
</blockquote>
</section>
</main> <!-- /main -->
<script id="quarto-html-after-body" type="application/javascript">
window.document.addEventListener("DOMContentLoaded", function (event) {
const toggleBodyColorMode = (bsSheetEl) => {
const mode = bsSheetEl.getAttribute("data-mode");
const bodyEl = window.document.querySelector("body");
if (mode === "dark") {
bodyEl.classList.add("quarto-dark");
bodyEl.classList.remove("quarto-light");
} else {
bodyEl.classList.add("quarto-light");
bodyEl.classList.remove("quarto-dark");
}
}
const toggleBodyColorPrimary = () => {
const bsSheetEl = window.document.querySelector("link#quarto-bootstrap");
if (bsSheetEl) {
toggleBodyColorMode(bsSheetEl);
}
}
toggleBodyColorPrimary();
const icon = "";
const anchorJS = new window.AnchorJS();
anchorJS.options = {
placement: 'right',
icon: icon
};
anchorJS.add('.anchored');
const clipboard = new window.ClipboardJS('.code-copy-button', {
target: function(trigger) {
return trigger.previousElementSibling;
}
});
clipboard.on('success', function(e) {
// button target
const button = e.trigger;
// don't keep focus
button.blur();
// flash "checked"
button.classList.add('code-copy-button-checked');
var currentTitle = button.getAttribute("title");
button.setAttribute("title", "Copied!");
let tooltip;
if (window.bootstrap) {
button.setAttribute("data-bs-toggle", "tooltip");
button.setAttribute("data-bs-placement", "left");
button.setAttribute("data-bs-title", "Copied!");
tooltip = new bootstrap.Tooltip(button,
{ trigger: "manual",
customClass: "code-copy-button-tooltip",
offset: [0, -8]});
tooltip.show();
}
setTimeout(function() {
if (tooltip) {
tooltip.hide();
button.removeAttribute("data-bs-title");
button.removeAttribute("data-bs-toggle");
button.removeAttribute("data-bs-placement");
}
button.setAttribute("title", currentTitle);
button.classList.remove('code-copy-button-checked');
}, 1000);
// clear code selection
e.clearSelection();
});
function tippyHover(el, contentFn) {
const config = {
allowHTML: true,
content: contentFn,
maxWidth: 500,
delay: 100,
arrow: false,
appendTo: function(el) {
return el.parentElement;
},
interactive: true,
interactiveBorder: 10,
theme: 'quarto',
placement: 'bottom-start'
};
window.tippy(el, config);
}
const noterefs = window.document.querySelectorAll('a[role="doc-noteref"]');
for (var i=0; i<noterefs.length; i++) {
const ref = noterefs[i];
tippyHover(ref, function() {
// use id or data attribute instead here
let href = ref.getAttribute('data-footnote-href') || ref.getAttribute('href');
try { href = new URL(href).hash; } catch {}
const id = href.replace(/^#\/?/, "");
const note = window.document.getElementById(id);
return note.innerHTML;
});
}
const findCites = (el) => {
const parentEl = el.parentElement;
if (parentEl) {
const cites = parentEl.dataset.cites;
if (cites) {
return {
el,
cites: cites.split(' ')
};
} else {
return findCites(el.parentElement)
}
} else {
return undefined;
}
};
var bibliorefs = window.document.querySelectorAll('a[role="doc-biblioref"]');
for (var i=0; i<bibliorefs.length; i++) {
const ref = bibliorefs[i];
const citeInfo = findCites(ref);
if (citeInfo) {
tippyHover(citeInfo.el, function() {
var popup = window.document.createElement('div');
citeInfo.cites.forEach(function(cite) {
var citeDiv = window.document.createElement('div');
citeDiv.classList.add('hanging-indent');
citeDiv.classList.add('csl-entry');
var biblioDiv = window.document.getElementById('ref-' + cite);
if (biblioDiv) {
citeDiv.innerHTML = biblioDiv.innerHTML;
}
popup.appendChild(citeDiv);
});
return popup.innerHTML;
});
}
}
});
</script>
<nav class="page-navigation">
<div class="nav-page nav-page-previous">
<a href="./commands.html" class="pagination-link">
<i class="bi bi-arrow-left-short"></i> <span class="nav-page-text"><span class="chapter-number">4</span>&nbsp; <span class="chapter-title">The <em>OBITools V4</em> commands</span></span>
</a>
</div>
<div class="nav-page nav-page-next">
<a href="./library.html" class="pagination-link">
<span class="nav-page-text"><span class="chapter-number">6</span>&nbsp; <span class="chapter-title">The GO <em>OBITools</em> library</span></span> <i class="bi bi-arrow-right-short"></i>
</a>
</div>
</nav>
</div> <!-- /content -->
</body></html>
doc/_obipairing.qmd
+139
View File
@@ -0,0 +1,139 @@
### `obipairing`
> Replace the `illuminapairedends` original *OBITools*
#### Alignment procedure {.unnumbered}
`obipairing` is introducing a new alignment algorithm compared to the `illuminapairedend` command of the `OBITools V2`.
Nethertheless this new algorithm has been design to produce the same results than the previous, except in very few cases.
The new algorithm is a two-step procedure. First, a FASTN-type algorithm [@Lipman1985-hw] identifies the best offset between the two matched readings. This identifies the region of overlap.
In the second step, the matching regions of the two reads are extracted along with a flanking sequence of $\Delta$ base pairs. The two subsequences are then aligned using a "one side free end-gap" dynamic programming algorithm. This latter step is only called if at least one mismatch is detected by the FASTP step.
Unless the similarity between the two reads at their overlap region is very low, the addition of the flanking regions in the second step of the alignment ensures the same alignment as if the dynamic programming alignment was performed on the full reads.
#### The scoring system {.unnumbered}
In the dynamic programming step, the match and mismatch scores take into account the quality scores of the two aligned nucleotides. By taking these into account, the probability of a true match can be calculated for each aligned base pair.
If we consider a nucleotide read with a quality score $Q$, the probability of misreading this base ($P_E$) is :
$$
P_E = 10^{-\frac{Q}{10}}
$$
Thus, when a given nucleotide $X$ is observed with the quality score $Q$. The probability that $X$ is really an $X$ is :
$$
P(X=X) = 1 - P_E
$$
Otherwise, $X$ is actually one of the three other possible nucleotides ($X_{E1}$, $X_{E2}$ or $X_{E3}$). If we suppose that the three reading error have the same probability :
$$
P(X=X_{E1}) = P(X=X_{E3}) = P(X=X_{E3}) = \frac{P_E}{3}
$$
At each position in an alignment where the two nucleotides $X_1$ and $X_2$ face each other (not a gapped position), the probability of a true match varies depending on whether $X_1=X_2$, an observed match, or $X_1 \neq X_2$, an observed mismatch.
**Probability of a true match when $X_1=X_2$**
That probability can be divided in two parts. First $X_1$ and $X_2$ have been correctly read. The corresponding probability is :
$$
\begin{aligned}
P_{TM} &= (1- PE_1)(1-PE_2)\\
&=(1 - 10^{-\frac{Q_1}{10} } )(1 - 10^{-\frac{Q_2}{10}} )
\end{aligned}
$$
Secondly, a match can occure if the true nucleotides read as $X_1$ and $X_2$ are not $X_1$ and $X_2$ but identical.
$$
\begin{aligned}
P(X_1==X_{E1}) \cap P(X_2==X_{E1}) &= \frac{P_{E1} P_{E2}}{9} \\
P(X_1==X_{Ex}) \cap P(X_2==X_{Ex}) & = \frac{P_{E1} P_{E2}}{3}
\end{aligned}
$$
The probability of a true match between $X_1$ and $X_2$ when $X_1 = X_2$ an observed match :
$$
\begin{aligned}
P(MATCH | X_1 = X_2) = (1- PE_1)(1-PE_2) + \frac{P_{E1} P_{E2}}{3}
\end{aligned}
$$
**Probability of a true match when $X_1 \neq X_2$**
That probability can be divided in three parts.
a. $X_1$ has been correctly read and $X_2$ is a sequencing error and is actually equal to $X_1$.
$$
P_a = (1-P_{E1})\frac{P_{E2}}{3}
$$
a. $X_2$ has been correctly read and $X_1$ is a sequencing error and is actually equal to $X_2$.
$$
P_b = (1-P_{E2})\frac{P_{E1}}{3}
$$
a. $X_1$ and $X_2$ corresponds to sequencing error but are actually the same base $X_{Ex}$
$$
P_c = 2\frac{P_{E1} P_{E2}}{9}
$$
Consequently :
$$
\begin{aligned}
P(MATCH | X_1 \neq X_2) = (1-P_{E1})\frac{P_{E2}}{3} + (1-P_{E2})\frac{P_{E1}}{3} + 2\frac{P_{E1} P_{E2}}{9}
\end{aligned}
$$
**Probability of a match under the random model**
The second considered model is a pure random model where every base is equiprobable, hence having a probability of occurrence of a nucleotide equals $0.25$. Under that hypothesis
$$
P(MATCH | \text{Random model}) = 0.25
$$
**The score is a log ration of likelyhood**
Score is define as the logarithm of the ratio between the likelyhood of the observations considering the sequencer error model over tha likelyhood u
```{r}
#| echo: false
#| warning: false
#| fig-cap: "Evolution of the match and mismatch scores when the quality of base is 20 while the second range from 10 to 40."
require(ggplot2)
require(tidyverse)
Smatch <- function(Q1,Q2) {
PE1 <- 10^(-Q1/10)
PE2 <- 10^(-Q2/10)
PT1 <- 1 - PE1
PT2 <- 1 - PE2
PM <- PT1*PT2 + PE1 * PE2 / 3
round((log(PM)+log(4))*10)
}
Smismatch <- function(Q1,Q2) {
PE1 <- 10^(-Q1/10)
PE2 <- 10^(-Q2/10)
PT1 <- 1 - PE1
PT2 <- 1 - PE2
PM <- PE1*PT2/3 + PT1 * PE2 / 3 + 2/3 * PE1 * PE2
round((log(PM)+log(4))*10)
}
tibble(Q = 10:40) %>%
mutate(Match = mapply(Smatch,Q,20),
Mismatch = mapply(Smismatch,Q,20),
) %>% pivot_longer(cols = -Q, names_to = "Class", values_to = "Score") %>%
ggplot(aes(x=Q,y=Score,col=Class)) +
geom_line() +
xlab("Q1 (Q2=20)")
```
doc/_utilities.qmd
+26
View File
@@ -0,0 +1,26 @@
## Utilities
### `obicount`
`obicount` counts the number of sequence records, the sum of the ``count`` attributes, and the sum
of the length of all the sequences.
*Example:*
``` bash
obicount seq.fasta
```
Prints the number of sequence records contained in the ``seq.fasta``
file and the sum of their ``count`` attributes.
*Options specific to the command*
- `--reads|-r ` Prints read counts.
- `--symbols|-s` Prints symbol counts.
- `--variants|-v` Prints variant counts.
### `obidistribute`
### `obifind`
> Replace the `ecofind` original *OBITools.*
doc/commands.qmd
+53 -155
View File
@@ -57,191 +57,89 @@ Several OBITools (*e.g.* obigrep, obiannotate) allow the user to specify some si
### Variables usable in the expression
#### sequence
- `sequence` is the sequence object on which the expression is evaluated.
- `annotations`is a map object containing every annotations associated to the currently processed sequence.
sequence is the sequence object on which the expression is evaluated
### Function defined in the language
#### annotation
#### Instrospection functions {.unnumbered}
### Function defined in the language
- `len(x)`is a generic function allowing to retreive the size of a object. It returns
the length of a sequences, the number of element in a map like `annotations`, the number
of elements in an array. The reurned value is an `int`.
#### len
#### Cast functions {.unnumbered}
#### ismap
- `int(x)` converts if possible the `x` value to an integer value. The function
returns an `int`.
- `numeric(x)` converts if possible the `x` value to a float value. The function
returns a `float`.
- `bool(x)` converts if possible the `x` value to a boolean value. The function
returns a `bool`.
#### hasattribute
#### String related functions {.unnumbered}
#### min
#### max
- `printf(format,...)` allows to combine several values to build a string. `format` follows the
classical C `printf` syntax. The function returns a `string`.
- `subspc(x)` substitutes every space in the `x` string by the underscore (`_`) character. The function
returns a `string`.
### Accessing to the sequence annotations
The `annotations` variable is a map object containing all the annotations associated to the currently processed sequence. Index of the map are the attribute names. It exists to possibillities to retreive
an annotation. It is possible to use the classical `[]` indexing operator, putting the attribute name
quoted by double quotes between them.
```go
annotations["direction"]
```
The above code retreives the `direction` annotation. A second notation using the dot (`.`) is often
more convenient.
```go
annotations.direction
```
Special attributes of the sequence are accessible only by dedicated methods of the `sequence` object.
- The sequence identifier : `Id()`
- THe sequence definition : `Definition()`
## Metabarcode design and quality assessment
#### `obipcr`
### `obipcr`
> Replace the `ecoPCR` original *OBITools*
## File format conversions
#### `obiconvert`
### `obiconvert`
## Sequence annotations
#### `obitag`
### `obiannotate`
### `obitag`
## Computations on sequences
### `obipairing`
{{< include _obipairing.qmd >}}
> Replace the `illuminapairedends` original *OBITools*
#### Alignment procedure
`obipairing` is introducing a new alignment algorithm compared to the `illuminapairedend` command of the `OBITools V2`.
Nethertheless this new algorithm has been design to produce the same results than the previous, except in very few cases.
The new algorithm is a two-step procedure. First, a FASTN-type algorithm [@Lipman1985-hw] identifies the best offset between the two matched readings. This identifies the region of overlap.
In the second step, the matching regions of the two reads are extracted along with a flanking sequence of $\Delta$ base pairs. The two subsequences are then aligned using a "one side free end-gap" dynamic programming algorithm. This latter step is only called if at least one mismatch is detected by the FASTP step.
Unless the similarity between the two reads at their overlap region is very low, the addition of the flanking regions in the second step of the alignment ensures the same alignment as if the dynamic programming alignment was performed on the full reads.
#### The scoring system
In the dynamic programming step, the match and mismatch scores take into account the quality scores of the two aligned nucleotides. By taking these into account, the probability of a true match can be calculated for each aligned base pair.
If we consider a nucleotide read with a quality score $Q$, the probability of misreading this base ($P_E$) is :
$$
P_E = 10^{-\frac{Q}{10}}
$$
Thus, when a given nucleotide $X$ is observed with the quality score $Q$. The probability that $X$ is really an $X$ is :
$$
P(X=X) = 1 - P_E
$$
Otherwise, $X$ is actually one of the three other possible nucleotides ($X_{E1}$, $X_{E2}$ or $X_{E3}$). If we suppose that the three reading error have the same probability :
$$
P(X=X_{E1}) = P(X=X_{E3}) = P(X=X_{E3}) = \frac{P_E}{3}
$$
At each position in an alignment where the two nucleotides $X_1$ and $X_2$ face each other (not a gapped position), the probability of a true match varies depending on whether $X_1=X_2$, an observed match, or $X_1 \neq X_2$, an observed mismatch.
**Probability of a true match when $X_1=X_2$**
That probability can be divided in two parts. First $X_1$ and $X_2$ have been correctly read. The corresponding probability is :
$$
\begin{aligned}
P_{TM} &= (1- PE_1)(1-PE_2)\\
&=(1 - 10^{-\frac{Q_1}{10} } )(1 - 10^{-\frac{Q_2}{10}} )
\end{aligned}
$$
Secondly, a match can occure if the true nucleotides read as $X_1$ and $X_2$ are not $X_1$ and $X_2$ but identical.
$$
\begin{aligned}
P(X_1==X_{E1}) \cap P(X_2==X_{E1}) &= \frac{P_{E1} P_{E2}}{9} \\
P(X_1==X_{Ex}) \cap P(X_2==X_{Ex}) & = \frac{P_{E1} P_{E2}}{3}
\end{aligned}
$$
The probability of a true match between $X_1$ and $X_2$ when $X_1 = X_2$ an observed match :
$$
\begin{aligned}
P(MATCH | X_1 = X_2) = (1- PE_1)(1-PE_2) + \frac{P_{E1} P_{E2}}{3}
\end{aligned}
$$
**Probability of a true match when $X_1 \neq X_2$**
That probability can be divided in three parts.
a. $X_1$ has been correctly read and $X_2$ is a sequencing error and is actually equal to $X_1$.
$$
P_a = (1-P_{E1})\frac{P_{E2}}{3}
$$
a. $X_2$ has been correctly read and $X_1$ is a sequencing error and is actually equal to $X_2$.
$$
P_b = (1-P_{E2})\frac{P_{E1}}{3}
$$
a. $X_1$ and $X_2$ corresponds to sequencing error but are actually the same base $X_{Ex}$
$$
P_c = 2\frac{P_{E1} P_{E2}}{9}
$$
Consequently :
$$
\begin{aligned}
P(MATCH | X_1 \neq X_2) = (1-P_{E1})\frac{P_{E2}}{3} + (1-P_{E2})\frac{P_{E1}}{3} + 2\frac{P_{E1} P_{E2}}{9}
\end{aligned}
$$
**Probability of a match under the random model**
```{r}
#| echo: false
#| warning: false
#| fig-cap: "Evolution of the match and mismatch scores when the quality of base is 20 while the second range from 10 to 40."
require(ggplot2)
require(tidyverse)
Smatch <- function(Q1,Q2) {
PE1 <- 10^(-Q1/10)
PE2 <- 10^(-Q2/10)
PT1 <- 1 - PE1
PT2 <- 1 - PE2
PM <- PT1*PT2 + PE1 * PE2 / 3
round((log(PM)+log(4))*10)
}
Smismatch <- function(Q1,Q2) {
PE1 <- 10^(-Q1/10)
PE2 <- 10^(-Q2/10)
PT1 <- 1 - PE1
PT2 <- 1 - PE2
PM <- PE1*PT2/3 + PT1 * PE2 / 3 + 2/3 * PE1 * PE2
round((log(PM)+log(4))*10)
}
tibble(Q = 10:40) %>%
mutate(Match = mapply(Smatch,Q,20),
Mismatch = mapply(Smismatch,Q,20),
) %>% pivot_longer(cols = -Q, names_to = "Class", values_to = "Score") %>%
ggplot(aes(x=Q,y=Score,col=Class)) +
geom_line() +
xlab("Q1 (Q2=20)")
```
#### `obimultiplex`
### `obimultiplex`
> Replace the `ngsfilter` original *OBITools*
#### `obicomplement`
### `obicomplement`
#### `obiclean`
### `obiclean`
#### `obiuniq`
### `obiuniq`
## Sequence sampling and filtering
## Sequence sampling and filtering
#### `obigrep`
### `obigrep`
### Utilities
{{< include _utilities.qmd >}}
#### `obicount`
#### `obidistribute`
#### `obifind`
> Replace the `ecofind` original *OBITools.*
doc/commands_files/figure-epub/unnamed-chunk-1-1.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

doc/commands_files/figure-pdf/unnamed-chunk-1-1.pdf
BIN
View File
Binary file not shown.