Testing RAM instead of mmap for blob alignment

exceptions in Python>=3.7

MANIFEST.in

@@ -1,11 +1,12 @@
 include setup.py
-recursive-include distutils.ext *.py *.c *.pem
-recursive-include python *.pyx *.pxd *.c *.h *.cfiles
+recursive-include python *.pyx *.pxd *.c *.h
 recursive-include src *.c *.h
+include src/CMakeLists.txt
 recursive-include doc/sphinx/source *.txt *.rst *.py
 recursive-include doc/sphinx/sphinxext *.py
 include doc/sphinx/Makefile
 include doc/sphinx/Doxyfile
-include README.txt
+include README.md
 include requirements.txt
+include scripts/obi
 

README.md

@@ -1,9 +1,9 @@
 The `OBITools3`: A package for the management of analyses and data in DNA metabarcoding   
 ---------------------------------------------
 
+**Website: <https://metabarcoding.org/obitools3>**
 
-
-DNA metabarcoding offers new perspectives for biodiversity research [1]. This approach of ecosystem studies relies heavily on the use of Next-Generation Sequencing (NGS), and consequently requires the ability to to treat large volumes of data. The `OBITools` package  satisfies this requirement thanks to a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context [2] - <http://metabarcoding.org/obitools>. Their capacity to filter and edit sequences while taking into account taxonomic annotation helps to setup tailored-made analysis pipelines for a broad range of DNA metabarcoding applications, including biodiversity surveys or diet analyses.   
+DNA metabarcoding offers new perspectives for biodiversity research [1]. This approach of ecosystem studies relies heavily on the use of Next-Generation Sequencing (NGS), and consequently requires the ability to to treat large volumes of data. The `OBITools` package  satisfies this requirement thanks to a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context [2] - <https://metabarcoding.org/obitools>. Their capacity to filter and edit sequences while taking into account taxonomic annotation helps to setup tailored-made analysis pipelines for a broad range of DNA metabarcoding applications, including biodiversity surveys or diet analyses.   
 
 
 
@@ -23,18 +23,19 @@ DNA metabarcoding offers new perspectives for biodiversity research [1]. This ap
 
 
 
-**Tools.** The `OBITools3` offer the same tools as the original `OBITools`. Eventually, new versions of `ecoPrimers` (PCR primer design) [3], `ecoPCR` (*in silico* PCR) [4], as well as `Sumatra` (sequence alignment) and `Sumaclust` (sequence alignment and clustering) [5] will be added, taking advantage of the database structure developed for the `OBITools3`.    
+**Tools.** The `OBITools3` offer the same tools as the original `OBITools`, plus `ecoPCR` (*in silico* PCR) [4] and `Sumatra` (sequence alignment, not multithreaded yet) [5]. 
+Eventually, new versions of `ecoPrimers` (PCR primer design) [3], as well as `Sumaclust` (sequence alignment and clustering) [5] will be added, taking advantage of the database structure developed for the `OBITools3`.    
 
 
 
-**Implementation and disponibility.** The lower layers managing the DMS as well as all the compute-intensive functions are coded in `C99` for efficiency reasons. A `Cython` (<http://www.cython.org>) object layer allows for a simple but efficient implementation of the `OBITools3` commands in `Python 3.5`. The `OBITools3` are still in development, and the first functional versions are expected for autumn 2016.    
+**Implementation and disponibility.** The lower layers managing the DMS as well as all the compute-intensive functions are coded in `C99` for efficiency reasons. A `Cython` (<http://www.cython.org>) object layer allows for a simple but efficient implementation of the `OBITools3` commands in `Python 3`. The `OBITools3` are now being released, check the wiki for more information.
 
 
 
 **References.**
 
 1. Taberlet P, Coissac E, Hajibabaei M, Rieseberg LH: Environmental DNA. Mol Ecol 2012:1789–1793.
-2. Boyer F, Mercier C, Bonin A, Le Bras Y, Taberlet P, Coissac E: OBITools: a Unix-inspired software package for DNA metabarcoding. Mol Ecol Resour 2015:n/a–n/a.
+2. Boyer F, Mercier C, Bonin A, Le Bras Y, Taberlet P, Coissac E: OBITools: a Unix-inspired software package for DNA metabarcoding. Mol Ecol Resour, 2016: 176-182.
 3. Riaz T, Shehzad W, Viari A, Pompanon F, Taberlet P, Coissac E: ecoPrimers: inference of new DNA barcode markers from whole genome sequence analysis. Nucleic Acids Res 2011, 39:e145.
 4. Ficetola GF, Coissac E, Zundel S, Riaz T, Shehzad W, Bessière J, Taberlet P, Pompanon F: An in silico approach for the evaluation of DNA barcodes. BMC Genomics 2010, 11:434.
 5. Mercier C, Boyer F, Bonin A, Coissac E (2013) SUMATRA and SUMACLUST: fast and exact comparison and clustering of sequences. Available: <http://metabarcoding.org/sumatra> and <http://metabarcoding.org/sumaclust>

c-sandbox/obicount/Makefile

@@ -1,18 +0,0 @@
-CC = gcc
-CFLAGS = -c -Wall
-LDFLAGS = 
-SOURCES = obicount.c ../obidmscolumn.c
-OBJECTS = $(SOURCES:.c=.o)
-EXECUTABLE = obicount
-
-all: $(SOURCES) $(EXECUTABLE)
-    
-$(EXECUTABLE): $(OBJECTS) 
-	$(CC) $(LDFLAGS) $(OBJECTS) -o $@
-
-.c.o:
-	$(CC) $(CFLAGS) $< -o $@
-
-clean:
-	rm *o
-	rm $(EXECUTABLE)

c-sandbox/obicount/obicount.c

@@ -1,87 +0,0 @@
-
-#include <stdio.h>
-#include <stdlib.h>
-#include <sys/mman.h>	/* mmap() is defined in this header */
-#include <stdint.h>
-#include <fcntl.h>
-#include <sys/stat.h>
-
-#include "../obitypes.h"
-#include "../obidmscolumn.h"
-
-
-/**
- * @brief Computes the size to map.
- *
- * * @param OBIDMSColumn_file The file to map.
- * @return The size to map.
- */
-int get_size_to_map(int OBIDMSColumn_file)
-// compute size to map : file size minus size of the header
-{
-	int size;
-	struct stat s;
-
-	fstat(OBIDMSColumn_file, &s);
-	size = (s.st_size) - HEADER_SIZE;
-	return(size);
-}
-
-/**
- * @brief Computes and prints the total number of sequences by summing their counts.
- *
- * * @param The count file.
- */
-int main(int argc, char const *argv[])
-{
-	char* map;
-	int size;
-	int OBIDMSColumn_file;
-	int count;
-	char c;
-	char num_str[10] = "";
-	int num_int;
-	int i,j;
-
-	// initialize variables
-	OBIDMSColumn_file = open(argv[1], O_RDONLY);	//read only
-	count = 0;
-	j = 0;
-
-	// compute size to map
-	size = get_size_to_map(OBIDMSColumn_file);
-
-	// map the data
-	map = obi_map_read_only(OBIDMSColumn_file, HEADER_SIZE, size);
-
-	// sum the counts
-	for (i=0; i<size; i++)
-	{
-		c = map[i];
-
-		if (c != SEPARATOR)		// reading lines
-		{
-			num_str[j] = c;
-			j++;
-		}
-
-		else if (c == SEPARATOR)	// end of a line
-		{
-			num_int = atoi(num_str);	// turn number from character string to int
-			count = count + num_int;	// add the number to the sum
-			j = 0;
-			num_str[j] = '\0';
-		}
-	}
-
-	// print the final count of sequences
-	fprintf(stderr, "Sequence count = %d\n", count);
-
-	// unmap
-	obi_unmap(size);
-
-	// close file
-	close(OBIDMSColumn_file);
-
-	return(0);
-}

distutils.ext/obidistutils/command/build.py

@@ -1,38 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-from distutils.command.build import build as ori_build
-from obidistutils.serenity.checksystem import is_mac_system
-
-
-class build(ori_build):
-    
-    def has_ext_modules(self):
-        return self.distribution.has_ext_modules()
-    
-    def has_pidname(self):
-        return is_mac_system()
-
-    def has_doc(self):
-        return True
-    
-    def has_littlebigman(self):
-        return True
-    
-    try:
-        from obidistutils.command.build_sphinx import build_sphinx  # @UnusedImport
-
-        sub_commands = [("littlebigman",has_littlebigman),
-                         ('pidname',has_pidname)
-                       ] \
-                       + ori_build.sub_commands + \
-                       [('build_sphinx',has_doc)]
-    except ImportError:
-        sub_commands = [("littlebigman",has_littlebigman),
-                         ('pidname',has_pidname)
-                       ] \
-                       + ori_build.sub_commands
-        

distutils.ext/obidistutils/command/build_cexe.py

@@ -1,85 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-from .build_ctools import build_ctools
-from .build_exe import build_exe
-from distutils.errors import DistutilsSetupError
-from distutils import log
-import os
-
-class build_cexe(build_ctools):
-
-    description = "build C/C++ executable distributed with Python extensions"
-
-
-    def initialize_options(self):
-        build_ctools.initialize_options(self)
-        self.built_files = None
-
-
-    def finalize_options(self):
-        # This might be confusing: both build-cexe and build-temp default
-        # to build-temp as defined by the "build" command.  This is because
-        # I think that C libraries are really just temporary build
-        # by-products, at least from the point of view of building Python
-        # extensions -- but I want to keep my options open.
-
-        build_cexe_dir = self.build_cexe
-        build_ctools.finalize_options(self)
-
-        if build_cexe_dir is None:
-            self.build_cexe=None
-            
-        self.set_undefined_options('build',
-                                   ('build_scripts',  'build_cexe'))
-
-        self.set_undefined_options('build_files',
-                                   ('files',  'built_files'))
-                   
-        self.executables   = self.distribution.executables
-#         self.build_cexe = os.path.join(os.path.dirname(self.build_cexe),'cbinaries') 
-#         self.mkpath(self.build_cexe)
-
-        if self.executables:
-            self.check_executable_list(self.executables)
-                            
-
-        # XXX same as for build_ext -- what about 'self.define' and
-        # 'self.undef' ?
-
-    def substitute_sources(self,exe_name,sources):
-        """
-        Substitutes source file name starting by an @ by the actual
-        name of the built file (see --> build_files)
-        """
-        sources = list(sources)
-        for i in range(len(sources)):
-            message = "%s :-> %s" % (exe_name,sources[i])
-            if sources[i][0]=='@':
-                try:
-                    filename = self.built_files[sources[i][1:]]
-                except KeyError:
-                    raise DistutilsSetupError(
-                         'The %s filename declared in the source '
-                         'files of the program %s have not been '
-                         'built by the installation process' % (sources[i],
-                                                                exe_name))
-                sources[i]=filename
-                log.info("%s changed to %s",message,filename)
-            else:
-                log.info("%s ok",message)
-
-        return sources
- 
-    def run(self):
-
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
-
-        build_exe.run(self)
-        
-
-   

distutils.ext/obidistutils/command/build_ctools.py

@@ -1,63 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-
-from .build_exe import build_exe
-from distutils import log
-
-class build_ctools(build_exe):
-    description = "build C/C++ executable not distributed with Python extensions"
-
-    def initialize_options(self):
-        build_exe.initialize_options(self)
-        
-        # List of built tools
-        self.ctools = None
-        self.littlebigman = None
-
-
-    def finalize_options(self):
-        # This might be confusing: both build-cexe and build-temp default
-        # to build-temp as defined by the "build" command.  This is because
-        # I think that C libraries are really just temporary build
-        # by-products, at least from the point of view of building Python
-        # extensions -- but I want to keep my options open.
-            
-        build_exe.finalize_options(self)
-        
-        
-        self.set_undefined_options('build',
-                                    ('build_temp', 'build_cexe'))
-
-        self.set_undefined_options('littlebigman',
-                                   ('littlebigman',  'littlebigman'))
-                   
-        self.executables = self.distribution.ctools
-        self.check_executable_list(self.executables)
-        
-
-        if self.littlebigman =='-DLITTLE_END':
-            if self.define is None:
-                self.define=[('LITTLE_END',None)]
-            else:
-                self.define.append('LITTLE_END',None)
-                
-        log.info('Look for CPU architecture... %s',self.define)
-
-        self.ctools = set()
-
-    def run(self):
-
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
-
-        
-        build_exe.run(self)
-        
-        for e,p in self.executables:                                  # @UnusedVariable
-            self.ctools.add(e)
-            
-

distutils.ext/obidistutils/command/build_exe.py

@@ -1,228 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-import os
-
-from distutils import sysconfig
-from distutils.core import Command
-from distutils.sysconfig import customize_compiler as customize_compiler_ori
-from distutils.errors import DistutilsSetupError
-from distutils import log
-from distutils.ccompiler import show_compilers
-
-def customize_compiler(compiler):
-    customize_compiler_ori(compiler)
-    compilername = compiler.compiler[0]
-    if ("gcc" in compilername or "g++" in compilername):
-        cc_cmd = ' '.join(compiler.compiler + ['-fopenmp'])
-        ccshared= ' '.join(x for x in sysconfig.get_config_vars("ccshared") if x is not None)
-            
-        compiler.set_executables(
-            compiler=cc_cmd,
-            compiler_so=cc_cmd + ' ' + ccshared
-            )
-            
-    
-    
-
-class build_exe(Command):
-    
-    description = "build an executable -- Abstract command "
-
-    user_options = [
-        ('build-cexe', 'x',
-         "directory to build C/C++ libraries to"),
-        ('build-temp', 't',
-         "directory to put temporary build by-products"),
-        ('debug', 'g',
-         "compile with debugging information"),
-        ('force', 'f',
-         "forcibly build everything (ignore file timestamps)"),
-        ('compiler=', 'c',
-         "specify the compiler type"),
-        ]
-
-    boolean_options = ['debug', 'force']
-
-    help_options = [
-        ('help-compiler', None,
-         "list available compilers", show_compilers),
-        ]
-
-    def initialize_options(self):
-        self.build_cexe = None
-        self.build_temp = None
-
-        # List of executables to build
-        self.executables = None
-
-        # Compilation options for all libraries
-        self.include_dirs = None
-        self.define = None
-        self.undef = None
-        self.extra_compile_args = None
-        self.debug = None
-        self.force = 0
-        self.compiler = None
-        self.sse = None
-        self.built_files=None
-
-    def finalize_options(self):
-        # This might be confusing: both build-cexe and build-temp default
-        # to build-temp as defined by the "build" command.  This is because
-        # I think that C libraries are really just temporary build
-        # by-products, at least from the point of view of building Python
-        # extensions -- but I want to keep my options open.
-        self.set_undefined_options('build',
-                                   ('build_temp', 'build_temp'),
-                                   ('compiler', 'compiler'),
-                                   ('debug', 'debug'),
-                                   ('force', 'force'))
-
-        if self.include_dirs is None:
-            self.include_dirs = self.distribution.include_dirs or []
-            
-        if isinstance(self.include_dirs, str):
-            self.include_dirs = self.include_dirs.split(os.pathsep)
-
-        self.sse = self.distribution.sse
-        
-        if self.sse is not None:
-            if self.extra_compile_args is None:
-                self.extra_compile_args=['-m%s' % self.sse]
-            else:
-                self.extra_compile_args.append('-m%s' % self.sse)
-
-
-        # XXX same as for build_ext -- what about 'self.define' and
-        # 'self.undef' ?
-
-    def run(self):
-
-        if not self.executables:
-            return
-
-        self.mkpath(self.build_cexe)
-
-        # Yech -- this is cut 'n pasted from build_ext.py!
-        from distutils.ccompiler import new_compiler
-        self.compiler = new_compiler(compiler=self.compiler,
-                                     dry_run=self.dry_run,
-                                     force=self.force)
-        customize_compiler(self.compiler)
-        
-        if self.include_dirs is not None:
-            self.compiler.set_include_dirs(self.include_dirs)
-        if self.define is not None:
-            # 'define' option is a list of (name,value) tuples
-            for (name,value) in self.define:
-                self.compiler.define_macro(name, value)
-
-        if self.undef is not None:
-            for macro in self.undef:
-                self.compiler.undefine_macro(macro)
-                
-        self.build_executables(self.executables)
-
-
-    def check_executable_list(self, executables):
-        """Ensure that the list of executables is valid.
-
-        `executable` is presumably provided as a command option 'executables'.
-        This method checks that it is a list of 2-tuples, where the tuples
-        are (executable_name, build_info_dict).
-
-        Raise DistutilsSetupError if the structure is invalid anywhere;
-        just returns otherwise.
-        """
-        if not isinstance(executables, list):
-            raise DistutilsSetupError("'executables' option must be a list of tuples")
-
-        for exe in executables:
-            if not isinstance(exe, tuple) and len(exe) != 2:
-                raise DistutilsSetupError("each element of 'executables' must a 2-tuple")
-
-            name, build_info = exe
-
-            if not isinstance(name, str):
-                raise DistutilsSetupError(
-                      "first element of each tuple in 'executables' " 
-                      "must be a string (the executables name)")
-                
-            if '/' in name or (os.sep != '/' and os.sep in name):
-                raise DistutilsSetupError(
-                       "bad executable name '%s': " 
-                       "may not contain directory separators" % exe[0])
-
-            if not isinstance(build_info, dict):
-                raise DistutilsSetupError(
-                      "second element of each tuple in 'executables' " 
-                      "must be a dictionary (build info)")
-
-    def get_executable_names(self):
-        # Assume the executables list is valid -- 'check_executable_list()' is
-        # called from 'finalize_options()', so it should be!
-        if not self.executables:
-            return None
-
-        exe_names = []
-        for (exe_name, build_info) in self.executables:                     # @UnusedVariable
-            exe_names.append(exe_name)
-        return exe_names
-
-
-    def get_source_files(self):
-        self.check_executable_list(self.executables)
-        filenames = []
-        for (exe_name, build_info) in self.executables:                     # @UnusedVariable
-            sources = build_info.get('sources')
-            if sources is None or not isinstance(sources, (list, tuple)):
-                raise DistutilsSetupError(
-                       "in 'executables' option (library '%s'), "
-                       "'sources' must be present and must be "
-                       "a list of source filenames" % exe_name)
-
-            filenames.extend(sources)
-        return filenames
-    
-    def substitute_sources(self,exe_name,sources):
-        return list(sources)
-
-    def build_executables(self, executables):
-        for (exe_name, build_info) in executables:
-            sources = build_info.get('sources')
-            if sources is None or not isinstance(sources, (list, tuple)):
-                raise DistutilsSetupError(
-                       "in 'executables' option (library '%s'), " 
-                       "'sources' must be present and must be " 
-                       "a list of source filenames" % exe_name)
-                
-            sources = self.substitute_sources(exe_name,sources)
-
-            log.info("building '%s' program", exe_name)
-
-            # First, compile the source code to object files in the library
-            # directory.  (This should probably change to putting object
-            # files in a temporary build directory.)
-            macros = build_info.get('macros')
-            include_dirs = build_info.get('include_dirs')
-            extra_args = self.extra_compile_args or []
-            
-            objects = self.compiler.compile(sources,
-                                            output_dir=self.build_temp,
-                                            macros=macros,
-                                            include_dirs=include_dirs,
-                                            extra_postargs=extra_args,
-                                            debug=self.debug)
-
-            # Now "link" the object files together into a static library.
-            # (On Unix at least, this isn't really linking -- it just
-            # builds an archive.  Whatever.)
-            self.compiler.link_executable(objects, exe_name,
-                                            output_dir=self.build_cexe,
-                                            debug=self.debug)
-
-        

distutils.ext/obidistutils/command/build_ext.py

@@ -1,143 +0,0 @@
-'''
-Created on 13 fevr. 2014
-
-@author: coissac
-'''
-
-from distutils import log
-import os
-
-from distutils import sysconfig
-from distutils.errors import DistutilsSetupError
-
-def _customize_compiler(compiler):
-    compilername = compiler.compiler[0]
-    if ("gcc" in compilername or "g++" in compilername):
-        cc_cmd  = ' '.join(compiler.compiler + ['-fopenmp'])
-        ccshared= ' '.join(x for x in sysconfig.get_config_vars("ccshared") if x is not None)
-            
-        compiler.set_executables(
-            compiler=cc_cmd,
-            compiler_so=cc_cmd + ' ' + ccshared
-            )
-
-
-try:
-    from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
-    from Cython.Compiler import Options as cython_options  # @UnresolvedImport
-    class build_ext(ori_build_ext):
-    
-        
-        def modifyDocScripts(self):
-            try:
-                os.mkdir("doc/sphinx")
-            except:
-                pass
-            build_dir_file=open("doc/sphinx/build_dir.txt","w")
-            print(self.build_lib,file=build_dir_file)
-            build_dir_file.close()
-            
-        def initialize_options(self):
-            ori_build_ext.initialize_options(self)  # @UndefinedVariable
-            self.littlebigman = None
-            self.built_files = None
-    
-        
-        def finalize_options(self):
-            super(build_ext, self).finalize_options()
-            
-    
-            self.set_undefined_options('littlebigman',
-                                       ('littlebigman',  'littlebigman'))
-            
-            self.set_undefined_options('build_files',
-                                       ('files',  'built_files'))
-    
-            self.cython_c_in_temp = 1
-                       
-            if self.littlebigman =='-DLITTLE_END':
-                if self.define is None:
-                    self.define=[('LITTLE_END',None)]
-                else:
-                    self.define.append('LITTLE_END',None)
-            
-        def substitute_sources(self,exe_name,sources):
-            """
-            Substitutes source file name starting by an @ by the actual
-            name of the built file (see --> build_files)
-            """
-            sources = list(sources)
-            for i in range(len(sources)):
-                message = "%s :-> %s" % (exe_name,sources[i])
-                if sources[i][0]=='@':
-                    try:
-                        filename = self.built_files[sources[i][1:]]
-                    except KeyError:
-                        tmpfilename = os.path.join(self.build_temp,sources[i][1:])
-                        if os.path.isfile       (tmpfilename):
-                            filename = tmpfilename
-                        else:
-                            raise DistutilsSetupError(
-                                 'The %s filename declared in the source '
-                                 'files of the program %s have not been '
-                                 'built by the installation process' % (sources[i],
-                                                                         exe_name))
-                    sources[i]=filename
-                    log.info("%s changed to %s",message,filename)
-                else:
-                    log.info("%s ok",message)
-    
-            return sources
-    
-        def build_extensions(self):
-            # First, sanity-check the 'extensions' list
-            
-            for ext in self.extensions:
-                ext.sources = self.substitute_sources(ext.name,ext.sources)
-                
-            self.check_extensions_list(self.extensions)
-            
-            print("pouic")
-            print(ext.sources)
-            print("pouac")
-            
-            for ext in self.extensions:
-                log.info("%s :-> %s",ext.name,ext.sources)
-                ext.sources = self.cython_sources(ext.sources, ext)
-                self.build_extension(ext)
-    
-        def build_extensions(self):  # TODO what?? double? is it supposed to be build_extension?
-            if hasattr(self, 'compiler'):
-                _customize_compiler(self.compiler)
-            if hasattr(self, 'shlib_compiler'):
-                _customize_compiler(self.shlib_compiler)
-            
-            ori_build_ext.build_extensions(self)
-
-            
-        def run(self):
-            self.modifyDocScripts()
-    
-            for cmd_name in self.get_sub_commands():
-                self.run_command(cmd_name)
-    
-            cython_options.annotate = True
-            ori_build_ext.run(self)  # @UndefinedVariable
-            
-    
-        def has_files(self):
-            return self.distribution.has_files()
-    
-        def has_executables(self):
-            return self.distribution.has_executables()
-        
-        sub_commands = [('build_files',has_files),
-                        ('build_cexe', has_executables)
-                        ] + ori_build_ext.sub_commands 
-
-except ImportError:
-    from distutils.command import build_ext  # @UnusedImport
-           
-
-
-        

distutils.ext/obidistutils/command/build_files.py

@@ -1,63 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-import os.path
-
-from distutils.core import Command
-from distutils import log
-
-class build_files(Command):
-            
-    def initialize_options(self):
-        self.files=None
-        self.ctools=None
-        self.build_temp=None
-        self.build_cexe=None
-
-    def finalize_options(self):
-        
-        self.set_undefined_options('build_ctools',
-                                   ('ctools',  'ctools'),
-                                   ('build_temp','build_temp'),
-                                   ('build_cexe','build_cexe'),
-                                   )
-        
-        self.files = {}
-        
-    def run(self):
-
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
-
-        
-        for dest,prog,command in self.distribution.files:
-            destfile = os.path.join(self.build_temp,dest)
-            if prog in self.ctools:
-                progfile = os.path.join(self.build_cexe,prog)
-            else:
-                progfile = prog 
-                
-            log.info("Building file : %s" % dest)
-            
-            commandline = command % {'prog' : progfile,
-                                     'dest' : destfile}
-
-            log.info("    --> %s" % commandline)
-            
-            os.system(commandline)
-            self.files[dest]=destfile
-            
-            log.info("Done.\n")
-            
-    def has_ctools(self):
-        return self.distribution.has_ctools()
-
-
-    sub_commands = [('build_ctools', has_ctools)] + \
-                   Command.sub_commands
-            
-            
-        

distutils.ext/obidistutils/command/build_scripts.py

@@ -1,104 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-import os.path
-
-from distutils.command.build_scripts import build_scripts as ori_build_scripts
-from distutils.util import convert_path
-from distutils import log, sysconfig
-from distutils.dep_util import newer
-from stat import ST_MODE
-import re
-
-first_line_re = re.compile('^#!.*python[0-9.]*([ \t].*)?$')
-
-class build_scripts(ori_build_scripts):
-            
-    def copy_scripts (self):
-        """Copy each script listed in 'self.scripts'; if it's marked as a
-        Python script in the Unix way (first line matches 'first_line_re',
-        ie. starts with "\#!" and contains "python"), then adjust the first
-        line to refer to the current Python interpreter as we copy.
-        """        
-        self.mkpath(self.build_dir)
-        rawbuild_dir = os.path.join(os.path.dirname(self.build_dir),'raw_scripts')
-        self.mkpath(rawbuild_dir)
-        
-        outfiles = []
-        for script in self.scripts:
-            adjust = 0
-            script = convert_path(script)
-            outfile = os.path.join(self.build_dir, os.path.splitext(os.path.basename(script))[0])
-            rawoutfile = os.path.join(rawbuild_dir, os.path.basename(script))
-            outfiles.append(outfile)
-
-            if not self.force and not newer(script, outfile):
-                log.debug("not copying %s (up-to-date)", script)
-                continue
-
-            # Always open the file but ignore failures in dry-run mode --
-            # that way, we'll get accurate feedback if we can read the
-            # script.
-            try:
-                f = open(script, "r")
-            except IOError:
-                if not self.dry_run:
-                    raise
-                f = None
-            else:
-                first_line = f.readline()
-                if not first_line:
-                    self.warn("%s is an empty file (skipping)" % script)
-                    continue
-
-                match = first_line_re.match(first_line)
-                if match:
-                    adjust = 1
-                    post_interp = match.group(1) or ''
-            
-            log.info("Store the raw script %s -> %s", script,rawoutfile)        
-            self.copy_file(script, rawoutfile)
-
-
-            if adjust:
-                log.info("copying and adjusting %s -> %s", script,
-                         self.build_dir)
-                if not self.dry_run:
-                    outf = open(outfile, "w")
-                    if not sysconfig.python_build:
-                        outf.write("#!%s%s\n" %
-                                   (self.executable,
-                                    post_interp))
-                    else:
-                        outf.write("#!%s%s\n" %
-                                   (os.path.join(
-                            sysconfig.get_config_var("BINDIR"),
-                           "python%s%s" % (sysconfig.get_config_var("VERSION"),
-                                           sysconfig.get_config_var("EXE"))),
-                                    post_interp))
-                    outf.writelines(f.readlines())
-                    outf.close()
-                if f:
-                    f.close()
-            else:
-                if f:
-                    f.close()
-                self.copy_file(script, outfile)
-
-        if os.name == 'posix':
-            for F in outfiles:
-                if self.dry_run:
-                    log.info("changing mode of %s", F)
-                else:
-                    oldmode = os.stat(F)[ST_MODE]
-                    oldmode = oldmode & 0o7777
-                    
-                    newmode = (oldmode | 0o555) & 0o7777
-                    if newmode != oldmode:
-                        log.info("changing mode of %s from %o to %o",
-                                 F, oldmode, newmode)
-                        os.chmod(F, newmode)
-

distutils.ext/obidistutils/command/build_sphinx.py

@@ -1,27 +0,0 @@
-'''
-Created on 10 mars 2015
-
-@author: coissac
-'''
-
-try:
-    from sphinx.setup_command import BuildDoc as ori_build_sphinx  # @UnresolvedImport
-        
-    class build_sphinx(ori_build_sphinx):
-        '''Build Sphinx documentation in html, epub and man formats 
-        '''
-    
-        description = __doc__
-    
-        def run(self):
-            self.builder='html'
-            self.finalize_options()
-            ori_build_sphinx.run(self)
-            self.builder='epub'
-            self.finalize_options()
-            ori_build_sphinx.run(self)
-            self.builder='man'
-            self.finalize_options()
-            ori_build_sphinx.run(self)
-except ImportError:
-    pass  

distutils.ext/obidistutils/command/install.py

@@ -1,19 +0,0 @@
-'''
-Created on 6 oct. 2014
-
-@author: coissac
-'''
-
-# try:
-#     from setuptools.command.install import install as install_ori
-# except ImportError:
-#     from distutils.command.install import install as install_ori
-
-from distutils.command.install import install as install_ori
-
-class install(install_ori):
-    
-    def __init__(self,dist):
-        install_ori.__init__(self, dist)
-#        self.sub_commands.insert(0, ('build',lambda self: True))
-        self.sub_commands.append(('install_sphinx',lambda self: self.distribution.serenity))

distutils.ext/obidistutils/command/install_scripts.py

@@ -1,47 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-# try:
-#     from setuptools.command.install_scripts import install_scripts as ori_install_scripts
-# except ImportError:
-#     from distutils.command.install_scripts import install_scripts as ori_install_scripts
-
-from distutils.command.install_scripts import install_scripts as ori_install_scripts
-
-import os.path
-from distutils import log
-
-class install_scripts(ori_install_scripts):
-
-    def initialize_options(self):
-        ori_install_scripts.initialize_options(self)
-        self.public_dir = None
-                        
-                    
-    def install_public_link(self):
-        self.mkpath(self.public_dir)
-        for file in self.get_outputs():
-            log.info("exporting file %s -> %s", file,os.path.join(self.public_dir,
-                                    os.path.split(file)[1]
-                                    ))
-            if not self.dry_run:
-                dest = os.path.join(self.public_dir,
-                                    os.path.split(file)[1]
-                                   )
-                if os.path.exists(dest):
-                    os.unlink(dest)
-                os.symlink(file,dest)
-                    
-
-
-    def run(self):
-        ori_install_scripts.run(self)
-        if self.distribution.serenity:
-            self.public_dir=os.path.join(self.install_dir,"../export/bin")
-            self.public_dir=os.path.abspath(self.public_dir)
-            self.install_public_link()
-
-

distutils.ext/obidistutils/command/install_sphinx.py

@@ -1,61 +0,0 @@
-'''
-Created on 10 mars 2015
-
-@author: coissac
-'''
-from distutils.core import Command
-import os.path
-import glob
-
-class install_sphinx(Command):
-    '''
-    Install the sphinx documentation
-    '''
-    
-    description = "Install the sphinx documentation in serenity mode"
-
-    boolean_options = ['force', 'skip-build']
-
-
-    def initialize_options (self):
-        self.install_doc = None
-        self.build_dir = None
-
-    def finalize_options (self):
-        self.set_undefined_options('build_sphinx', ('build_dir', 'build_dir'))
-        self.set_undefined_options('install',
-                                   ('install_scripts', 'install_doc'))
-
-    def run (self):
-        if self.distribution.serenity:
-            self.install_doc = os.path.join(self.install_doc,"../export/share")
-            self.install_doc=os.path.abspath(self.install_doc)
-            self.mkpath(self.install_doc)
-            self.mkpath(os.path.join(self.install_doc,'html'))
-            outfiles = self.copy_tree(os.path.join(self.build_dir,'html'),  # @UnusedVariable
-                                      os.path.join(self.install_doc,'html'))
-                
-            self.mkpath(os.path.join(self.install_doc,'man','man1'))
-            outfiles = self.copy_tree(os.path.join(self.build_dir,'man'),  # @UnusedVariable
-                                      os.path.join(self.install_doc,'man','man1'))
-
-            for epub in glob.glob(os.path.join(self.build_dir,'epub/*.epub')):
-                self.copy_file(os.path.join(epub), 
-                               os.path.join(self.install_doc,os.path.split(epub)[1]))
-            
-    def get_outputs(self):
-        directory=os.path.join(self.install_doc,'html')
-        files = [os.path.join(self.install_doc,'html', f) 
-                 for dp, dn, filenames in os.walk(directory) for f in filenames]  # @UnusedVariable
-        
-        directory=os.path.join(self.build_dir,'man')
-        files.append(os.path.join(self.install_doc,'man','man1', f) 
-                 for dp, dn, filenames in os.walk(directory) for f in filenames)  # @UnusedVariable
-
-        directory=os.path.join(self.build_dir,'epub')
-        files.append(os.path.join(self.install_doc, f) 
-                 for dp, dn, filenames in os.walk(directory)  # @UnusedVariable
-                 for f in glob.glob(os.path.join(dp, '*.epub')) )
-        
-        return files
-        

distutils.ext/obidistutils/command/littlebigman.py

@@ -1,59 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-import os
-
-from obidistutils.command.build_exe import build_exe
-from distutils import log
-
-import subprocess
-
-class littlebigman(build_exe):
-    
-    description = "build the littlebigman executable testing endianness of the CPU"
-
-
-    def initialize_options(self):
-        build_exe.initialize_options(self)
-
-        self.littlebigman = None
-
-
-    def finalize_options(self):
-        # This might be confusing: both build-cexe and build-temp default
-        # to build-temp as defined by the "build" command.  This is because
-        # I think that C libraries are really just temporary build
-        # by-products, at least from the point of view of building Python
-        # extensions -- but I want to keep my options open.
-        
-        build_exe.finalize_options(self)
-        
-        self.set_undefined_options('build',
-                                   ('build_temp', 'build_cexe'))
-
-        # self.ctools = self.distribution.ctools
-        
-        if os.path.isfile("distutils.ext/src/littlebigman.c"):
-            self.executables = [('littlebigman',{"sources":["distutils.ext/src/littlebigman.c"]})]
-            self.check_executable_list(self.executables)
-        else:
-            self.executables = []
-
-
-    def run_littlebigman(self):
-        p = subprocess.Popen("'%s'" % os.path.join(self.build_temp,
-                                                   'littlebigman'), 
-                             shell=True, 
-                             stdout=subprocess.PIPE)
-        little = p.communicate()[0]
-        return little.decode('latin1')
-
-    def run(self):
-        build_exe.run(self)
-        self.littlebigman=self.run_littlebigman()
-        log.info("Your CPU is in mode : %s" % self.littlebigman)
-        
-        

distutils.ext/obidistutils/command/pidname.py

@@ -1,55 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-import os
-
-from obidistutils.command.build_exe import build_exe
-from obidistutils.serenity.checksystem import is_mac_system
-
-
-class pidname(build_exe):
-    
-    description = "build the pidname executable returning the executable path from a PID on a mac"
-
-
-    def initialize_options(self):
-        build_exe.initialize_options(self)
-
-        self.pidname = False
-
-
-    def finalize_options(self):
-        # This might be confusing: both build-cexe and build-temp default
-        # to build-temp as defined by the "build" command.  This is because
-        # I think that C libraries are really just temporary build
-        # by-products, at least from the point of view of building Python
-        # extensions -- but I want to keep my options open.
-        
-        build_exe.finalize_options(self)
-        
-        self.set_undefined_options('build',
-                                   ('build_scripts', 'build_cexe'))
-
-        # self.ctools = self.distribution.ctools
-        
-        if os.path.isfile("distutils.ext/src/pidname.c"):
-            self.executables = [('pidname',{"sources":["distutils.ext/src/pidname.c"]})]
-            self.check_executable_list(self.executables)
-        else:
-            self.executables = []
-
-#         self.build_cexe = os.path.join(os.path.dirname(self.build_cexe),'cbinaries') 
-#         self.mkpath(self.build_cexe)
-
-
-
-    def run(self):
-        if is_mac_system():
-            build_exe.run(self)
-            self.pidname=True
-        else:
-            self.pidname=False
-        

distutils.ext/obidistutils/command/sdist.py

@@ -1,42 +0,0 @@
-'''
-Created on 10 mars 2015
-
-@author: coissac
-'''
-
-import os.path
-
-from distutils.command.sdist import sdist as orig_sdist
-from distutils import dir_util
-
-class sdist(orig_sdist):
-
-    def make_distribution(self):
-        """Create the source distribution(s).  First, we create the release
-        tree with 'make_release_tree()'; then, we create all required
-        archive files (according to 'self.formats') from the release tree.
-        Finally, we clean up by blowing away the release tree (unless
-        'self.keep_temp' is true).  The list of archive files created is
-        stored so it can be retrieved later by 'get_archive_files()'.
-        """
-        # Don't warn about missing meta-data here -- should be (and is!)
-        # done elsewhere.
-        base_dir = self.distribution.get_fullname()
-        base_name = os.path.join(self.dist_dir,base_dir)
-
-        self.make_release_tree(os.path.join('tmp',base_dir), self.filelist.files)
-        archive_files = []              # remember names of files we create
-        # tar archive must be created last to avoid overwrite and remove
-        if 'tar' in self.formats:
-            self.formats.append(self.formats.pop(self.formats.index('tar')))
-
-        for fmt in self.formats:
-            file = self.make_archive(base_name, fmt, root_dir='tmp',base_dir=base_dir,
-                                     owner=self.owner, group=self.group)
-            archive_files.append(file)
-            self.distribution.dist_files.append(('sdist', '', file))
-
-        self.archive_files = archive_files
-
-        if not self.keep_temp:
-            dir_util.remove_tree(os.path.join('tmp',base_dir), dry_run=self.dry_run)

distutils.ext/obidistutils/core.py

@@ -1,234 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-from os import path
-import os.path
-import glob
-import sys
-
-try:
-    from setuptools.extension import Extension
-except ImportError:
-    from distutils.extension import Extension
-
-# from distutils.extension import Extension
-
-from obidistutils.serenity.checkpackage import  install_requirements,\
-                                                check_requirements, \
-                                                RequirementError
-                                                
-from obidistutils.serenity.rerun import enforce_good_python
-from obidistutils.serenity.rerun import rerun_with_anothe_python
-from distutils import log
-
-from obidistutils.dist import Distribution
-from obidistutils.serenity import is_serenity
- 
- 
-def findPackage(root,base=None):
-    modules=[]
-    if base is None:
-        base=[]
-    for module in (path.basename(path.dirname(x)) 
-                   for x in glob.glob(path.join(root,'*','__init__.py'))):
-        modules.append('.'.join(base+[module]))
-        modules.extend(findPackage(path.join(root,module),base+[module]))
-    return modules
-    
-def findCython(root,base=None,pyrexs=None):
-    setupdir = os.path.dirname(sys.argv[0])
-    pyrexs=[]
-
-    if base is None:
-        base=[]
-    for module in (path.basename(path.dirname(x)) 
-                   for x in glob.glob(path.join(root,'*','__init__.py'))):
-                               
-        for pyrex in glob.glob(path.join(root,module,'*.pyx')):
-            libabspath = os.path.abspath('obi_libdir')
-            obiabspath = os.path.abspath('.')
-            pyrexs.append(Extension('.'.join(base+[module,path.splitext(path.basename(pyrex))[0]]),
-                                    [pyrex],
-                                    library_dirs=[libabspath],
-                                    include_dirs=[libabspath],
-                                    libraries=["obi3"],
-                                    runtime_library_dirs=[libabspath],
-                                    extra_link_args=["-Wl,-rpath,"+libabspath, "-L"+libabspath]
-                                    )
-                          )
-            try:
-                cfiles = os.path.splitext(pyrex)[0]+".cfiles"
-                cfilesdir = os.path.dirname(cfiles)
-                cfiles = open(cfiles)
-                cfiles = [os.path.relpath(os.path.join(cfilesdir,y),setupdir).strip() 
-                          if y[0] !='@' else y.strip()
-                          for y in cfiles]
-                
-                log.info("Cython module : %s",cfiles)   
-                incdir = set(os.path.dirname(x) for x in cfiles if x[-2:]==".h")
-                #cfiles = [x for x in cfiles if x[-2:]==".c"]                
-                #pyrexs[-1].sources.extend(cfiles)
-                pyrexs[-1].include_dirs.extend(incdir)
-                pyrexs[-1].extra_compile_args.extend(['-msse2',
-                                                      '-Wno-unused-function',
-                                                      '-Wmissing-braces',
-                                                      '-Wchar-subscripts',
-                                                      '-fPIC'
-                                                      ])
-                                
-            except IOError:
-                pass
-            
-        pyrexs.extend(findCython(path.join(root,module),base+[module]))
-    return pyrexs
-    
-
-def rootname(x):
-    return os.path.splitext(x.sources[0])[0]
-
-
-def prepare_commands():
-    from obidistutils.command.build import build
-    from obidistutils.command.littlebigman import littlebigman
-#    from obidistutils.command.serenity import serenity
-    from obidistutils.command.build_cexe import build_cexe
-    from obidistutils.command.build_ext import build_ext
-    from obidistutils.command.build_ctools import build_ctools
-    from obidistutils.command.build_files import build_files
-    from obidistutils.command.build_scripts import build_scripts
-    from obidistutils.command.install_scripts import install_scripts
-    from obidistutils.command.install_sphinx import install_sphinx
-    from obidistutils.command.install import install
-    from obidistutils.command.pidname import pidname
-    from obidistutils.command.sdist import sdist
-    
-        
-    
-    COMMANDS = {'build':build,
-#                'serenity':serenity,
-                'littlebigman':littlebigman,
-                'pidname':pidname,
-                'build_ctools':build_ctools, 
-                'build_files':build_files,
-                'build_cexe':build_cexe, 
-                'build_ext': build_ext,
-                'build_scripts':build_scripts, 
-                'install_scripts':install_scripts,
-                'install_sphinx':install_sphinx,
-                'install':install,
-                'sdist':sdist}
-    
-#     try:
-#         from setuptools.commands import egg_info
-#         COMMANDS['egg_info']=egg_info
-#     except ImportError:
-#         pass
-    try:
-        from obidistutils.command.build_sphinx import build_sphinx
-        COMMANDS['build_sphinx']=build_sphinx
-    except ImportError:
-        pass
-
-    return COMMANDS
-
-
-CTOOLS =[]
-CEXES  =[]
-FILES  =[]
-
-def setup(**attrs):
-    
-    log.set_threshold(log.INFO)
-    
-    minversion      = attrs.get("pythonmin",'3.7')
-    maxversion      = attrs.get('pythonmax',None)    
-    fork            = attrs.get('fork',False)
-    requirementfile = attrs.get('requirements','requirements.txt')
-
-    try:
-        del attrs['pythonmin']
-    except KeyError:
-        pass
-
-    try:
-        del attrs['pythonmax']
-    except KeyError:
-        pass
-
-    try:
-        del attrs['fork']
-    except KeyError:
-        pass
-
-    try:
-        del attrs['requirements']
-    except KeyError:
-        pass
-    
-    if is_serenity():
-    
-        
-        enforce_good_python(minversion, maxversion, fork)
-        
-        if (install_requirements(requirementfile)):
-            rerun_with_anothe_python(sys.executable,minversion,maxversion,fork)
-        
-    
-        try:
-            check_requirements(requirementfile)
-        except RequirementError as e :
-            log.error(e)                                   
-            sys.exit(1)
-        
-    if 'distclass' not in attrs:
-        attrs['distclass']=Distribution
-
-    if 'python_src' not in attrs:
-        SRC = 'python'
-    else:
-        SRC = attrs['python_src']
-        del(attrs['python_src'])
-    
-    if 'scripts' not in attrs:
-        attrs['scripts'] = glob.glob('%s/*.py' % SRC)
-
-    if 'package_dir' not in attrs:
-        attrs['package_dir'] = {'': SRC}
-
-    if 'packages' not in attrs:
-        attrs['packages'] = findPackage(SRC)
-    
-    if 'cmdclass' not in attrs:
-        attrs['cmdclass'] = prepare_commands()
-
-    if 'ctools' not in attrs:
-        attrs['ctools'] = CTOOLS
-    
-    if 'executables' not in attrs:
-        attrs['executables'] = CEXES
-        
-    if 'files' not in attrs:
-        attrs['files'] = FILES
-        
-    if 'sse' not in attrs:
-        attrs['sse']=None
-        
-    if 'serenity' not in attrs:
-        attrs['serenity']=False
-    
-    EXTENTION=findCython(SRC)
-    
-    if 'ext_modules' not in attrs:
-        attrs['ext_modules'] = EXTENTION
-        
-#     try:
-#         from setuptools.core import setup as ori_setup
-#     except ImportError:
-#         from distutils.core import setup as ori_setup
-
-    from distutils.core import setup as ori_setup
-    
-    return ori_setup(**attrs)

distutils.ext/obidistutils/dist.py

@@ -1,58 +0,0 @@
-'''
-Created on 20 oct. 2012
-
-@author: coissac
-'''
-
-try:
-    from setuptools.dist import Distribution as ori_Distribution
-except ImportError:
-    from distutils.dist import Distribution as ori_Distribution
-
-# from distutils.dist import Distribution as ori_Distribution
-
-class Distribution(ori_Distribution):
-    
-    def __init__(self,attrs=None):
-        self.executables = None
-        self.ctools = None
-        self.files = None
-        self.build_cexe  = None
-        self.deprecated_scripts = None
-        self.zip_safe=False
-        self.sse = None
-        self.serenity=attrs['serenity']
-        
-        ori_Distribution.__init__(self, attrs)
-        
-        self.global_options.insert(0,('serenity', None, "install or build the package in a python virtualenv "
-                                                       "without polluting the installed python and with many "
-                                                       "checks during the installation process"
-                                     ))
-        self.global_options.insert(0,('virtualenv', None, "if the installation is done using the serenity mode "
-                                                        "this option allows for specifying the virtualenv name. "
-                                                        "By default the name is PACKAGE-VERSION"
-                                     ))
-        
-    def run_commands(self):
-        """Run each command that was seen on the setup script command line.
-        Uses the list of commands found and cache of command objects
-        created by 'get_command_obj()'.
-        """
-#        self.run_command('littlebigman')
-        ori_Distribution.run_commands(self)
-        
-        
-    def has_executables(self):
-        return self.executables is not None and self.executables
-        
-    def has_ctools(self):
-        return self.ctools is not None and self.ctools
-        
-    def has_files(self):
-        return self.files is not None and self.files
-        
-    def has_deprecated_scripts(self):
-        return self.deprecated_scripts is not None and self.deprecated_scripts
-
-    

distutils.ext/obidistutils/serenity/__init__.py

@@ -1,118 +0,0 @@
-import sys
-
-from distutils import util
-from distutils import sysconfig
-from distutils import log
-from distutils.version import LooseVersion, StrictVersion
-import glob
-import os
-import subprocess
-import re 
-from distutils.errors import DistutilsError
-import tempfile
-
-from importlib.util import spec_from_file_location  # @UnresolvedImport
-import zipimport
-
-import argparse
-
-import base64
-
-from .checkpython import is_python_version
-                        
-                        
-from obidistutils.serenity.rerun import  enforce_good_python
-from obidistutils.serenity.rerun import rerun_with_anothe_python
-
-from obidistutils.serenity.virtual import serenity_virtualenv
-                        
-from obidistutils.serenity.checksystem import is_mac_system, \
-                                              is_windows_system
-                        
-from obidistutils.serenity.checkpackage import install_requirements
-from obidistutils.serenity.checkpackage import check_requirements
-
-from obidistutils.serenity.util import save_argv
-                        
-from obidistutils.serenity.snake import snake
-                            
-    
-def serenity_snake(envname,package,version):
-    old = log.set_threshold(log.INFO)
-
-    log.info("Installing %s (%s) in serenity mode" % (package,version))
-
-    enforce_good_python()
-
-    virtualpython=serenity_virtualenv(envname,package,version)
-    
-    if virtualpython!=os.path.realpath(sys.executable):
-        log.info("Restarting installation within the %s virtualenv" % (envname))
-        rerun_with_anothe_python(virtualpython)
-        
-    log.info("%s will be installed with python : %s" % (package,virtualpython))
-        
-    if install_requirements():    
-        log.info("Restarting installation with all dependencies ok")
-        rerun_with_anothe_python(virtualpython)
-    
-    log.set_threshold(old)
-    
-def serenity_assert(version):
-    check_requirements()
-
-
-def is_serenity():
-    from obidistutils.serenity.globals import local_serenity
-    return local_serenity and local_serenity[0]
-
-def serenity_mode(package,version):
-    
-    save_argv()
-
-    
-    from obidistutils.serenity.globals import saved_args
-    from obidistutils.serenity.globals import local_serenity
-    
-
-    old = log.set_threshold(log.INFO)
-    
-    argparser = argparse.ArgumentParser(add_help=False)
-    argparser.add_argument('--serenity',
-                           dest='serenity', 
-                           action='store_true',
-                           default=True, 
-                           help='Switch the installer in serenity mode. Everythings are installed in a virtualenv')
-
-    argparser.add_argument('--no-serenity',
-                           dest='serenity', 
-                           action='store_false',
-                           default=True, 
-                           help='Switch the installer in the no serenity mode.')
-
-    argparser.add_argument('--virtualenv',
-                           dest='virtual', 
-                           type=str,
-                           action='store',
-                           default="%s-%s" % (package,version), 
-                           help='Specify the name of the virtualenv used by the serenity mode [default: %s-%s]' % (package,version))    
-    
-    args, unknown = argparser.parse_known_args()
-    sys.argv = [sys.argv[0]] + unknown
-    
-    if args.serenity:
-        local_serenity.append(True)
-        serenity_snake(args.virtual,package,version)
-    else:
-        local_serenity.append(False)       
-    
-    log.set_threshold(old)
-    
-    return args.serenity
-    
-    
-def getVersion(source,main,version):
-    path  = os.path.join(source,main,'%s.py' % version)
-    spec = spec_from_file_location('version',path)
-    return spec.loader.load_module().version.strip()
-    

distutils.ext/obidistutils/serenity/bootstrappip.py

@@ -1,36 +0,0 @@
-'''
-Created on 22 janv. 2016
-
-@author: coissac
-'''
-
-import sys
-from urllib import request
-import os.path
-
-from obidistutils.serenity.util import get_serenity_dir
-from obidistutils.serenity.rerun import rerun_with_anothe_python
-from obidistutils.serenity.checkpython import is_a_virtualenv_python
-
-getpipurl="https://bootstrap.pypa.io/get-pip.py"
-
-def bootstrap():
-
-    getpipfile=os.path.join(get_serenity_dir(),"get-pip.py")
-    
-    with request.urlopen(getpipurl) as getpip:
-        with open(getpipfile,"wb") as out:
-            for l in getpip:
-                out.write(l)
-                
-    python = sys.executable
-    
-    if is_a_virtualenv_python():
-        command= "%s %s" % (python,getpipfile)        
-    else:
-        command= "%s %s --user" % (python,getpipfile)
-    
-    os.system(command)
-    
-    rerun_with_anothe_python(python)
-    

distutils.ext/obidistutils/serenity/checkpackage.py

@@ -1,175 +0,0 @@
-'''
-Created on 2 oct. 2014
-
-@author: coissac
-'''
-
-import re
-
-from distutils.version import StrictVersion             # @UnusedImport
-from distutils.errors import DistutilsError
-from distutils import log
-
-import os.path
-import sys
-import subprocess
-
-
-class RequirementError(Exception):
-    pass
-
-def is_installed(requirement):
-    pipcommand = os.path.join(os.path.dirname(sys.executable),'pip')
-    pipjson    = subprocess.run([pipcommand,"list","--format=json"], 
-                                 capture_output=True).stdout
-    packages = eval(pipjson) 
-                                
-    
-    
-    requirement_project,requirement_relation,requirement_version = parse_package_requirement(requirement)
-    
-    package = [x for x in packages if x["name"]==requirement_project]
-    
-    if len(package)==1:
-        if (     requirement_version is not None 
-             and requirement_relation is not None):    
-            rep = (len(package)==1) and eval("StrictVersion('%s') %s StrictVersion('%s')" % (package[0]["version"],
-                                                                                           requirement_relation,
-                                                                                           requirement_version)
-                                             )
-        else:
-            rep=True
-    else:
-        rep=False
-    
-    if rep:
-        if requirement_version is not None and requirement_relation is not None:        
-            log.info("Look for package %s (%s%s) : ok version %s installed" % (requirement_project,
-                                                                               requirement_relation,
-                                                                               requirement_version,
-                                                                               package[0]["version"]))
-        else:
-            log.info("Look for package %s : ok version %s installed" % (requirement_project,
-                                                                        package[0]["version"]))
-    else:
-        if len(package)!=1:
-            if requirement_version is not None and requirement_relation is not None: 
-                log.info("Look for package %s (%s%s) : not installed" % (requirement_project,
-                                                                         requirement_relation,
-                                                                         requirement_version))
-            else:
-                log.info("Look for package %s : not installed" % requirement_project)                
-        else:
-            log.info("Look for package %s (%s%s) : failed only version %s installed" % (requirement_project,
-                                                                                        requirement_relation,
-                                                                                        requirement_version,
-                                                                                        package[0]["version"]))
-        
-    return rep
-
-
-def get_requirements(requirementfile='requirements.txt'):
-    
-    try:
-        requirements = open(requirementfile).readlines()
-        requirements = [x.strip() for x in requirements]
-        requirements = [x for x in requirements if x[0]!='-']
-    
-    except IOError:
-        requirements = []
-        
-    return requirements
-    
-    
-def install_requirements(requirementfile='requirements.txt'):
-    
-    install_something=False
-
-    requirements = get_requirements(requirementfile)
-
-    log.info("Required packages for the installation :")
-    for x in requirements:
-        ok = is_installed(x)
-        if not ok:
-            log.info("  Installing requirement : %s" % x)
-            pip_install_package(x,requirement=requirementfile)
-            install_something=True
-            if x[0:3]=='pip':
-                return True
-                
-    return install_something
- 
-
-def check_requirements(requirementfile='requirements.txt'):
-    
-    requirements = get_requirements(requirementfile)
-    
-    log.info("Required packages for the installation :")
-    for x in requirements:
-        ok = is_installed(x)
-        if not ok:
-            raise RequirementError("  Missing requirement : %s -- Package installation stopped" % x)
-                
- 
-
-
-def parse_package_requirement(requirement):
-    
-    version_pattern = re.compile('[=><]+(.*)$')
-    project_pattern  = re.compile('[^=><]+')
-    relationship_pattern = re.compile('[=><]+')
-    
-    try:
-        requirement_project = project_pattern.search(requirement).group(0)
-        requirement_version = version_pattern.search(requirement)
-        if requirement_version is not None:
-            requirement_version=requirement_version.group(1)
-        requirement_relation= relationship_pattern.search(requirement)
-        if requirement_relation is not None:
-            requirement_relation=requirement_relation.group(0)
-    except:
-        raise DistutilsError("Requirement : %s not correctly formated" % requirement)
-    
-    return requirement_project,requirement_relation,requirement_version
-    
-
-def get_package_requirement(package,requirementfile='requirements.txt'):            
-    requirements = get_requirements(requirementfile)
-    req = [x for x in requirements
-             if x[0:len(package)]==package
-          ]
-    
-    if len(req)==1:
-        return req[0]
-    else:
-        return None
-        
-        
-def pip_install_package(package,directory=None,requirement=None):
-
-    pipcommand = os.path.join(os.path.dirname(sys.executable),'pip')
-    if directory is not None:
-        log.info('    installing %s in directory %s' % (package,str(directory)))
-        
-
-    if 'http_proxy' in os.environ and 'https_proxy' not in os.environ:
-        os.environ['https_proxy']=os.environ['http_proxy']
-
-    args = ['install']
-    
-    if requirement:
-        args.append('--requirement')
-        args.append(requirement)
-    
-    if 'https_proxy' in os.environ:
-        args.append('--proxy=%s' % os.environ['https_proxy'])
-        
-    if directory is not None:
-        args.append('--target=%s' % directory)
-    
-    args.append(package)
-        
-    pip = subprocess.run([pipcommand] + args)
-    
-    return pip
-

distutils.ext/obidistutils/serenity/checkpython.py

@@ -1,138 +0,0 @@
-'''
-Created on 24 mai 2015
-
-@author: coissac
-'''
-from distutils.version import StrictVersion
-from distutils import sysconfig
-import subprocess
-import os
-import glob
-import re
-
-from obidistutils.serenity.checksystem import is_windows_system
-import sys
-
-
-def is_python_version(path=None,minversion='3.4',maxversion=None):
-    '''
-    Checks that the python version is in the range {minversion,maxversion[
-    
-        @param path: if None consider the running python
-                     otherwise the python pointed by the path
-        @param minversion: the minimum version to consider
-        @param maxversion: the maximum version to consider (strictly inferior to)
-                     
-        @return: True if the python version match
-        @rtype: bool
-    '''
-    if path is None:
-        pythonversion = StrictVersion(sysconfig.get_python_version())
-    else:
-        command = """'%s' -c 'from distutils import sysconfig; """ \
-                  """print(sysconfig.get_python_version())'""" % path
-                  
-        p = subprocess.Popen(command, 
-                             shell=True, 
-                             stdout=subprocess.PIPE)
-        pythonversion=str(p.communicate()[0],'utf8').strip()
-        pythonversion = StrictVersion(pythonversion)
-                        
-    return  (        pythonversion >=StrictVersion(minversion) 
-             and (   maxversion is None    
-                  or pythonversion < StrictVersion(maxversion))
-            )
-           
-
-def lookfor_good_python(minversion='3.4',maxversion=None,followLink=False):
-    '''
-    Look for all python interpreters present in the system path that
-    match the version constraints.
-    
-    @param minversion: the minimum version to consider
-    @param maxversion: the maximum version to consider (strictly inferior to)
-    @param followLink: a boolean value indicating if link must be substituted 
-                       by their real path.
-                       
-    @return: a list of path to interpreters
-    '''
-    exe = []
-    if not is_windows_system():
-        paths = os.environ['PATH'].split(os.pathsep)
-        for p in paths:
-            candidates = glob.glob(os.path.join(p,'python*')) 
-            pexe = []
-            pythonpat=re.compile('python([0-9]|[0-9]\.[0-9])?$')
-            for e in candidates:
-                print(e)
-                if pythonpat.search(e) is not None:
-                    if followLink and os.path.islink(e):
-                        e = os.path.realpath(e)
-                    if (os.path.isfile(e) and 
-                       os.access(e, os.X_OK) and 
-                       is_python_version(e,minversion,maxversion)):
-                        pexe.append(e)
-            exe.extend(set(pexe))
-        
-    return exe
-
-def is_a_virtualenv_python(path=None):
-    '''
-    Check if the python is belonging a virtualenv
-    
-        @param path: the path pointing to the python executable.
-                     if path is None then the running python is
-                     considered.
-        @param path: str or None 
-                     
-        @return: True if the python belongs a virtualenv
-                 False otherwise
-        @rtype: bool
-                 
-    '''
-    if path is None:
-        rep = sys.base_exec_prefix != sys.exec_prefix
-    else:
-        command = """'%s' -c 'import sys; print(sys.base_exec_prefix != sys.exec_prefix)'""" % path
-        p = subprocess.Popen(command, 
-                             shell=True, 
-                             stdout=subprocess.PIPE)
-        rep = eval(str(p.communicate()[0],'utf8'))
-        
-    return rep
-
-
-def which_virtualenv(path=None,full=False):
-    '''
-    Returns the name of the virtualenv.
-        @param path: the path to a python binary or None
-                     if you want to consider the running python
-        @type path: str or None
-                     
-        @param full: if set to True, returns the absolute path,
-                     otherwise only return a simple directory name
-        @type full: bool
-                 
-        @return: the virtual environment name or None if the
-                 path does not belong a virtualenv
-        @rtype: str or None
-    '''
-    if path is None:
-        path = sys.executable
-    
-    if is_a_virtualenv_python(path):
-        parts = path.split(os.sep)
-        try:
-            if full:
-                rep = os.sep.join(parts[0:parts.index('bin')])
-                rep = os.path.realpath(rep)
-            else:
-                rep = parts[parts.index('bin')-1]
-        except ValueError:
-            rep = None
-    else:
-        rep=None
-        
-    return rep
-        
-

distutils.ext/obidistutils/serenity/checksystem.py

@@ -1,18 +0,0 @@
-'''
-Created on 24 mai 2015
-
-@author: coissac
-'''
-
-from distutils import util
-
-def is_mac_system():
-    platform = util.get_platform().split('-')[0]
-    
-    return platform=='macosx'
-
-def is_windows_system():
-    platform = util.get_platform().split('-')[0]
-    
-    return platform=='Windows'
-        

distutils.ext/obidistutils/serenity/globals.py

@@ -1,14 +0,0 @@
-'''
-Created on 24 mai 2015
-
-@author: coissac
-'''
-
-
-saved_args=[]
-tmpdir=[]
-local_pip=[]
-local_virtualenv=[]
-local_cython=[]
-local_serenity=[]
-        

distutils.ext/obidistutils/serenity/rerun.py

@@ -1,61 +0,0 @@
-'''
-Created on 24 mai 2015
-
-@author: coissac
-'''
-
-import sys
-import os
-from distutils import log
-from distutils.errors import DistutilsError
-
-
-from obidistutils.serenity.globals import saved_args
-from obidistutils.serenity.checkpython import is_python_version,\
-    lookfor_good_python
-
-
-def rerun_with_anothe_python(path, minversion='3.4',maxversion=None, fork=False):
-        
-    if saved_args:
-        args = saved_args
-    else:
-        args = list(sys.argv)
-        
-          
-    assert is_python_version(path,minversion,maxversion), \
-           'the selected python is not adapted to the installation of this package'
-                   
-    args.insert(0, path)
-        
-    sys.stderr.flush()
-    sys.stdout.flush()
-    
-    if fork:
-        log.info('Forking a new install process')
-        os.system(' '.join(list(args)))
-        log.info('External process ended')
-        sys.exit(0)
-    else:
-        log.info('Install script restarting...')
-        os.execv(path,list(args))
-
-def enforce_good_python(minversion='3.4',maxversion=None, fork=False):
-    if is_python_version(minversion=minversion,maxversion=maxversion):
-        log.info('You are running the good python')
-        return True
-    
-    goodpython = lookfor_good_python(minversion,maxversion)
-    
-    if not goodpython:
-        raise DistutilsError('No good python identified on your system')
-
-    goodpython=goodpython[0]
-    
-    log.warn("========================================")    
-    log.warn("")
-    log.warn("    Switching to python : %s" % goodpython)
-    log.warn("")
-    log.warn("========================================")    
-
-    rerun_with_anothe_python(goodpython)

distutils.ext/obidistutils/serenity/snake.py

@@ -1,35 +0,0 @@
-'''
-Created on 2 oct. 2014
-
-@author: coissac
-'''
-
-snake ="""
-
-        ___
-      ,'._,`.
-     (-.___.-)
-     (-.___.-)
-     `-.___.-'
-      ((  @ @|              .            __
-       \   ` |         ,\   |`.    @|   |  |      _.-._
-      __`.`=-=mm===mm:: |   | |`.   |   |  |    ,'=` '=`.
-     (    `-'|:/  /:/  `/  @| | |   |, @| @|   /---)W(---\ 
-      \ \   / /  / /         @| |   '         (----| |----) ,~
-      |\ \ / /| / /            @|              \---| |---/  |
-      | \ V /||/ /                              `.-| |-,'   |
-      |  `-' |V /                                 \| |/    @'
-      |    , |-'                                 __| |__
-      |    .;: _,-.                         ,--""..| |..""--.
-      ;;:::' "    )                        (`--::__|_|__::--')
-    ,-"      _,  /                          \`--...___...--'/
-   (    -:--'/  /                           /`--...___...--'\  
-    "-._  `"'._/                           /`---...___...---'\  
-        "-._   "---.                      (`---....___....---')
-         .' ",._ ,' )                     |`---....___....---'|
-         /`._|  `|  |                     (`---....___....---')
-        (   \    |  /                      \`---...___...---'/
-         `.  `,  ^""                        `:--...___...--;'
-           `.,'               hh              `-._______.-'
-
-"""

distutils.ext/obidistutils/serenity/util.py

@@ -1,27 +0,0 @@
-'''
-Created on 2 oct. 2014
-
-@author: coissac
-'''
-
-import sys 
-import tempfile
-
-
-from obidistutils.serenity.globals import tmpdir         # @UnusedImport
-from obidistutils.serenity.globals import saved_args     # @UnusedImport
-
-def get_serenity_dir():
-    global tmpdir
-
-    if not tmpdir:
-        tmpdir.append(tempfile.mkdtemp())
-    return tmpdir[0]
-        
-def save_argv():
-    global saved_args
-
-    del saved_args[:]
-    saved_args.extend(list(sys.argv))
-
-           

distutils.ext/obidistutils/serenity/virtual.py

@@ -1,69 +0,0 @@
-'''
-Created on 2 oct. 2014
-
-@author: coissac
-'''
-
-import os
-import sys
-import venv
-
-from distutils.errors import DistutilsError
-from .globals import local_virtualenv  # @UnusedImport
-from .checkpython import   which_virtualenv,\
-                           is_python_version, \
-                           is_a_virtualenv_python
-                                                
-
-    
-    
-def serenity_virtualenv(envname,package,version,minversion='3.4',maxversion=None):
-        
-    #
-    # Checks if we are already running under the good virtualenv
-    #
-    ve = which_virtualenv(full=True)
-    if ve == os.path.realpath(envname) and is_python_version(minversion=minversion,maxversion=maxversion):
-        return sys.executable
-        
-    #
-    # Check if the virtualenv exist
-    # 
-        
-    python = None
-    
-    if os.path.isdir(envname):
-        python = os.path.join(envname,'bin','python')
-        ok = (is_python_version(python,
-                                minversion=minversion,
-                                maxversion=maxversion) and 
-              is_a_virtualenv_python(python))
-        
-        
-        #
-        # The virtualenv already exist but it is not ok
-        #
-        if not ok:
-            raise DistutilsError("A virtualenv %s already exists but not with the required python")
-                 
-    else:
-        ok = False
-                
-        
-    #
-    # Creates a new virtualenv
-    #
-    if not ok:
-        venv.create(envname, 
-                    system_site_packages=False, 
-                    clear=True, 
-                    symlinks=False, 
-                    with_pip=True)
-                
-        # check the newly created virtualenv
-        return serenity_virtualenv(envname,package,version)
-    
-    return os.path.realpath(python)
-    
-    
-    

distutils.ext/src/littlebigman.c

@@ -1,24 +0,0 @@
-/*
- * littlebigman.c
- *
- *  Created on: 11 juil. 2012
- *      Author: coissac
- */
-
-#include <stdio.h>
-
-int main(int argc, char *argv[])
-{
-    union { int entier;
-            char caractere[4] ;
-    } test;
-
-    test.entier=0x01020304;
-
-    if (test.caractere[3] == 1)
-       printf("-DLITTLE_END");
-    else
-        printf("-DBIG_END");
-
-	return 0;
-}

distutils.ext/src/pidname.c

@@ -1,24 +0,0 @@
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <errno.h>
-#include <libproc.h>
-
-int main (int argc, char* argv[])
-{
-	pid_t pid; int ret;
-	char pathbuf[PROC_PIDPATHINFO_MAXSIZE];
-
-	if ( argc > 1 ) {
-		pid = (pid_t) atoi(argv[1]);
-		ret = proc_pidpath (pid, pathbuf, sizeof(pathbuf));
-		if ( ret <= 0 ) {
-			fprintf(stderr, "PID %d: proc_pidpath ();\n", pid);
-			fprintf(stderr,	"    %s\n", strerror(errno));
-		} else {
-			printf("proc %d: %s\n", pid, pathbuf);
-		}
-	}
-
-	return 0;
-}

doc/.gitignore

@@ -1,5 +0,0 @@
-/build/
-/doxygen/
-/build_dir.txt
-/.DS_Store
-/.gitignore

doc/Makefile

@@ -1,203 +0,0 @@
-# Makefile for Sphinx documentation
-#
-
-# You can set these variables from the command line.
-SPHINXOPTS    =
-SPHINXBUILD   = sphinx-build
-PAPER         =
-BUILDDIR      = build
-DOXYGENDIR    = doxygen
-
-# User-friendly check for sphinx-build
-ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1)
-$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/)
-endif
-
-# Internal variables.
-PAPEROPT_a4     = -D latex_paper_size=a4
-PAPEROPT_letter = -D latex_paper_size=letter
-ALLSPHINXOPTS   = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
-# the i18n builder cannot share the environment and doctrees with the others
-I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source
-
-.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext
-
-help:
-	@echo "Please use \`make <target>' where <target> is one of"
-	@echo "  html       to make standalone HTML files"
-	@echo "  dirhtml    to make HTML files named index.html in directories"
-	@echo "  singlehtml to make a single large HTML file"
-	@echo "  pickle     to make pickle files"
-	@echo "  json       to make JSON files"
-	@echo "  htmlhelp   to make HTML files and a HTML help project"
-	@echo "  qthelp     to make HTML files and a qthelp project"
-	@echo "  applehelp  to make an Apple Help Book"
-	@echo "  devhelp    to make HTML files and a Devhelp project"
-	@echo "  epub       to make an epub"
-	@echo "  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter"
-	@echo "  latexpdf   to make LaTeX files and run them through pdflatex"
-	@echo "  latexpdfja to make LaTeX files and run them through platex/dvipdfmx"
-	@echo "  text       to make text files"
-	@echo "  man        to make manual pages"
-	@echo "  texinfo    to make Texinfo files"
-	@echo "  info       to make Texinfo files and run them through makeinfo"
-	@echo "  gettext    to make PO message catalogs"
-	@echo "  changes    to make an overview of all changed/added/deprecated items"
-	@echo "  xml        to make Docutils-native XML files"
-	@echo "  pseudoxml  to make pseudoxml-XML files for display purposes"
-	@echo "  linkcheck  to check all external links for integrity"
-	@echo "  doctest    to run all doctests embedded in the documentation (if enabled)"
-	@echo "  coverage   to run coverage check of the documentation (if enabled)"
-
-clean:
-	rm -rf $(BUILDDIR)/*
-	rm -rf $(DOXYGENDIR)/*
-
-html:
-	@echo "Generating Doxygen documentation..."
-	doxygen Doxyfile
-	@echo "Doxygen documentation generated. \n"
-	$(SPHINXBUILD) -b html -c ./ $(ALLSPHINXOPTS) $(BUILDDIR)/html
-	@echo
-	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
-
-dirhtml:
-	$(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml
-	@echo
-	@echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml."
-
-singlehtml:
-	$(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml
-	@echo
-	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
-
-pickle:
-	$(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle
-	@echo
-	@echo "Build finished; now you can process the pickle files."
-
-json:
-	$(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json
-	@echo
-	@echo "Build finished; now you can process the JSON files."
-
-htmlhelp:
-	$(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp
-	@echo
-	@echo "Build finished; now you can run HTML Help Workshop with the" \
-	      ".hhp project file in $(BUILDDIR)/htmlhelp."
-
-qthelp:
-	$(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp
-	@echo
-	@echo "Build finished; now you can run "qcollectiongenerator" with the" \
-	      ".qhcp project file in $(BUILDDIR)/qthelp, like this:"
-	@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/OBITools-3.qhcp"
-	@echo "To view the help file:"
-	@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/OBITools-3.qhc"
-
-applehelp:
-	$(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp
-	@echo
-	@echo "Build finished. The help book is in $(BUILDDIR)/applehelp."
-	@echo "N.B. You won't be able to view it unless you put it in" \
-	      "~/Library/Documentation/Help or install it in your application" \
-	      "bundle."
-
-devhelp:
-	$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
-	@echo
-	@echo "Build finished."
-	@echo "To view the help file:"
-	@echo "# mkdir -p $$HOME/.local/share/devhelp/OBITools-3"
-	@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/OBITools-3"
-	@echo "# devhelp"
-
-epub:
-	$(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub
-	@echo
-	@echo "Build finished. The epub file is in $(BUILDDIR)/epub."
-
-latex:
-	@echo "Generating Doxygen documentation..."
-	doxygen Doxyfile
-	@echo "Doxygen documentation generated. \n"
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo
-	@echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex."
-	@echo "Run \`make' in that directory to run these through (pdf)latex" \
-	      "(use \`make latexpdf' here to do that automatically)."
-
-latexpdf:
-	@echo "Generating Doxygen documentation..."
-	doxygen Doxyfile
-	@echo "Doxygen documentation generated. \n"
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo "Running LaTeX files through pdflatex..."
-	$(MAKE) -C $(BUILDDIR)/latex all-pdf
-	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
-
-latexpdfja:
-	$(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex
-	@echo "Running LaTeX files through platex and dvipdfmx..."
-	$(MAKE) -C $(BUILDDIR)/latex all-pdf-ja
-	@echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex."
-
-text:
-	$(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text
-	@echo
-	@echo "Build finished. The text files are in $(BUILDDIR)/text."
-
-man:
-	$(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man
-	@echo
-	@echo "Build finished. The manual pages are in $(BUILDDIR)/man."
-
-texinfo:
-	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
-	@echo
-	@echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo."
-	@echo "Run \`make' in that directory to run these through makeinfo" \
-	      "(use \`make info' here to do that automatically)."
-
-info:
-	$(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo
-	@echo "Running Texinfo files through makeinfo..."
-	make -C $(BUILDDIR)/texinfo info
-	@echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo."
-
-gettext:
-	$(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale
-	@echo
-	@echo "Build finished. The message catalogs are in $(BUILDDIR)/locale."
-
-changes:
-	$(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes
-	@echo
-	@echo "The overview file is in $(BUILDDIR)/changes."
-
-linkcheck:
-	$(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck
-	@echo
-	@echo "Link check complete; look for any errors in the above output " \
-	      "or in $(BUILDDIR)/linkcheck/output.txt."
-
-doctest:
-	$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
-	@echo "Testing of doctests in the sources finished, look at the " \
-	      "results in $(BUILDDIR)/doctest/output.txt."
-
-coverage:
-	$(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage
-	@echo "Testing of coverage in the sources finished, look at the " \
-	      "results in $(BUILDDIR)/coverage/python.txt."
-
-xml:
-	$(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml
-	@echo
-	@echo "Build finished. The XML files are in $(BUILDDIR)/xml."
-
-pseudoxml:
-	$(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml
-	@echo
-	@echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml."

doc/conf.py

@@ -1,300 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# OBITools3 documentation build configuration file, created by
-# sphinx-quickstart on Mon May  4 14:36:57 2015.
-#
-# This file is execfile()d with the current directory set to its
-# containing dir.
-#
-# Note that not all possible configuration values are present in this
-# autogenerated file.
-#
-# All configuration values have a default; values that are commented out
-# serve to show the default.
-
-import sys
-import os
-import shlex
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#sys.path.insert(0, os.path.abspath('.'))
-
-# -- General configuration ------------------------------------------------
-
-# If your documentation needs a minimal Sphinx version, state it here.
-#needs_sphinx = '1.0'
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.todo',
-    'sphinx.ext.coverage',
-    'sphinx.ext.imgmath',
-    'sphinx.ext.ifconfig',
-    'sphinx.ext.viewcode',
-    'breathe',
-]
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
-
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
-
-# The encoding of source files.
-#source_encoding = 'utf-8-sig'
-
-# The master toctree document.
-master_doc = 'source/index'
-
-# General information about the project.
-project = u'OBITools3'
-copyright = u'2015, Céline Mercier, Eric Coissac, Frédéric Boyer'
-author = u'Céline Mercier, Eric Coissac, Frédéric Boyer'
-
-# The version info for the project you're documenting, acts as replacement for
-# |version| and |release|, also used in various other places throughout the
-# built documents.
-#
-# The short X.Y version.
-version = '0.0'
-# The full version, including alpha/beta/rc tags.
-release = '0.0.0'
-
-# The language for content autogenerated by Sphinx. Refer to documentation
-# for a list of supported languages.
-#
-# This is also used if you do content translation via gettext catalogs.
-# Usually you set "language" from the command line for these cases.
-language = None
-
-# There are two options for replacing |today|: either, you set today to some
-# non-false value, then it is used:
-#today = ''
-# Else, today_fmt is used as the format for a strftime call.
-#today_fmt = '%B %d, %Y'
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-exclude_patterns = []
-
-# The reST default role (used for this markup: `text`) to use for all
-# documents.
-#default_role = None
-
-# If true, '()' will be appended to :func: etc. cross-reference text.
-#add_function_parentheses = True
-
-# If true, the current module name will be prepended to all description
-# unit titles (such as .. function::).
-#add_module_names = True
-
-# If true, sectionauthor and moduleauthor directives will be shown in the
-# output. They are ignored by default.
-#show_authors = False
-
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
-
-# A list of ignored prefixes for module index sorting.
-#modindex_common_prefix = []
-
-# If true, keep warnings as "system message" paragraphs in the built documents.
-#keep_warnings = False
-
-# If true, `todo` and `todoList` produce output, else they produce nothing.
-todo_include_todos = True
-
-
-# -- Options for HTML output ----------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-html_theme = 'bizstyle'
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-#html_theme_options = {}
-
-# Add any paths that contain custom themes here, relative to this directory.
-#html_theme_path = []
-
-# The name for this set of Sphinx documents.  If None, it defaults to
-# "<project> v<release> documentation".
-#html_title = None
-
-# A shorter title for the navigation bar.  Default is the same as html_title.
-#html_short_title = None
-
-# The name of an image file (relative to this directory) to place at the top
-# of the sidebar.
-#html_logo = None
-
-# The name of an image file (within the static path) to use as favicon of the
-# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
-# pixels large.
-#html_favicon = None
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
-
-# Add any extra paths that contain custom files (such as robots.txt or
-# .htaccess) here, relative to this directory. These files are copied
-# directly to the root of the documentation.
-#html_extra_path = []
-
-# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
-# using the given strftime format.
-#html_last_updated_fmt = '%b %d, %Y'
-
-# If true, SmartyPants will be used to convert quotes and dashes to
-# typographically correct entities.
-#html_use_smartypants = True
-
-# Custom sidebar templates, maps document names to template names.
-#html_sidebars = {}
-
-# Additional templates that should be rendered to pages, maps page names to
-# template names.
-#html_additional_pages = {}
-
-# If false, no module index is generated.
-#html_domain_indices = True
-
-# If false, no index is generated.
-#html_use_index = True
-
-# If true, the index is split into individual pages for each letter.
-#html_split_index = False
-
-# If true, links to the reST sources are added to the pages.
-#html_show_sourcelink = True
-
-# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
-#html_show_sphinx = True
-
-# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
-#html_show_copyright = True
-
-# If true, an OpenSearch description file will be output, and all pages will
-# contain a <link> tag referring to it.  The value of this option must be the
-# base URL from which the finished HTML is served.
-#html_use_opensearch = ''
-
-# This is the file name suffix for HTML files (e.g. ".xhtml").
-#html_file_suffix = None
-
-# Language to be used for generating the HTML full-text search index.
-# Sphinx supports the following languages:
-#   'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja'
-#   'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr'
-#html_search_language = 'en'
-
-# A dictionary with options for the search language support, empty by default.
-# Now only 'ja' uses this config value
-#html_search_options = {'type': 'default'}
-
-# The name of a javascript file (relative to the configuration directory) that
-# implements a search results scorer. If empty, the default will be used.
-#html_search_scorer = 'scorer.js'
-
-# Output file base name for HTML help builder.
-htmlhelp_basename = 'OBITools3doc'
-
-# -- Options for LaTeX output ---------------------------------------------
-
-latex_elements = {
-# The paper size ('letterpaper' or 'a4paper').
-#'papersize': 'letterpaper',
-
-# The font size ('10pt', '11pt' or '12pt').
-#'pointsize': '10pt',
-
-# Additional stuff for the LaTeX preamble.
-#'preamble': '',
-
-# Latex figure (float) alignment
-#'figure_align': 'htbp',
-}
-
-# Grouping the document tree into LaTeX files. List of tuples
-# (source start file, target name, title,
-#  author, documentclass [howto, manual, or own class]).
-latex_documents = [
-  (master_doc, 'OBITools3.tex', u'OBITools3 Documentation',
-   u'Céline Mercier, Eric Coissac, Frédéric Boyer', 'manual'),
-]
-
-# The name of an image file (relative to this directory) to place at the top of
-# the title page.
-#latex_logo = None
-
-# For "manual" documents, if this is true, then toplevel headings are parts,
-# not chapters.
-#latex_use_parts = False
-
-# If true, show page references after internal links.
-#latex_show_pagerefs = False
-
-# If true, show URL addresses after external links.
-#latex_show_urls = False
-
-# Documents to append as an appendix to all manuals.
-#latex_appendices = []
-
-# If false, no module index is generated.
-#latex_domain_indices = True
-
-
-# -- Options for manual page output ---------------------------------------
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'obitools3', u'OBITools3 Documentation',
-     [author], 1)
-]
-
-# If true, show URL addresses after external links.
-#man_show_urls = False
-
-
-# -- Options for Texinfo output -------------------------------------------
-
-# Grouping the document tree into Texinfo files. List of tuples
-# (source start file, target name, title, author,
-#  dir menu entry, description, category)
-texinfo_documents = [
-  (master_doc, 'OBITools3', u'OBITools3 Documentation',
-   author, 'OBITools3', 'One line description of project.',
-   'Miscellaneous'),
-]
-
-# Documents to append as an appendix to all manuals.
-#texinfo_appendices = []
-
-# If false, no module index is generated.
-#texinfo_domain_indices = True
-
-# How to display URL addresses: 'footnote', 'no', or 'inline'.
-#texinfo_show_urls = 'footnote'
-
-# If true, do not generate a @detailmenu in the "Top" node's menu.
-#texinfo_no_detailmenu = False
-
-#Breathe configuration
-sys.path.append( "breathe/" )
-breathe_projects = { "OBITools3": "doxygen/xml/" }
-breathe_default_project = "OBITools3"
-#breathe_projects_source = {
-#     "auto" : ( "../src", ["obidms.h", "obiavl.h"] )
-#     }

doc/source/DMS.rst

@@ -1,160 +0,0 @@
-*********************************************
-The OBItools3 Data Management System (OBIDMS)
-*********************************************
-
-A complete DNA metabarcoding experiment relies on several kinds of data.
-
-	- The sequence data resulting from the sequencing of the PCR products,
-	- The description of the samples including all their metadata,
-	- One or several reference databases used for the taxonomic annotation,
-	- One or several taxonomy databases.
-
-Up to now, each of these categories of data were stored in separate
-files, and nothing made it mandatory to keep them together.
-
-
-The `Data Management System` (DMS) of OBITools3 can be viewed like a basic 
-database system.
-
-
-OBIDMS UML
-==========
-
-.. image:: ./UML/OBIDMS_UML.png
-
-:download:`html version of the OBIDMS UML file <UML/ObiDMS_UML.class.violet.html>`
-
-
-An OBIDMS directory contains :
-  * one `OBIDMS history file <#obidms-history-files>`_
-  * OBIDMS column directories
-
-
-OBIDMS column directories
-=========================
-
-OBIDMS column directories contain :
-    * all the different versions of one OBIDMS column, under the form of different files (`OBIDMS column files <#obidms-column-files>`_)
-    * one `OBIDMS version file <#obidms-version-files>`_
-
-The directory name is the column attribute with the extension ``.obicol``.
-
-Example: ``count.obicol``
-
-
-OBIDMS column files
-===================
-
-Each OBIDMS column file contains :
-	* a header of a size equal to a multiple of PAGESIZE (PAGESIZE being equal to 4096 bytes
-	  on most systems) containing metadata
-	* Lines of data with the same `OBIType <types.html#obitypes>`_
-
-
-Header
-------
-
-The header of an OBIDMS column contains :
-
-	* Endian byte order
-	* Header size (PAGESIZE multiple)
-	* Number of lines of data
-	* Number of lines of data used
-	* `OBIType <types.html#obitypes>`_ (type of the data)
-	* Date of creation of the file
-	* Version of the OBIDMS column
-	* The column name
-	* Eventual comments
-
-
-Data
-----
-
-A line of data corresponds to a vector of elements. Each element is associated with an element name. 
-Elements names are stored in the header. The correspondance between an element and its name is done 
-using their order in the lists of elements and elements names.  This structure allows the storage of 
-dictionary-like data.
-
-Example: In the header, the attribute ``elements_names`` will be associated with the value ``"sample_1;
-sample_2;sample_3"``, and a line of data with the type ``OBInt_t`` will be stored as an ``OBInt_t`` vector 
-of size three e.g. ``5|8|4``.
-
-
-Mandatory columns
------------------
-
-Some columns must exist in an OBIDMS directory :
-    * sequence identifiers column (type ``OBIStr_t``)
-
-
-File name
----------
-
-Each file is named with the attribute associated to the data it contains, and the number of
-its version, separated by an ``@``, and with the extension ``.odc``.
-
-Example : ``count@3.odc``
-
-
-Modifications
--------------
-
-An OBIDMS column file can only be modified by the process that created it, and while its status is set to Open. 
-
-When a process wants to modify an OBIDMS column file that is closed, it must first clone it. Cloning creates a new version of the
-file that belongs to the process, i.e., only that process can modify that file, as long as its status is set to Open. Once the process
-has finished writing the new version of the column file, it sets the column file's status to Closed, and the file can never be modified
-again.
-
-That means that one column is stored in one file (if there is only one version)
-or more (if there are several versions), and that there is one file per version.
-
-All the versions of one column are stored in one directory.
-
-
-Versioning
-----------
-
-The first version of a column file is numbered 0, and each new version increments that
-number by 1.
-
-The number of the latest version of an OBIDMS column is stored in the `OBIDMS version file <#obidms-version-files>`_ of its directory.
-
-
-OBIDMS version files
-====================
-
-Each OBIDMS column is associated with an OBIDMS version file in its directory, that contains the number of the latest
-version of the column.
-
-File name
----------
-
-OBIDMS version files are named with the attribute associated to the data contained in the column, and
-have the extension ``.odv``.
-
-Example : ``count.odv``
-
-
-OBIDMS views
-============
-
-An OBIDMS view consists of a list of OBIDMS columns and lines. A view includes one version
-of each mandatory column. Only one version of each column is included. All the columns of
-one view contain the same number of lines in the same order.
-
-
-OBIDMS history file
-===================
-
-An OBIDMS history file consists of an ordered list of views and commands, those commands leading
-from one view to the next one.
-
-This history can be represented in the form of a ?? showing all the
-operations ever done in the OBIDMS directory and the views in between them :
-
-.. image:: ./images/history.png
-    :width: 150 px
-    :align: center
-
-

doc/source/UML/OBITypes_UML.class.violet.html

@@ -1,832 +0,0 @@
-<HTML>
-<HEAD>
-<META name="description"
-	content="Violet UML Editor cross format document" />
-<META name="keywords" content="Violet, UML" />
-<META charset="UTF-8" />
-<SCRIPT type="text/javascript">
-	function switchVisibility() {
-		var obj = document.getElementById("content");
-		obj.style.display = (obj.style.display == "block") ? "none" : "block";
-	}
-</SCRIPT>
-</HEAD>
-<BODY>
-	This file was generated with Violet UML Editor 2.1.0.
-	&nbsp;&nbsp;(&nbsp;<A href=# onclick="switchVisibility()">View Source</A>&nbsp;/&nbsp;<A href="http://sourceforge.net/projects/violet/files/violetumleditor/" target="_blank">Download Violet</A>&nbsp;)
-	<BR />
-	<BR />
-	<SCRIPT id="content" type="text/xml"><![CDATA[<ClassDiagramGraph id="1">
-  <nodes id="2">
-    <ClassNode id="3">
-      <children id="4"/>
-      <location class="Point2D.Double" id="5" x="520.0" y="30.0"/>
-      <id id="6" value="a6688f6e-9346-46c6-9cf5-4fa6148f613f"/>
-      <revision>1</revision>
-      <backgroundColor id="7">
-        <red>255</red>
-        <green>255</green>
-        <blue>255</blue>
-        <alpha>255</alpha>
-      </backgroundColor>
-      <borderColor id="8">
-        <red>0</red>
-        <green>0</green>
-        <blue>0</blue>
-        <alpha>255</alpha>
-      </borderColor>
-      <textColor reference="8"/>
-      <name id="9" justification="1" size="3" underlined="false">
-        <text>OBIType_t
-</text>
-      </name>
-      <attributes id="10" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="11" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="12">
-      <children id="13"/>
-      <location class="Point2D.Double" id="14" x="780.0" y="100.0"/>
-      <id id="15" value="7edd4f08-c5e5-4e41-bc05-8b357cb5e629"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="16" justification="1" size="3" underlined="false">
-        <text>OBIContainer_t</text>
-      </name>
-      <attributes id="17" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="18" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="19">
-      <children id="20"/>
-      <location class="Point2D.Double" id="21" x="330.0" y="110.0"/>
-      <id id="22" value="dbb15831-2f0b-4e97-83e7-5ecdda6d6075"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="23" justification="1" size="3" underlined="false">
-        <text>OBIElementary_t</text>
-      </name>
-      <attributes id="24" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="25" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="26">
-      <children id="27"/>
-      <location class="Point2D.Double" id="28" x="670.0" y="240.0"/>
-      <id id="29" value="9693da23-1b47-4bf3-9544-86390a533713"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="30" justification="1" size="3" underlined="false">
-        <text>OBIList_t</text>
-      </name>
-      <attributes id="31" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="32" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="33">
-      <children id="34"/>
-      <location class="Point2D.Double" id="35" x="780.0" y="240.0"/>
-      <id id="36" value="b2f4d561-0c10-4443-b8f6-d3628ab9bcfe"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="37" justification="1" size="3" underlined="false">
-        <text>OBISet_t</text>
-      </name>
-      <attributes id="38" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="39" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="40">
-      <children id="41"/>
-      <location class="Point2D.Double" id="42" x="890.0" y="240.0"/>
-      <id id="43" value="8cc209c6-18c7-4a90-a5d4-ab7246638b2f"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="44" justification="1" size="3" underlined="false">
-        <text>OBIDictionnary_t</text>
-      </name>
-      <attributes id="45" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="46" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="47">
-      <children id="48"/>
-      <location class="Point2D.Double" id="49" x="170.0" y="220.0"/>
-      <id id="50" value="cb77086b-7535-49dc-ab33-b58d16eec496"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="51" justification="1" size="3" underlined="false">
-        <text>OBIAtomic_t</text>
-      </name>
-      <attributes id="52" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="53" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="54">
-      <children id="55"/>
-      <location class="Point2D.Double" id="56" x="500.0" y="240.0"/>
-      <id id="57" value="5a32037d-eaf1-4bbc-977e-06589f1d2ca5"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="58" justification="1" size="3" underlined="false">
-        <text>OBIComposite_t
-</text>
-      </name>
-      <attributes id="59" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="60" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="61">
-      <children id="62"/>
-      <location class="Point2D.Double" id="63" x="560.0" y="400.0"/>
-      <id id="64" value="84fa636d-0c5a-4df8-bac7-79df8e546c12"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="65" justification="1" size="3" underlined="false">
-        <text>OBIString_t</text>
-      </name>
-      <attributes id="66" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="67" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="68">
-      <children id="69"/>
-      <location class="Point2D.Double" id="70" x="450.0" y="400.0"/>
-      <id id="71" value="752b5f9b-0ece-4902-8de5-e2c465d281dd"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="72" justification="1" size="3" underlined="false">
-        <text>OBITaxid_t
-</text>
-      </name>
-      <attributes id="73" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="74" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="75">
-      <children id="76"/>
-      <location class="Point2D.Double" id="77" x="220.0" y="400.0"/>
-      <id id="78" value="9b89f530-cedc-4b33-a36f-371ccfb2ffae"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="79" justification="1" size="3" underlined="false">
-        <text>OBIInteger_t
-</text>
-      </name>
-      <attributes id="80" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="81" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="82">
-      <children id="83"/>
-      <location class="Point2D.Double" id="84" x="0.0" y="400.0"/>
-      <id id="85" value="01da8ca2-da98-4fde-9ffe-9753a3f202bf"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="86" justification="1" size="3" underlined="false">
-        <text>OBIFloat_t</text>
-      </name>
-      <attributes id="87" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="88" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="89">
-      <children id="90"/>
-      <location class="Point2D.Double" id="91" x="110.0" y="400.0"/>
-      <id id="92" value="1134dbf0-087e-4c9b-be7a-6157bb10ebf0"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="93" justification="1" size="3" underlined="false">
-        <text>OBIBool_t
-</text>
-      </name>
-      <attributes id="94" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="95" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-    <ClassNode id="96">
-      <children id="97"/>
-      <location class="Point2D.Double" id="98" x="330.0" y="400.0"/>
-      <id id="99" value="a78dbab0-8879-4149-b5c9-aed41e7da0bb"/>
-      <revision>1</revision>
-      <backgroundColor reference="7"/>
-      <borderColor reference="8"/>
-      <textColor reference="8"/>
-      <name id="100" justification="1" size="3" underlined="false">
-        <text>OBIChar_t</text>
-      </name>
-      <attributes id="101" justification="0" size="4" underlined="false">
-        <text></text>
-      </attributes>
-      <methods id="102" justification="0" size="4" underlined="false">
-        <text></text>
-      </methods>
-    </ClassNode>
-  </nodes>
-  <edges id="103">
-    <InheritanceEdge id="104">
-      <start class="ClassNode" reference="12"/>
-      <end class="ClassNode" reference="3"/>
-      <startLocation class="Point2D.Double" id="105" x="50.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="106" x="70.0" y="40.0"/>
-      <transitionPoints id="107"/>
-      <id id="108" value="debdd86e-d072-4413-9d4c-dcabf70e44f9"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="109">
-      <start class="ClassNode" reference="19"/>
-      <end class="ClassNode" reference="3"/>
-      <startLocation class="Point2D.Double" id="110" x="90.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="111" x="20.0" y="50.0"/>
-      <transitionPoints id="112"/>
-      <id id="113" value="1491704a-dd29-47dc-92e4-2b53c62fd634"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="114">
-      <start class="ClassNode" reference="47"/>
-      <end class="ClassNode" reference="19"/>
-      <startLocation class="Point2D.Double" id="115" x="70.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="116" x="40.0" y="40.0"/>
-      <transitionPoints id="117"/>
-      <id id="118" value="8475a565-b6dd-404b-8dd3-07f89b3a2853"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="119">
-      <start class="ClassNode" reference="54"/>
-      <end class="ClassNode" reference="19"/>
-      <startLocation class="Point2D.Double" id="120" x="60.0" y="20.0"/>
-      <endLocation class="Point2D.Double" id="121" x="80.0" y="40.0"/>
-      <transitionPoints id="122"/>
-      <id id="123" value="a8696d02-b718-4800-bacb-d076aa2ed3ce"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="124">
-      <start class="ClassNode" reference="68"/>
-      <end class="ClassNode" reference="54"/>
-      <startLocation class="Point2D.Double" id="125" x="50.0" y="20.0"/>
-      <endLocation class="Point2D.Double" id="126" x="40.0" y="30.0"/>
-      <transitionPoints id="127"/>
-      <id id="128" value="4c4010cd-e981-4051-8cf6-a28ebc154bc7"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="129">
-      <start class="ClassNode" reference="61"/>
-      <end class="ClassNode" reference="54"/>
-      <startLocation class="Point2D.Double" id="130" x="50.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="131" x="80.0" y="50.0"/>
-      <transitionPoints id="132"/>
-      <id id="133" value="b99845fa-28f7-4625-9228-313aa4a9cd17"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="134">
-      <start class="ClassNode" reference="26"/>
-      <end class="ClassNode" reference="12"/>
-      <startLocation class="Point2D.Double" id="135" x="50.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="136" x="50.0" y="50.0"/>
-      <transitionPoints id="137"/>
-      <id id="138" value="3410cc22-8aee-4667-9dcf-8f24edfe9ce2"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="139">
-      <start class="ClassNode" reference="33"/>
-      <end class="ClassNode" reference="12"/>
-      <startLocation class="Point2D.Double" id="140" x="60.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="141" x="70.0" y="50.0"/>
-      <transitionPoints id="142"/>
-      <id id="143" value="0316447c-b7b7-480a-80cd-87d66ab6452b"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="144">
-      <start class="ClassNode" reference="40"/>
-      <end class="ClassNode" reference="12"/>
-      <startLocation class="Point2D.Double" id="145" x="50.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="146" x="90.0" y="50.0"/>
-      <transitionPoints id="147"/>
-      <id id="148" value="d995fa9f-7a1d-4340-b519-d29256c972ce"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="149">
-      <start class="ClassNode" reference="82"/>
-      <end class="ClassNode" reference="47"/>
-      <startLocation class="Point2D.Double" id="150" x="60.0" y="20.0"/>
-      <endLocation class="Point2D.Double" id="151" x="30.0" y="20.0"/>
-      <transitionPoints id="152"/>
-      <id id="153" value="cdf54d62-5b64-46ab-8fdc-ee0184ea9a43"/>
-      <revision>1</revision>
-      <bentStyle name="STRAIGHT"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="154">
-      <start class="ClassNode" reference="89"/>
-      <end class="ClassNode" reference="47"/>
-      <startLocation class="Point2D.Double" id="155" x="80.0" y="20.0"/>
-      <endLocation class="Point2D.Double" id="156" x="20.0" y="30.0"/>
-      <transitionPoints id="157"/>
-      <id id="158" value="9d2d9603-b709-499b-8357-4d844bfa227d"/>
-      <revision>1</revision>
-      <bentStyle name="STRAIGHT"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="159">
-      <start class="ClassNode" reference="75"/>
-      <end class="ClassNode" reference="47"/>
-      <startLocation class="Point2D.Double" id="160" x="50.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="161" x="70.0" y="50.0"/>
-      <transitionPoints id="162"/>
-      <id id="163" value="4c3e06c4-c978-410f-8925-32a64eea92f8"/>
-      <revision>1</revision>
-      <bentStyle name="STRAIGHT"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <InheritanceEdge id="164">
-      <start class="ClassNode" reference="96"/>
-      <end class="ClassNode" reference="47"/>
-      <startLocation class="Point2D.Double" id="165" x="20.0" y="10.0"/>
-      <endLocation class="Point2D.Double" id="166" x="90.0" y="50.0"/>
-      <transitionPoints id="167"/>
-      <id id="168" value="37c12090-8ace-4fbe-b7f9-e86eb9bf2805"/>
-      <revision>1</revision>
-      <bentStyle name="STRAIGHT"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </InheritanceEdge>
-    <CompositionEdge id="169">
-      <start class="ClassNode" reference="12"/>
-      <end class="ClassNode" reference="19"/>
-      <startLocation class="Point2D.Double" id="170" x="10.0" y="50.0"/>
-      <endLocation class="Point2D.Double" id="171" x="70.0" y="20.0"/>
-      <transitionPoints id="172"/>
-      <id id="173" value="7df9ed88-49f1-4828-9f45-b9d693d877aa"/>
-      <revision>1</revision>
-      <bentStyle name="AUTO"/>
-      <startLabel></startLabel>
-      <middleLabel></middleLabel>
-      <endLabel></endLabel>
-    </CompositionEdge>
-  </edges>
-</ClassDiagramGraph>]]></SCRIPT>
-	<BR />
-	<BR />
-	<IMG alt="embedded diagram image" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA/cAAAGzCAIAAADlsnDLAABO0UlEQVR42u29f1SUdd7/n4g/CtQR
-JyNFIJt0TFI0LNab7tDo3HTfbk1F5+akJpvUsh02qWilo7u2sa4VZ6PiVjY5yWlpmy1W2RZbtjwt
-3WFZsbtkeKvBETbxNCIp50gL6Yh+Xuv7eH3nO78YmBnm1+PxB2fm4vr5vF7X+/24Zq65rssuAAAA
-2HAZDAcKBgCCtDEnAgAAsLN8QiArAMDyAQAAcyUrAAAsHwAAMFeyAgDA8gEAAHMNzqy+uARpAACW
-DwAAWD6WDwCA5QMAAJaP5QMAYPkAAIDlY/kAgOUDAABg+Vg+AGD5AACA5WP5WD4AYPkAAIDlY/kA
-AFg+AABg+Vg+AACWDwAAWD6WDwA0UEQAAABYPpYPAFg+AABg+YDlAwCWDwAAWD6WDwCA5QMAAJaP
-5QMAYPkAAIDlY/kAAFg+AABg+Vg+AGD5AACA5QOWDwBYPgAAYPlYPgAAlg8AAFg+lg8AgOUDAACW
-j+UDAGD5AACA5WP5AIDlAwBA2NHV1eUnyz9//nxHR4eb+UeC5VssFmoMALB8AAAIgJ6WlZUNDAz4
-0PLPnDmzadOmmJiYqKgomUlcXNzLL788ODio/nvllVdedpFZs2atXr36m2++0YYnJCTIi2uuueYy
-Z3z55Zd+CuHs2bP+sPyamhq+DwEALB8AAAJj+UVFRQaDwWw2W61Wn1j+I488ItOWlJScPn365MmT
-a9askbfPPvusZvMLFy5sbm5+7LHHZPiGDRvsLL+jo+PwRRYvXiwjtLS0qLfeu7hTli5dOmPGDN9a
-fnt7e1ZWlslkwvIBAMsHAIDAWL6yUlHS9PT0pqYmLy3/yJEjUVFRkydPPnfunBrS09MzZsyYK664
-oq+vT9n8zTffLC/eeecdWcRzzz1nZ/kaGRkZMsK3336r3q5bt85oNB48eFC9ffLJJ+X8RGn6XXfd
-tWPHjqSkpOTk5D/84Q9qhAMHDtx6662xsbELFiz4+OOPna6tnHuMHz8+Ojpa5rx3716fWH5paamc
-NdXV1V3gtw0AgOUDAEAALV8hip+Wlia6L9I/YkPdtWuXTHjnnXfaDly0aJEMbG5uVjY/e/bszZs3
-y9+cnJxTp055aPlvvPGGvP3pT38qr3t7e+W0Yc+ePWrCKVOmyClKfn7+uHHjrr76ahkoU82YMeP6
-66/fvXt3ampqYmKidsmQLa2trbNmzZo6daqsdnd3tzcxyqxkJvPnz5dzD1k9b86UAACwfAAA8Jnl
-K2pqagwGg6jqyAx169at6nId24H333+/DKyvr1dSPmHChJiYGL1ebzabz58/76Hlywsx++uuu05e
-v/TSS+LT2oQzZ85Uo6kFHThw4M0335QXzzzzTE9Pz4YNG+T1/v37na6wnAmoEwNvkFXNy8uTMw11
-JuMmXgAALB8AAJx7uW9xXIQI8eTJk+VfnZ2dw109dR3OsmXLbAfOnj1bfdp94dIVOwMDAy+88IIM
-XLdunYeWL+Tm5qrvBIxG4yuvvKJNuGDBAvW6tLRURvjd7373/PPPy4tx48ZNuIT64N8fli8pybLm
-zJnj+CPmy8IXjkQALB8AAHxs+X6dW0NDQ0pKyqpVq0a2IIvFoqxaU96vvvpKZjVt2jQ1RLsuf3Bw
-cMqUKXFxcerjfE8s/+2335YhMrlM1d/f72j5q1evlhGOHj2qLu8pLi4ecoXF8qdPn+59jMuXL7/p
-ppsOHDjgv50VrkUIAFg+AAD40fJbWlqys7OzsrLkhTcLeuqpp2TalStXdnV1tbW13X777fJ227Zt
-mpQbjca33npr3bp1So614UNa/pkzZ6ZOnSoD169frw2UCSdNmlRdXf3hhx/K67lz58rAkydPynmF
-nAy8/vrrHR0ddXV1siZO11bWU2bY1NSknTaMLMYvvvjixRdfTExM3Lhxo3aGg+UDAJYPAAABs3zR
-8fz8/NTU1Pr6eu1+miNe0Llz5zZv3hwTE6Mu7RDbrqqqspVyNVx8/d577/3ss888t3xB1jM6Ovro
-0aO2M5w5c+aSJUtkZJ1Op67+F0T65XRCLWv69Onvv/++07Xdu3evnBjIONu3b/fS8oVPPvmkqKhI
-lqsuEMLyAQDLBwCAwFj+xo0bDQZDZWWlr+6XrxgcHGxra/P5s2+XLVt233332Q7Rrtjp7u7Wbt+p
-cerUKe2ON26QabXfAXtj+ep++c3NzWlpaSO+6gnLBwAsHwAAy/d2biUlJU49OAhNbseOHbJWdje2
-t70u3z1zHfBhjHbPvpVTpvLyciwfALB8AAAIgGC5+aw9CE3ugw8+2L17t93AP/3pT3/5y188mdzi
-gP8sX1siRQgAWD4AAASRYGFy3ls+RQgAWD4AAGD5WD5FCABYPgAAlo/JYflYPgBg+QAACBYmh+VT
-GwBYPgAAYPmA5VMbAFg+AABg+Vg+RQgAWD4AAGD5WD5FCABYPgAAlo/JYflYPgBg+QAACBYmh+VT
-GwBYPgAAIFiYHJZPbQBg+QAAgOVj+Vg+AGD5AACA5WP5FCEAYPkAAIDlY/lYPgBg+QAACBYmh+VT
-GwBYPgAAIFiYHJZPbQBg+QAAgOUDlg8AWD4AAGD5WD5FCABYPgAAYPlYPpYPAFg+AACChclh+dQG
-AGD5AAAIFiaH5VMbAFg+AABg+YDlAwCWDwAAWD6WTxECAJYPAABYPpaP5QMAlg8AgGD5dEHgOVg+
-AGD5AAAQYoL1BXgMRQgAWD4AAGD5WD5FCABYPgAA7TWWj+Vj+QCA5QMAIFgAFCEAlg8AAAgWAEUI
-gOUDAACCBRQhAGD5AACAYAFFCABYPgAAIFhAEQIAlg8AgGABUIQAHLBEAACAYAFQhE63FzyHAwTL
-BwAABAsoQraXrADLBwAAOlGgCNlesgIsHwCAThSAImR7yQrLBwAAOlEAipCDbuRZfXEJ0sDyAQAA
-4QCKkO3F8gHLBwBAsAAoQrYXywcsHwAA4QCgCNleLB/LBwAAhAOAIuSgw/KxfAAAQDiAImR7sXws
-H8sHgFFrcyGcnviIcADWy/Zi+YDlAwD9U7jlzA4FWhW2F8sHLB8A6J+wfACKkO3F8rF8AKB/Aiwf
-gCLkoMPysXwAoH+C0ezb2KFAq8L20hIClg8A9E9YPgBFyPZi+Vg+ANA/AZYPQBFy0GH5WD4A0D8B
-lg+0KmwvYPlYPgDQP9G3sUOBVoXtpSUELB8A6J/o29ihQKvC9tISApYPAPRPWD6A/4qQJ1sDlo/l
-AwCWT9/GDgWgFaUlBCwfAOif6NvYoQC0orSEgOUDAP0Tlg8AHHRYPmD5APRPgOUDwAgPuvPnz3d0
-dHR1dY14njKHY8eOHTx48PTp035d+cbGxo8//hjLx/IBAMsHLB8AXB50Z86c2bRpU0xMTFRUlAyP
-i4t7+eWXBwcH1X+vvPJK9WvdWbNmrV69+ptvvtGGJyQkqNf9/f0/+9nPJk2aJKONHTtW/ur1+hGv
-29mzZ92PIItesGCBn5JxXDqWj+UDAJaP5bNDAUKvFX3kkUfkbUlJyenTp0+ePLlmzRp5++yzz2pK
-vXDhwubm5scee0yGb9iwwdHy165dK/+S+Rw/ftxqte7fv/+1114b2YotXbp0xowZ7sc5duzY119/
-7Y9YnC4dy8fyAQDLx/LZoQAh1ooeOXIkKipq8uTJ586dU0N6enrGjBlzxRVX9PX1KZu/+eab5cU7
-77wjUz333HN2lt/W1ibjz58///z5844Lkv9mZ2dPmTIlMTGxuLh4YGBA8+m77rpr165dc+bMWbJk
-yWeffSYD5dRi/Pjx0dHRRqNx7969MuSHP/zhDTfcMHXq1LvvvltriNS0buYjHDhw4NZbb42NjV2w
-YIF2eY8a+dVXX501a9bnn39ut6qOS8fysXwAwPKxfHYoQEi2ouLH8vrOO++0/e+iRYtkYHNzs7L5
-2bNnb968Wf7m5OScOnXKzvJ37twpIz/66KOOS+nv7xe5j4uLq66uLiwslNHkrza5TqdLSUkpKCiQ
-4bm5uTKwtbVV/FucXtaqu7v7wsXvGeT1u+++e/nll99///3atNoVO07n8+23386YMeP666/fvXt3
-amqqrIO6AElGlvMN2ZC1a9d2dHTYra3j0rF8LB8AsHwsnx0KEJKt6NatW9XlOrb/FZ+WgfX19cqM
-J0yYEBMTo9frzWaz9oG9ZvlO56Cora2Vf4l/X7j421zR/fHjx6svDWRyeasuvLnqqquuvfZaNYmo
-+dVXX203H3FukXXtWho7y3ecz5tvvinLfeaZZ3p6ejZs2CCv9+/fr0aWrbBYLK5icbp0LB/LBwAs
-Pwxz5jGcAOHdiqrrcJYtW2b739mzZ8vA1tbWC5eu2BkYGHjhhRdk4Lp16+wsf/fu3TJ88eLFjkt5
-/vnn5V+VlZXqbWZmprw9evSonanPvoijZw8ODq5fv37evHmXX375uHHjtOF2lu84H7VcmWTCJfbs
-2XPBg5/tYvlYPgBg+ZGSM30bQHi3ohaLRXmwdsX8V199Jf+dNm2aGqJdly/OPWXKlLi4OPVxvmb5
-x44dE5+WSRzvwllTUyPDn3jiCfU2KSkpKirqu+++c2/506dPV69/+9vfql8CWK3WjIwMzy3/jTfe
-kAmLi4vt1scTy9eWTkuI5QMAlo/lA0AIt6JPPfWUvF25cqVoeltb2+233y5vt23bppmx0Wh86623
-1q1bJ8OXL1+uDdfusSMeL/+68cYb33//fTlJkJEffvjhb7/9tru7W10HL22I2WyWcbRfzbqyfFkN
-Ga2pqam/v/+VV16R11u3bt25c6fId0xMzD//+U9PLP/kyZNyliInJK+//npHR0ddXZ1slyeWb7t0
-WkIsHwCwfCwfAEK4FT137tzmzZvFodXlc+LHVVVV2n+1++VPnTr13nvv1W5iY2v5Z8+e/cUvfhEb
-G6vGnDx58i233KKM/L333ktMTJSBY8aMueOOO7Q7YLqy/L17986dO1fG3759+6lTpxYuXCivlyxZ
-8tBDD8mL1atXe2L5wocffignJ2p95AxBTj88sXzbpdMSYvkA4C/cPILRE8v3/jmOdjz99NO/+tWv
-sHzBarVSnwBh9lnJ4OBgW1ubl8++7ezsPHLkiOMtNY8fP2776fiQdHd3azPRnsN14sSJM2fODGuV
-5Dyht7d3uBtiu3QsH8sHAL90QmVlZdqlop5bvvfPcdTGUcjcLvj5aYuuGPIZkH6ds9O+7cc//rEM
-/4//+I8R9J0AELSWH2nMdWC4n3cAlg8AI++EioqKDAaD2Wy2+/DYff/k/XMc5XV8fPzhS6gzgdG3
-fE+eAenXOdv1be+88860adO0k5/o6Ggxfj7XB8DyQxGLA1g+lg8Ao9oJtbe3m0ym9PT0pqYmT/on
-75/jaPdaw9byXT1V8YEHHti+fXtSUpKcSLS2tv76179OTEyUMbXHr7h5HKM3z4BUj3JcuXKljHzw
-4EH1ryeffFJOkxwjcvV8Rzd924cffvjv//7vEqPjHTAnTZq0Y8cOyhUAyw/jrLB8LB8A/NUJieKn
-paWJ7ov0u++fvH+Oo3otMl19kZ07d9pZvpunKk6bNk3WUz1QxmAwyMnJPffco51LuJnQm2dAao9y
-fOmll2Tyn/70pzK8t7dXTmzULaLtcPV8R6e7oKWl5cEHHxw7dqz7u93LtnzwwQcULQCWj+UDlg8A
-w+6EampqRJ2Liorc9E/eP8dRvRavVbdrsL1tnLJ8N09VTEpKOn36dH9/f1RU1Lx58+TFiRMnZIQ1
-a9a4n9CbZ0Bqj3KUswgx++uuu05ei/HPnz/fVUpO5+x0F8jJgIePtZJNXrp06bFjxyhdACwfywcs
-HyDMW0afPw9VtHjy5Mnyr87OTqcL9f45jheGumLHzVMVMzMz1cjyX5PJJC9kQZrle/I4Rm+eASnk
-5uaqby2MRuMrr7zijeVLwpcBQPhCJ4XlY/kAEJjPihwnb2hoSElJWbVqlZs5e/8cxyEt381TFd1b
-viePY/TmGZDC22+/LaPJBsp2ublvndPnOzrdBbJFM2fO1G6k7QpZ4nXXXSe7Rl1SBQDh3T5j+YDl
-A9CL+GbylpaW7OzsrKwseTHknL1/jqO6DObdS3z66ae2Pu3mqYruLd+TxzF68wzICxfvIqqusVm/
-fr2biJw+39FV3/biiy+K6C9atCg6OtrR7w0Gw0033aTtGgDA8rF8wPIB6EWGnlxMPT8/PzU1tb6+
-Xrtpo/s5e/8cR7v75aenp9v5tKunKrq3/AsePI7Rm2dAKiQu0fGjR4+6icjp8x3d9G2ffPJJUVFR
-UlLSvHnztFjkdOXf/u3fZK1sdw0AYPlYPmD5APQiQ0y+ceNGg8FQWVk5rPvlK7x/juOQjOypisOd
-cLjPgFy2bNl999033Dl70rc1NzenpaV973vfk9MhOfORsxHHXQMAWD6WD1g+AL3IEJOXlJQ4tWH6
-J1fs2LFDwrG9C77nz3f0pG8Tpy8vL3ezawAAy8fyAcsHoBdxh5uP4emfXPHBBx/s3r3bdojnz3f0
-vG/j83sALB/LBywfgF6E/om+DQCwfFpCwPIB6EXon+jbAADLpyUELB8Aywf6NgCgFaUlxPIBAMsH
-+jYA2megJcTyAQDLp28DANpnWkLA8gEAy6dvAwAsn5YQsHwAehH6J/o2AMDyaQkBywfA8iEQfduw
-broPAFg+lg9YPgC9CP1TsPdtNTU15A9A+4zlA5YPQC9C/xQmfVt7e3tWVpbJZCJ/ANpnLB+wfAB6
-EfqncOjbSktLDQZDXV0d+QPQPmP5gOUD0IvQP4X2Hmxtbd21a9f8+fOLiop6e3vJH4D2GcsHLB+A
-lhHLD20yMjLy8vLS09Obm5vJH4D22RM6OzsDMq3/5ozlY/kAgOWHFdIpSs5z5swZGBhwzH8EEClA
-ULXP/iAjIyMg0/p7zlg+lg8AWH647cHly5ffdNNNBw4c8DJ/dhlA0PKFj2htbc3Ly7vhhhtuu+22
-0Zx2NOdMtWD5AIDlh8kelF7txRdfTExM3Lhxo/ahPpYPgOU7smvXrvT0dKvVunTpUmk3Rm3a0Zwz
-1YLlAwCWHz6WL3zyySdFRUVGo3HPnj1YPgCW75T58+er3/B8+eWXiYmJH3300ehMO5pzplqwfADA
-8sPK8lXfJj1lWlraqlWrsHwAsKO0tLSoqMjVW/9NG6g5A5YPgOVj+eFj+YLVai0vL8fyAcCW9vZ2
-g8Gg3WxXtRUpKSl2t+fy+bSBmjNg+QA4IpYfbpavsFgs7DIA0MjOzlaPzLOlqakpLS1NxNp/0wZq
-zoDlA+CIWH54Wj67DAA0ampqTCaT03/l5+eXl5f7aVr/rRVg+QCA5WP57DKAiMZisQx5s/muri6f
-T+u/tQIsHwCwfCyfXQYAvjzY6TUAywfA8mGUwPIBAMsHLB8AAtCqfgGjAl0sAGD5gOUDAJaP5dPF
-AtDyY/mA5QPQ1mP5WD4AYPlYPmD5AFg+UAwAgOXTBGH5AIDYAcUAAFg+YPkAQKsKFAMAYPmA5QMA
-rSpQDAAc7Fg+YPkAtPVAMQAABzuWD1g+AGIHFAMAYPk0QVg+ACB2QDEAAJYPWD4A0KoCxQAAWD5g
-+QBAqwoUAwBg+YDlA9DWA8UAABzsWD5g+QCIHVAMAIDl0wQBlg+A2AHFAABYPk0Qlg8AtKpAMQAA
-lg9YPgDQqgLFAABYPmD5ALT1QDEAAAc7lg9YPgBiBxQDAGD5NEGA5QMgdkAxAACWTxOE5QMArSpQ
-DACA5QOWDwC0qkAxAACWD1g+AG09UAwAwMGO5QOWD4DYAcUAAFg+TRBg+QCIHVAMAIDl0wRh+QCA
-2AHFAABYPmD5AECrChQDAGD5gOUD0NYDUAwAHOxYPmD5ALT1QDEAAJZPEwRYPgBiBxQDQEDLG2wJ
-TssHn+wjLB8AEDugGIDyBpIkZywfgHYEKAYAypskgZyxfADaEaAYAChvkgQsHwBoR4BiAKC8SZKc
-sXwAoL0GigGA8iZJcsbyAYD2GigGoLyBJMkZywegHQGKAYDyBpIkZywfgHYEKAYAypskAcsHANoR
-oBgAKG+SBCwfAGhHgGIAoLxJkpyxfACgvQaKAShvIElyxvIBaEeAYgCgvIEkyRnLB6AdAYoBgPIm
-SSBnLB+AdgQoBgDKmyQBywcA2hGgGAAob5IkZywfAGivgWIAypvyJklyxvIBaEeAYgCgvIEkyRnL
-B6AdAYoBgPImSSBnLB+AdgQoBgDKmyQBy4cwrWDwHNoRoFMBoLxJkpyxfKCCyYq9ABQDAOVNkuSM
-5QMVjOUDZQZAeQNJkjOWD1Qwlg+UGQDlTZJAzlg+0FKMPKsvLsFeAA5JAMqbJMkZywcqGMtnLwDF
-AJQ35U2S5IzlAxWM5QOHJADlDSRJzlg+0FJg+cAhCUB5kySQM5YPtBRYPnBIAlDeJAlYPlDBWD6W
-DxySAJQ3SZIzlg9UMJbPXgCKAYDyJklyxvKBCsbygUMSgPIGkiRnLB9oKbB84JAEoLxJEsgZywda
-CiwfOCQBKG+SBCwfqGDA8oFDEoDyJklyxvKBCsby2QtAMQBQ3iRJzlg+UMFYPgDFAJQ3kCQ5Y/lA
-BWP5wCEJQHmTJJAzlg+0FFg+cEgCUN4kCVg+UMGA5QOHJADlTZLkjOUDFYzlsxcgWIrhMhgtgrOQ
-wCd7kHBIMqT3AnqBUgCWD+Fp+eQZsa0oe99XWZEkSYb0XiB0qgGwfMDyActnD5IkSWL5QP+E5bMX
-AMsHLB83BZLE8iEM+qfz5893dHR0dXUF7bY0NjZ+/PHHWD6gjNRVeJz/s/eDYQ+SJEmG9F4gdJRi
-CM6cObNp06aYmJioqCgZPy4u7uWXXx4cHFT/vfLKK9WvPWbNmrV69epvvvlGG56QkGA7n3vuuUdG
-k1nZzf/s2bM+2RZZ4oIFC4Y1ybAWjeUDlg9YPm4KJInlQ/goxSOPPCKjlZSUnD59+uTJk2vWrJG3
-zz77rObWCxcubG5ufuyxx2T4hg0bnFq+2P/48eOvuOKK5OTk8+fPa8OXLl06Y8YMn2zLsWPHvv76
-a8/HH+6isXzA8gHLx02BJLF8CBOlOHLkSFRU1OTJk8+dO6eG9PT0jBkzRny9r69P2fzNN98sL955
-5x2Z23PPPefU8l9++eV58+Y9/vjjMk5jY6MaKKcKov7R0dFGo3Hv3r0ypK2tLTs7e8qUKYmJicXF
-xQMDA5qRP/DAA9u3b09KSpKTitbW1l//+tcyzq233trR0aGNc9ddd2mrfeedd06bNk2v169du9Zx
-uxwXjeUDlg9YPnuQJEkSy4dIUYpdu3bJOGLMtgMXLVokA5ubm5XNz549e/PmzfI3Jyfn1KlTTi1/
-8eLFW7Zs+fzzz2XCNWvWqIEi67NmzZo6daospbu7u7+/X8Q9Li6uurq6sLBQxpS/2txE2dPS0u6/
-/34ZbjAY0tPT1SVAtucV6oodmU9ycvLEiRPXr1//5ptvVlZWOm6X3aKxfMDyActnD5IkSWL5EEFK
-sXXrVnW5ju1Apdr19fXKrSdMmBATE6PX681ms3Y1jq3lS6lFRUWpX+4uXLhQRlbfAwjXX3/91Vdf
-rV7X1tbKbAsKCi5c/LGv6P748ePVdwgyt6SkpNOnT4vBy6zmzZsnL06cOGF7zqBZ/u9//3sZLivp
-ftNsF43lA5YPWD57kCRJEsuHCFIKdR3OsmXLbAfOnj1bBra2tl64dMXOwMDACy+8IAPXrVvnaPmP
-P/54bGxszkUMBoOMVl1d7ajazz//vPxL++g9MzNT3h49elTNTd6q4ePGjTOZTPJCFurU8svKymS4
-nJ/43PIj6tmZgOUDls8eJEmSxPIhbJXCYrFMuIh2ifxXX30lU02bNk0N0a7LHxwcnDJlSlxcnPo4
-X7N8q9U6ffr0u+++e91FHn300bFjx956662aast/1euamhqZ8xNPPKHeJiUlRUVFfffdd8O1fDWf
-Bx98cEjL1xYdun05YPl4HpbP3sdNSRKwfBjJUffUU0/JaCtXruzq6mpra7v99tvl7bZt2zS3NhqN
-b731lhi8DF++fLk2XFn+H//4R9tLdITs7OwxY8aoX83KbGWqpqam/v7+7u5uOU+YPXu2lKbZbJbh
-2q9ph2X5aj46na66ulrW+YMPPnC6XbaLxvIBywcsnz1IkiSJ5UNkKcW5c+c2b94spq6uLZk2bVpV
-VZX2X+1++VOnTr333ns/++wzO8u/5557HnjgAdsZKoNXN87fu3fv3Llz5e327dvl7XvvvZeYmChv
-5TTgjjvu0O6MOSzLV/NJSkpSK5aRkeF0u+wWjeUDlg9YPnuQJEkSy4eIU4rBwcG2tjY/Pfu2u7vb
-9ib6x48f9/Dz9SFne/r06WEtGssHLB+wfPYgSZIklg8oRWgw14Gw6cuBQxLPw/LZ+7gpSQKWDxF6
-1FkcwPIBywcsH3BTksTyAaUALB+wfMDycVMgSSwfsHz6cgAsn5aBth03JUmSxPKBnoC+HADLp2Vg
-7+OmJAlYPtAT0JcDhyRHNy0Dex83JUnA8oGjDssHLB+wfMBNSRLLB5QCsHzA8gHLx02BJLF8wPLp
-ywGwfFoG2nbclCRJEssHegL6coAAWf758+c7Ojq8eaa1zOHYsWMHDx4c8jnTo0ljY+PHH3+M5fsE
-74skgHsnCN3Uyzxl8sOHD3/zzTdhZvk+L7OR1VuwFSeWD1g+lg8cksOe1ZkzZzZt2hQTExMVFSVj
-xsXFvfzyy4ODg+q/V1555WUXmTVr1urVqzWlkOEJCQnqdX9//89+9rNJkybJaGPHjpW/er0+SGKU
-9VywYIH29uzZsyOYiYdThbHle18ktq9d7Z2RhT+yfRpYy/c+z5///Ofx8fFq8ltuuSV4MvQmSZ+U
-2WWXuPrqq7Ozsz/77DPP6802B0+KM7AMudewfMDysXyIdMt/5JFHZISSkpLTp0+fPHlyzZo18vbZ
-Z5/VurqFCxc2Nzc/9thjMnzDhg2OPevatWvlXzKf48ePW63W/fv3v/baa0ES47Fjx77++mv1eunS
-pTNmzBjuHDyfKowt3/sicWr5tntnZOGPbJ8G3PK9zPPw4cPiwfn5+QMDAx0dHe+//37wZOhNkj4p
-Mzn5kXz27du3bdu2SZMmjR079q9//asn9WaXw5DFGVg82WtYPmD5WD5EtOUfOXJEdGHy5Mnnzp1T
-Q3p6esaMGXPFFVf09fWpXvPmm2+WF++8847M57nnnrPrWdva2mT8+fPnnz9/3nH+8t/s7OwpU6Yk
-JiYWFxeLlGhd1AMPPLB9+/akpCTpuVtbW3/961/LOLfeeqtYizbOXXfdtWPHDhknOTn5D3/4g/t5
-fvTRR9///vfV8JqaGtuZyAtxhfHjx0dHRxuNxr1798qQAwcOyOJiY2MXLFjg6qt5x6ki0PK9LxJX
-lq/tHae7b8jwh7V3gsfyvc/zz3/+swxfsWLFd999Zztnx5Ie/QxHnKQ/yqyyslLG/K//+i/HepPF
-3XnnndOmTdPr9WvXrnXMwXZkN+2YjLNr1645c+YsWbJE+97A1fAf/vCHN9xww9SpU++++24tGTXy
-q6++OmvWrJUrV8oKHDx4UP3rySefLCoqGvFew/IBy8fyIaItX/oh+a/0drYDFy1aJAObm5tVrzl7
-9uzNmzfL35ycnFOnTtn1pjt37pSRH330UceZ9/f3S6cYFxdXXV1dWFgoo8lfbXLpX9PS0u6//34Z
-bjAY0tPT77nnHrvOW7pVGZ6fnz9u3Lirr77a/TxvvPFG6VNFFt966y35q81Efe0uJxLSiUr/Kpvc
-3d397bffzpgx4/rrr9+9e3dqaqrMU7swwBa7qSLT8r0vkgseXLHjuPuGDH9Yeyd4LN/7PM+ePSuH
-jIwv6f3tb39T/3Va0qOf4YiT9EeZiZ3L5ElJSXb1Js1IcnLyxIkT169f/+abb8rJgGMOtiO7acd0
-Ol1KSkpBQYEMz83NdT/8kUcekfm/++67l19+uTR9tg2dbJScbLz00ksy/k9/+lMZ3tvbK2c4e/bs
-GfFew/IBy8fyIaItf+vWreorctuByrzr6+tVDzRhwoSYmBi9Xm82m7UP7LXe1OkcFLW1tfIv6ecu
-XPxFnXST48ePVx/UyeTS9Z4+fVp60KioqHnz5smLEydOyPhr1qzRFjFz5kxxF22VDhw44Gqe8loU
-R16/9tprtr5u65EiQOpUQZCuXebzzDPP9PT0bNiwQV7v37/faUS2U0Wm5XtfJENavqvdN2T4nu+d
-4LF8n+Qpjnv77bfLJKKqv/nNb9yU9ChnOOIk/VRm48aNk1bCrt5+//vfy2w1z3aagzay+3ZM3qoL
-e6666qprr71Wm9bpcIV4uZyGadfbyMiyRRaLRZ2qidlfd9118lqMf/78+d7sNSwfsHwsH4K3zHyI
-q6Wo776XLVtmO3D27NkysLW19cKlb8kHBgZeeOEFGbhu3Tq73nT37t0yfPHixY4zf/755+VflZWV
-6m1mZqa8PXr0qJpc3mrdsMlkkheyFDvL1wS9tLRU/vW73/3OzTz/+Mc/Tp8+Xd4uWbLk73//u3vL
-V/ORRU+4hNPPzLB8nxTJBQ8+y3e6+8LS8n2SpzJOEUGpYTlPFnF0VdKhYvn+KDP1wUFaWppdvZWV
-lclwOa/wxPLdt2NaAc++iGNha8Pl9HX9+vXz5s27/PLLtS8nLzj8zDc3N1d9fWE0Gl955RUsH7B8
-LB9ghEe3xWJRQqBdafrVV1/J+NOmTVNDtGthpYuaMmVKXFyc+ghN602PHTsmPZZM4njnu5qaGhn+
-xBNPqLdJSUmiI+pK4uFa/urVq1XP6mae6gPOjRs3yghqnR0tXzxSvX7jjTdktOLi4iHTs50qMi3f
-+yK54Nk9dhx335Dhe753gsfyfZKnxoMPPijT1tbWuirpUc5wxEn6o8yUoP/kJz+xqzfVjEh0bnKw
-G9lVO+a55f/2t79VVyRardaMjAxXlv/222+rQ0A2sL+/35u9huUDlo/lQ6Qf3U899ZSMsHLlStH0
-trY2dRnAtm3btB7IaDS+9dZb69atk+HLly937E2l/5N/3Xjjje+//750zDLyww8//O2333Z3d6vr
-TeVIMZvNMo72azYPLX/SpEnV1dUffvihvJ47d+6Fi192O52nLO7VV1/9xz/+cfLkyalTp952222O
-Pahso4zf1NQkfaeMJvYg/ejrr7/e0dFRV1cn2+40H9upItPyfVIk6rKEdy+xb98+273javcNGb7n
-eyd4LN/7PPfu3bt9+/YjR44cOnRIxhwzZkxra6urkh7lDL1J0odl9vvf/15GGzt2rMh0b2+vXWug
-mhGdTifNiyzrgw8+cMzBbmRX7Zjnlv/KK6+oLxB27twpgh4TE/PPf/7T0fLPnDkjh4CMuX79ejcJ
-e7LXsHzA8rF8iPSj+9y5c5s3b5YuR13bI6JQVVVl+4GWGi4dz7333qvdLMK2Zz179uwvfvGL2NhY
-NebkyZNvueUW1YG99957iYmJMlBE5I477tDuTOeh5c+cOXPJkiUyUPpjdW2uq3meOnVKBsoQ6ddv
-uummTz/91LG7FTeSUwWZUAxJ3srJg0iDWmfpdF3djtBuqsi0fO+LxPZG5tpFFNrecbX7hgzf870T
-VJbvZZ47duxQI0hiixYt0i4mcVrSo5yhN0n6sMyioqIMBsMPfvCDEydO2M5Baw2kGUlKSlIjZ2Rk
-OOZgN7Krdsxzy5ciX7hwobom7aGHHpIXq1evvuDsxvz5+fnR0dHqoiBXeLLXsHzA8rF84Oj+F4OD
-g21tbV4++7azs/PIkSOOt9Q8fvz4CD4mtP0sTbu5nvt5yhDtzhuukLnZrqGMr33U5/lUkWb5vioS
-97jafUOG78neCSrL9z5Pyerw4cNO76/itKRHLUPvk/R3mdlttd2zut3kMLJ2zA7tYV5y+nHmzBmn
-4yxbtuy+++7zcP3d7DUsH7B8LB84uoOU0X/25FwHwqBlCJu23cu9E4SWH5YZRkiS/kN9UWN7F/wR
-7zUsHzjqsHzg6A5S/vSnP/3lL38ZzSVaHMDygwcv9w5uOjoZYvle8sEHH+zevdsnew3LB446LB84
-uiGcWwb2Pm5KkuwFLJ9qoACwfODoBiyfPYibkiSWD3jAJXp6eqxW68im/cc//jHi5R4+fHjE0x4/
-fhzLB45uwPIBNyVJLB/Cvxo8JyYmJjExcd68efPnz584ceLIHsYpExYVFY34WZ7Tpk2Tv2PHjh3x
-HK666irZhJkzZ45gJlg+YPmA5eOmQJJYPoQYX/z/+fDDD5XaXnfddbfccst///d/FxcXl5eXm83m
-xsbG+Rf5YpgcPHhQFD89PV1mu3v37i+Gz7Zt26Kjo/V6/Y9+9KMRTH7ttdeWlpaWlZXl5OTEx8dP
-mTJl+fLljz76aFVV1SeffOL5fKgWwPIBy8dNgSSxfAhJyxeqq6vT0tK0505r9PX1xcbGvvrqq8My
-7E8//VT8vrCwUGa4ePHi7du3j0DTxc4rKio6OzvnzJnz/e9/v6WlZViT5+XlbdmyRdsQmY+ctMjZ
-izqf2bVrF5YPWD5g+exBkiRJLB/C3PKFH/3oR+LldmOWl5cP94P8119/PTk5uba2Vs3h/vvvf/rp
-p4er+J988sm0adPUUzPkTGP58uU33njjRx995PkcfvnLX/7nf/6n3ebs2bNnypQplZWVfJYPWD5g
-+exBkiRJLB8iAqvVmp6eXldXZzvQYDDU1NR4Pofi4mKZSXt7uzZw48aNpaWlw10ZOUlQj6TW5pyf
-n280Gm3n7J7Ozk69Xm/7o+GGhgadTldfX8++BiwfsHz2IEmSJJYPEURXV1diYqJm0mL8CQkJHt5d
-R6bVrtKxHV5ZWVlQUDDcNcnNzS0vL7cbuGXLFhH3pqYmD2cSHx+vbcuePXsmTZr09ttvs5cBywcs
-nz1IkiSJ5UNkIVo/ffr01NRUZeqZmZm2l7a7obm5WV3vXlNT09fXZ/uv+vr6FStWDGs1ZOmxsbF2
-j3+Ts4jq6uro6GhZSktLiyfzycnJkUkuXPoUf+nSpVlZWY6/PQDA8gHLZw+SJEli+RC2lJaWxsfH
-79mzp7i4uLCwUMRdVLu3t9fDyXt6eqqqqrKzs2Uqk8kkei1DZLgYuZw2DPdkQ7tcp7W1VVZs8eLF
-er1eZi6LULP1BDlFKSgokC0SxRfRt1qteXl56enpns8BAMsHLB83BZLE8iFUEf0tKirSrnpXF+iL
-rI/gShtBTgxWrlx5zTXXiFuvWLGivLx84sSJw5qDnCSI0MuZhsFgSE5Ovueee2688Ua7rwg8obGx
-UY4Bu2vxZaNktnZfFABg+YDl46ZERJJYPoQVYs/i4pmZmbafcHd1dUn1eP5TVztSU1PVZ/nyV3xd
-ZuX5dTKyPjJ+fHz8pk2b9u3bJ6ccMkRM3fNvFexm1dDQYDdcTmnk5GHEWweA5dOnsvdxU5IkSSwf
-gprOzk4x8ry8PEcLH7EEt7S0iKPb/mZXBH1Ylu94RY3JZKqoqBjByhw6dMjp8PLycr1ej+gDlg9Y
-PnuQJEkSy4dwo6mpSX1k7uFddDwkJyenpKTEt6uqvhPw+TxF9BsbG6kEwPIBy2cPkiRJYvkQJqjb
-zmjPrvIVPT090dHRrj5BHzG9vb3D+imwh9TU1Mh5DqIPWD5g+exBkiRJLB/CgYqKiuTk5D179vh8
-zqWlpbm5uf5Y5+zsbMc76HuPKH5CQoLnT/4CwPLpU7F83JQkSRLLh6DDarUWFhYajcbOzk5/zFyM
-2fHXrj7BbDYP9777HtLa2mowGPxxCgGA52H57H3clCQBywe/09PTk5GRkZWV5fNLXxT19fWiy769
-yt925XU6nZ9udd/e3h4fH19UVESRAJYPWD5uCiSJ5UMo0dXVJQqel5fnJwu/cPGimrKyMv9tQk5O
-jv8+ce/s7ExJSSksLPRfPgB4HpbP3sdNSRKwfPAlzc3NCQkJpaWl/lPY9vb2kd3V3nNqamqysrL8
-N39Z+czMTJPJRMEAlg9YPm4KJInlQ7BjNpvj4+P9/QPTgoKCvLw8vy6ir68vNja2q6vLf4sYGBjI
-yMhYsWLFCB61C4DnYfnsfdyUJEkSy4dRoqKiQhS/ubnZr0sROdbpdD6/gaYjJpOpsrLSr4uwWq2y
-lPT0dEQfsHz3+ONH/MG8XCwfNwWSxPIhWCgsLExJSRmFHlHOJUSLR2GLqqurMzMzR2FBRUVFqamp
-FouFKoKQaOtHn4yMjIhariI4LR98sgcJhyRDei9g+RFEb29vTk5OVlbW6HwgLUIs/j062+Xvi3Y0
-SktLk5OT29vbKScIIb4YFVpbW/Py8m644Ybbbrvti1EkUMt1JJL3fnhAkiQZZnsBy48UDh06ZDQa
-8/PzBwYGRmFxLS0tCQkJo3Zrmuzs7IqKitFZlhJ92UCKCrB8W3bt2pWeni5H/dKlS1988cVR68kC
-tVwsHzcFksTyIfA0NzfHx8ePmgdfuPi725KSklFbnNlsHp2rgxQ1NTWj8MMGgNDyvPnz56uD4ssv
-v0xMTPzoo4/Ce7lYPm4KJInlQ4Cpra0VJa2vrx+1Jfb09ERHR4/mj+FkibGxsX56PJZTGhoaJFU/
-PdMXIOQoLS21fYSc3dvwWy4AQPCD5Yc5W7ZsGf3ryMvLy0f/BvN+fTyWU/bs2aPX60fz9AkgOJEW
-xmAw2D4Zw2q1pqSk+Pv7rkAtFwAAy4cAU1xcnJqaOso3mJNeVs4rxIBHeWNramqys7NHeaHq5wdV
-VVUUG0QycujV1dXZDWxqakpLS/Prj3MCtVwAACwfAkZfX5/JZJIucHR+a2tLQ0ODwWAY/S5WNnni
-xImjc6cdW2SJsr3+vmE/QNAiJ9iuvrvLz8/33zdsgVouAACWDwGjvb3daDQWFxcH5NOsnJycQCmv
-dPkB6do7OztTUlIKCwv5+BAiDYvFMuTNm/1x7h2o5QIAYPkQMBobG+Pj4wPl2XKCodPpbC+THU2q
-qqqysrICsmjZ5LS0tFWrViH6EOmdSoCecMmTNQEAsPxwpq6uThS/qakpUCtQUlJSUFAQqKWP5uOx
-nC49MzMzIFdJAWD5WD4AAJYftmzZsiUhISGAN5cQu9XpdIcOHQpgCCLZAbxE3mq15ufnp6enj87T
-hQGwfCwfAADLD3MKCwtTU1NH84bxjlRXVwfqghnbdcjMzAzsOhQUFAR8XwBg+QAAWD6ENgMDAyaT
-KScnJ+CfH4tei2QHdh0Ce9GORmlpqdFoHOXHFABg+QAAgOWHCSK1GRkZJSUlAf/RZ2tra3JycjD8
-9jSwF+1oBOR5ZABYPgAAYPkhT1NTk3hkaWlpMKxMfn6+eG0wrEkwXLSjMJvNer2+sbGRWgUsH8sH
-AMDywSPq6up0Ol19fX0wrExPT4+sTJDcnTpILtrRdpOIfpDsJgAsHwAAy4egpqKiIrC307GjsrIy
-JycnePIJkot2FPv27ZOdVVNTQ90Clo/lAwBg+eCSkpKS1NRUi8USJOtjtVoNBkMAb9LvSPBctKNo
-aWnR6/XBc+IBgOUDAGD5EET09fWZTKZVq1YF1e3Y6+vrjUZjUAUVVBftKDo7O5OTk8vKyihjwPKx
-fAAALB/+P8RZ09LSioqKguE+NrbIiUcQymtQXbSj7UE5HZI9SDEDlo/lAwBg+fAvWltbDQZDRUVF
-EJ57xMfH9/b2BtuKBdtFOwoJStYqPz9/YGCAqgYsH8sHAMDyI5r6+nox6eC8T8umTZuKi4uDcMWC
-8KIdRV9fn4h+VlYWog9YPpYPAIDlRy7l5eV6vT54bqdji3iqrFtra2twRheEF+1oueXm5qanpwfV
-7ysAsHwAACwfRomioiJxwSD8QFphNpszMjKCNr3gvGhHYbVa8/LyUlJSOjs7qXPA8rF8AAAsP1JQ
-t9PJysoK5o97xaHr6uqCdvWC9qIdjbKyMqPR2N7eTsEDlo/lAwBg+eGPup1OcXFxMF+63dTUlJyc
-HGw3/LEjaC/a0aioqJAYW1paKHvA8rF8AAAsP5xpbW0V7Qv+JygVFBRs3LgxyFcymC/a0airqwva
-n14AYPkAAFg++ID6+noRvtra2iBfz56eHp1OF8wXwyhkDWNjY4PwRp+O+z0+Pt5sNnMIAJaP5QMA
-YPnhRnl5ucFgCNpb1titqslkColUMzMzq6urg389W1paRPRramo4EADLx/IBALD88KG4uDiEfogp
-ZyONjY0hsaqVlZXZ2dkhsapK9OUMisMBsHwsHwAAyw95BgYG8vLy0tPTe3p6QmKF6+rq5IQkyH93
-qxEqF+0oOjs7U1JSNm3axHEBWD6WDwCA5YcwYp+ZmZmFhYUh9CRUk8lUUVERQiGHykU7CovFkpqa
-WlRUxNEBWD6WDwCA5Yck7e3tBoMhtD64lXXW6XSh8tG4IoQu2tHO/bKysvLy8kLo3A8AywcAwPLh
-XzQ2NiYnJwf/7XTsKCkpKSwsDK11Dq2LdhR9fX1yZmIymRB9wPKxfAAALD9kMJvNCQkJIXeLdDFO
-vV4fio9wCq2LdhRWqzU/Pz+EfrABgOUDAGD5EU1paanRaAyJO2baUVlZmZGREYqZh9xFOxoFBQUp
-KSmdnZ0cOIDlY/kAAFh+kDIwMLBq1SrRzb6+vlBcf1H8EH1yUyhetGN3WojoA5aP5QMAYPnBiLqd
-Tm5ubohead3Y2JiQkBAqN9B0JBQv2tHYsmVLcnJyKH7/A1g+lg8AgOWHM+p2OhUVFaFryXJ+EtL3
-cQ/di3YUDQ0Ncpa1b98+jibA8rF8AAAsPyhobGyMj48P3Q+SL1y8ibtOp5O/obsJIX3RjqK5uVmv
-14vuc0wBlo/lAwBg+QHGbDYnJyc3NTWF9FaUlpbm5OSE+r4I6Yt2FPv27ZMzxpqaGo4swPKxfAAA
-LD+QcpySktLe3h7SW2G1Wg0GQ2NjY6jvjlC/aEdx6NAhOW8MrccPA5aP5QMAYPlhgpixyWQKj5ud
-m81m2ZAw2CldXV0TJ04M6Yt2FBaLJTU1tbCwMHR/5gFYPpYPAIDlh6SEiRbn5eWFh4SFwYUuGitW
-rKisrAyDDZFzFamxVatWIfqA5WP5AABY/migLqgoKSkJD/2SzdHpdCF6909HRPGzsrLCY1tkp5hM
-puzs7LDZO4DlY/kAAFh+kGI2m8Psx5EFBQVyxhI2m2OxWMLjoh2NnJycjIyMcNoiwPKxfAAALD+4
-2LJlS0JCQnNzc9hskbijXq8P9V8P22EymcLmAqQLF38BkpeXl5qaGtL3OQUsH8sHAMDyg5SCggKj
-0djZ2RlOGxUeN6Wxo6KiImwu2tHYuHFjcnJymJUfYPlYPgAAlh9IBgYGVqxYkZmZGQa307FDzlvC
-7xlMspvC7KIdRVlZWXx8fEtLC4ckYPkAAECz6C2dnZ0pKSl5eXnh9wvIffv2GQyGsLyFi8lkCo87
-7dhRW1sroh/qj2ADX1lvoAi/7aWcAADLjzhaWlpEqrZs2RKWWyenLuG6aVVVVRkZGWG5aQ0NDXq9
-Pvy+gYERWC8hkCQAYPkwcp0Kp9vp2NLb26vT6cL1zi0WiyU2NjZct665uTnMbvQEuClJAgBg+aNE
-+N1Ox3ED8/LywngPhtOjvhxpaWmRU9CKigoOVdwUSBIAsHzwlLC8nY4tVqvVYDDs27cvjHdiOD0e
-yynq6WylpaUcsLgpkCQAYPkwBGF8Ox1bGhoa0tPTw3tXWiyWML4kSSEnonK2VlhYGJY/oQbclCQB
-ALB8n3lhWlpaQUFB2DtTdnZ2GF/NoiEnbGF5px1b5HRUTthWrVqF6OOmQJIAgOWDE8L7djq2tLe3
-6/X68LsxqCOi+OF6px1bZFdmXyQS9ingpiQJAIDlD4Pwvp2OHSUXiYQtDdfHYzlitVpXrVolpzSR
-sLGAm5IkAACW7xGVlZXhfTsdWwYGBuR8pr29PUJ2rslkioRrkxT5+fmpqakWi4WDGjcFkgQALD/S
-Cfvb6dghypubmxtRp3DhfacdOwoLC5OTkyOnnnFTIEkAwPLBngi5nY4d6enpjY2NkbO9fX19Yfx4
-LKeUlZXFx8e3tLRwjOOmQJIAgOVHHJFzOx1b9u3bZzQaI21fm0wms9kcUZtcXV2t1+ubmpo40nFT
-IEkAwPIjCPU4oUi4nY4deXl5YX9nSUeqqqoyMzMjbavr6+tF9BsaGjjecVMgSQDA8iMCdTsdcaBI
-2/De3t74+PgIvAeLbHKkXbSjaG5ulj0eIXeOwk2BJAEAy49oIup2OnZs2bKloKAgMvd7hDwFzJGW
-lhY5p62oqODYx02BJAEAyw9bSkpKIup2OnbIth86dCgyt91sNkfUnXZskZ0uZ7alpaW0ALgpkCQA
-YPnhhrqdTnZ2dl9fX2Qm0NjYGIHXpmv09PTodLqIupmSLXJmK+d4EfIoNNwUSBIAsPxIITJvp2NH
-bm5upN1nxg45zYvkK9TlDEdO8/Ly8iL5KMBNgSQBAMsPHyL2djq2dHV1SQgRrneReacdW/r6+rKz
-s00m08DAAC0jbgokCQBYfgjT2NiYkJAQgbfTsWPTRSI8hN7e3okTJ0bsRTsKdelaVlZWBN5xCDcF
-kgQALD9MiOTb6dhitVqTk5O7uro4GEwmUwQ+LsCxHvLz81NTUyP8hAc3BZIEACw/JCkpKRGPsVgs
-VIDZbM7NzSWHCxefCJudnU0OwsaNGzlAcFMgSQDA8kOJgYGBVatWRfLtdOzIzMxsbGwkhwsXL0yP
-jY3lM2zFli1bDAZDxN5YFjcFkgQALD+UsFgsGRkZEX47HVsOHTpkNBrJQcNkMlVVVZGDoqKigqva
-cFMgSQDA8oOdlpYWUZbIfMSpK/Lz85FaW2pra3NycshBQ44XnU7X0NBAFLgpSQIAYPnByJ49e+Lj
-4yP8lvB2DAwMSCbcTcWWvr4+vV7PrSRtqampEdGX8x+iwE1JEgAAyw8uKioqRGdbW1vZ33axFBQU
-kIMdOTk5nA3a0dLSIic/iD5uSpIAAFh+sGC1WgsLC9PS0trb29nZdhiNRs58HGloaMjKyiIHR9GX
-U2U5MyQK3JQkAQCw/AAzMDCQm5ubnp7ORSmONDY2yskPOTg9MxSd5U47jsg5ocFg2LhxI1HgpiQJ
-AIDlBwyLxSJ+X1hYyDXWTsnOzua6FFesWrWKHyU7RU6Y5eQwPz+fu1ThpiQJAIDlBwB1dcGmTZtw
-EVenQJIPTwxwRW1trclkIgc3oi8nQhxcuClJAgBg+aNKY2NjcnLynj172LuuKC4ullMgcnCFnP/o
-dDou9HIj+llZWdnZ2XxRhpuSJAAAlj9KlJeXi+LzMFc3WK1WvV7f1dVFFG5YsWJFXV0dObipory8
-vIyMDM6FcFOSBADA8v1OQUGBKL7FYmG/uqG6ujo7O5sc3FNfX8+ddoYkPz8/NTWVIw43JUkAACzf
-X/T19eXm5q5YsYJbowxJSkoKjzIdEqvVqtPp8NchKSkpSUhI4E61uClJAgBg+b6nq6srLS0tJyeH
-q4SHpLm5WZyM3016gpw3cqcdT5CUDAbDoUOHiAI3JUkAACzfZ7S3t4u2lpWVYa6eUFhYyP3OPcRs
-NnOnHQ+prq7W6/VyDkkUuClJAgBg+T6gtrY2Pj5eDIMd6Qk9PT06nY7f3XpIb2+vxMX9Rj2krq5O
-RJ+LwXBTkgQAwPK9ZdOmTQkJCa2trexFDykrK8vNzSUHz1mxYgXPDvOcxsZGOesmMdyUJAEAsPwR
-YrVa8/PzU1JSuBR4WBgMBj5qHRY1NTUi+uTgOS0tLXq9nt8z4KYkCQCA5Q+bvr4+Ea/09HQupRgW
-9fX1qamp/HphuOeTXOM0XNrb25OTk8vKyogCNyVJAAAsfxgCYTAYCgoKuJ3OcDGZTIjXCOBOOyOg
-s7NTRL+kpISzStyUJAEAsPyhaW1t1ev1xcXFqMNwOXTokE6n4zGlI6C2tpaHiI0Ai8WSlpYm50ic
-kOOmJAkAgOW7w2w2x8fH19fXs89GgJwaFRQUkMMI6Ovri42N5QRpBIjfywlSTk4Op+W4KUkCAGD5
-zlG302lsbGSHjQB1R0juZT5iRFW5b8zIEL9fsWKFBMgn+rgpSQIAYPn2llBUVGQwGCwWC3trZJSX
-l2dmZpLDiKmqquJOO94cwvn5+RkZGT09PaSBm5IkAACW/y96e3uzL4Lie0NaWhoPDvOGgYEBnU5H
-EXrDpk2bjEZjZ2cnUeCmJAkAEOmWL0JgMBhMJhMX9XpDQ0NDfHw810t4SW5ubkVFBTl4Q2lpqZQi
-z7jATUkSACCiLb+lpUWEQLQAxfeSVatWlZSUkIOXmM3mrKwscvCS6upqOa737dtHFLgpSQIAjELz
-BcOAKEmSJCMwSZpRKjacKhYAIsjyg3C1gvPGhUP2T0G4zsF5rQ5JRnKSoXh081lvJFdsKCYJABC8
-lk//RJJAklg+FQtkBQBYPv0TSZIkSWL5VCxJAgBg+SMK64tL0OaSJElGTpL4FhUbThULAFg+0D+R
-JEmSJJZPxWL5AIDl0z8BSZIklg9ULJYPAFg+/RNJkiRJYvnkTJJYPgBg+fRPJAkkieVTsSSJ5QMA
-lk//RJIkCVg+FUvFAgCWD/RPJEmSJInlU7FYPgBg+bSq9E8kSZJYPlCxWD4AYPn0TyRJkiSJ5ZMz
-SWL5AIDl0z+RJJAklk/FApYPAFg+/RNJkiRJYvlULBULAFg+0D+RJEmSJJZPxWL5AIDl06rSP5Ek
-SeJM5EzFYvkAgOXTPwFJkiSWT85ULJYPAFg+/RNJAkli+VQsYPkAgOXTP5EkSZIklk/FUrEAQENB
-q0r/RJIkSZJYPhWL5QMAlk+rSv9EkiSJM5EzFYvlAwCWT/8EJEmSWD5QsVg+AGD59E8kCSSJ5VOx
-gOUDQIAs//z58x0dHV1dXb5aoaeffvpXv/pVZPZPPg8zaPP3VZLBkBg1OeQcjh07dvDgwdOnTwfz
-YR4Qyw9sATc2Nn788ceOw7ds2fLLX/4yhCrWyxhl8sOHD3/zzTde5oblA0D4WP6ZM2c2bdoUExMT
-FRUlM4mLi3v55ZcHBwfVf6+88srLLjJr1qzVq1drDagMT0hIsBtH8ZOf/EQNXLBgwchW6ezZs16O
-ECij8mGYduMMlyHz9zJDXyXpk8S01/7eXt+WcajUZH9//89+9rNJkybJaGPHjpW/er3ewzLza2jB
-YPlexnvNNddc5owvv/zS+4N99uzZycnJfkrbtxXrfZX+/Oc/j4+PV5PfcsstnmxjkFQvlg8AfrT8
-Rx55RKYtKSk5ffr0yZMn16xZI2+fffZZrR1cuHBhc3PzY489JsM3bNjgVLOkeT18iRMnTnjTgC5d
-unTGjBnejBBAo/JJmE7H8a3le5+hr5L0SWJDWr6vtteHZRxCNbl27Vr5l8zn+PHjVqt1//79r732
-mj88ySe7aZQt38t4Ozo6VLO5ePFiGaGlpUW9HZYvHjt27Ouvvx6u5XuZtm8r1ssYJTHx+/z8/IGB
-AYn0/fff92QbXeU2ytWL5QOAvyz/yJEj0jhOnjz53LlzakhPT8+YMWOuuOKKvr4+1YzefPPN8uKd
-d96RRTz33HMeapZt99/W1padnT1lypTExMTi4mJpiNXwH/7whzfccMPUqVPvvvtu1cBJsz5+/Pjo
-6Gij0bh3717HFR5yhAAala/CdDqOqwxdDXejXz7J0CdJ+rz8pKO96667du3aNWfOnCVLlnz22WdO
-t/fAgQO33nprbGysRKR9Zf/pp5+mpqbqdLqCgoI777zztttuU8OdjqwW9Oqrr86aNevzzz/3PuGg
-rUkpMBl//vz558+fd3UyaRe400PbH6EF3PJ9UsCKjIwMGeHbb7/VhjhmKPFKOD/4wQ/ktcVikeTf
-ffddLVg11V//+ldVxjL5zJkzXVm+92n7sGK9j/HPf/6zDF+xYsV3333nZhvtKtA2N6dNh5tmwYd5
-YvkA4C/Ll0ZNJpTGy3bgokWLZGBzc7NqRmfPnr1582b5m5OTc+rUKaeaFR8f336Rzs5OO8vs7+8X
-AY2Li6uuri4sLJQ5y1/t8xtZAemoLr/88vvvv1+GtLa2SvsrHZsM7+7udlzhIUcIoFH5KkzHcVxl
-6CZbN5bvkwx9kqSvErN9Lf1xSkqKdMkyk9zcXMftFZGaMWPG9ddfv3v3bum/JUB1YcDcuXPlZKmq
-qurHP/7x2LFjlR65GlkWJCPLWq1du7ajo8P7hIO2Jnfu3CkjP/roo07n7zRwp4e2P0ILuOX7pIBd
-Wb7TDO+55x4ZTYavXLnye9/7njr1sj3YpVbljPR//ud/Hn74YVFnV5bvfdo+rFjvYzx79qzBYJDx
-JYe//e1vrrbRrgJtc3NVyU6bBd/mieUDgL8sf+vWrep7UtuB0qPIwPr6etX2TZgwISYmRq/Xm81m
-7fM8V9fl2w5UDWhtba0Ml6bzwsVfR4mSjh8/XvvMRpA2UfxJ+65Teqmrr77azToPOUKgjMpXYTqO
-4ypDN9m6v5TC+wx9kqSvErN9LSGob+Gvuuqqa6+91nF733zzTZn/M88809PTs2HDBnm9f/9+6fLl
-RV5enhpHu9TB6chqQbJKFovFVwkHbU06nYOt5TsN3Omh7fPQAm75PilgV5bvNMOuri6ReLFJMc6/
-//3vdo3tsWPHZCarV6+2K2N/pO3DivVJjKL+t99+u0wyceLE3/zmN0630a4C7SzfsZJdNQu+zRPL
-BwB/Wb76AnTZsmW2A6Utk4Gtra0XLn1VOjAw8MILL8jAdevWOVWr+Pj4Ixf56quv7BrQ559/Xias
-rKxUwzMzM+Xt0aNHBwcH169fP2/evMsvv3zcuHFa+xi6lu+rMB3HcZWhq+GhYvm+Sszx3FLNR3Dc
-XhWalNyES+zZs2fHjh0yUJZi1507HfmCB9ejB4nle5/w7t27ZfjixYtdWb5j4K4ObZ+HFnDL90kB
-O7V8VxkKjz/+uIyZnZ3tuBfEbuVf2l2PQsXyfRWj2P9LL70kcUVFRakP1B0t37YC7SzfsZJdNQtY
-PgCEhuVbLBblLtr13KLpMqtp06apIdoFkdLrTJkyJS4uTvuO2MPr8mtqamSGTzzxhBqelJQkTfB3
-333329/+Vl1habVapYeztfzp06e7VwH3IwTKqHwVpuM4rjJ0NdwTy/cyQ58k6fPyc2P52va+8cYb
-soji4mLbNdm3b5/6gam87u3tlQWp7tzpyB4K67ASDtqaPHbsmGiTTOL0/oZOA3d1aPs8tIBbvk8K
-2Knlu8rw5MmTEpGof3R0tPpayTZYOQXVJPjcuXMylXsr9SZtH1asD2MUHnzwQZm2trbWcRuHa/mu
-mgXf5onlA4C/LF946qmnZNqVK1dKL97W1qa+9Ny2bZvW9hmNxrfeekt6Dhm+fPly9x+mOjag3d3d
-6lJIacLMZrPMRP3g6ZVXXpHXW7du3blzpzSOMTEx//znP2W4rIkMb2pq6u/vd7rCQ44QKKPyVZiO
-47jK0NXwIY3K+wx9laRvy8+V5dtur3iS2IN02K+//npHR0ddXZ0sV5RIp9NdccUVP/7xj5csWRIb
-G6umdTqyJ8I63ISDuSblNFL+deONN77//vuiXzLyww8/rHzUaeCuDm2fhxZwy/dJvE4t31WGDzzw
-wKJFi/73f/9XzudFfLVfiahgxYknTpx41VVXbd++/b777pPTMzdW6mXavq1YL2Pcu3evbPKRI0cO
-HTokY44ZM0Z9CWC3jcO1fFfNgm/zxPIBwI+WLw3Z5s2bpRdRF9aL01RVVdm2g2r41KlT7733Xu3O
-A8O6x857772XmJgoM5HG94477lDXPp46dWrhwoUyUFrPhx56SLucVNrruXPnyltptZ2u8JAjBNCo
-fBKm03GcZuhmuHuj8j5DXyXp2/JzZfl22/vhhx+KCqg5i0Kp++698cYbs2bNuuaaa0ToxZPmz5+v
-pnU68pDCOtyEg7kmz549+4tf/EIUR405efLkW265xVHctcBdHdo+Dy0YLN/7eJ1avtMM//znP4u4
-q9sTiXeq0wC7YJ9++mk5AZgwYUJpaan7K0y8TNu3FetljOrSGtUMylmQdhGj3TYO1/LdNAs+zBPL
-BwA/Wr5icHCwra3Nr89uPH78uOOHHNrzTU6cOHHmzBlteHd3t9M793k+QkCMyt9hOs3QzXD3eJOh
-b5MchfJz3F4Rqd7eXu1tR0eH1WpVWi9rnpOTYzut3cg+Tzj4a1I2pLOz88iRIx5ukatD299lGZBn
-3/qpgEeWYV9fnzoH82va/qhYb2KUBvDw4cNO72/jTUW5bxZ8siwsHwD8bvnByVwHfBWWX40qEjIM
-vyQXLlwYGxs7Z84cWW2DwSDGMJoJR1RN+rUsA2L5EZh2hFSs02bBt3li+QAQoZZvcQA3DZIMwy/J
-U6dO/eUvf9m+ffv//d//jX7CkWb5/itLLH900o6QinXaLPg2TywfACLU8v0XVkQZFUmSJElGsuVT
-seFasQCA5QP9E0mSJEli+VQslg8AWD79E5AkSWL5QMVi+QCA5dM/kSRJkiSWT84kieUDAJZP/0SS
-QJJYPhVLklg+AGD59E8kSZKA5VOxVCwAYPlA/0SSJEmSWD4Vi+UDAJZPq0r/RJIkieUDFYvlAwCW
-T/9EkiRJklg+OZMklg8AWD79E0kCSWL5VCxg+QCA5dM/kSRJkiSWT8VSsQCA5QP9E0mSJEli+VQs
-lg8AWD6tKv0TSZIkzkTOVCyWDwBYPv0TkCRJYvnkTMVi+QCA5dM/kSSQJJZPxQKWDwBYPv0TSZIk
-SWL5VCwVCwA0FLSq9E8kSZIkieVTsVg+AGD5tKr0TyRJkjgTOVOxWD4AYPn0T0CSJInlAxWL5QMA
-lk//RJJAklg+FQtYPgBg+fRPJEmSJInlU7EkCQCA5dM/kSRJApZPxWL5AIDlExb9E0mSJM5EzlQs
-lg8AIWL54Dnu+ycgSZIMyyRpRqnYcKpYAIgUy7d98wV4jPtYyYckSTJckyRnKjacKhYAsHygfyJJ
-kiRJLJ+KxfIBAMunfwKSJEksH6hYLB8AgsHyAQAAAAAgDPh/BFQraMwlB3wAAAAASUVORK5C" />
-</BODY>
-</HTML>

doc/source/containers.rst

@@ -1,29 +0,0 @@
-===============
-Container types
-===============
-
-Containers allow to manage collections of values of homogeneous type.
-Three container types exist.
-
-A container is a non-mutable structure once it has been locked.
-Consequently, only insertion procedures are needed.
-
-Lists
------
-
-Correspond to an ordered collection of values belonging to an elementary type.
-
-At its creation, ...
-
-
-Sets
-----
-
-Correspond to an unordered collection of values belonging to an elementary type.
-
-
-Dictionaries
-------------
-
-Dictionaries allow to associate a `key` to a `value`. Values can be retrieved through their associated key.
-Values must belong to an elementary type and keys must be *OBIStr_t*.

doc/source/data.rst

@@ -1,16 +0,0 @@
-#################
-Data in OBITools3
-#################
-
-The OBITools3 introduce a new way to manage DNA metabarcoding data. 
-They rely on a `Data management System` (DMS) that can be viewed like 
-a simplified database system.
-
-
-.. toctree::
-   :maxdepth: 2
-
-   The data management system <DMS>
-   The data types <types>
-
-

doc/source/elementary.rst

@@ -1,40 +0,0 @@
-================
-Elementary types
-================
-
-They correspond to simple values.
-
-Atomic types
-------------
-
-=========   =========   ============   ==============================
- Type        C type      OBIType        Definition
-=========   =========   ============   ==============================
- integer     int32_t     OBIInt_t       a signed integer value
- float       double      OBIFloat_t     a floating value
- boolean     bool        OBIBool_t      a boolean true/false value
- char        char        OBIChar_t      a character
- index       size_t      OBIIdx_t       an index in a data structure
-=========   =========   ============   ==============================
-
-
-The composite types
--------------------
-
-Character string type
-.....................
-
-================   ======   ========   ==================
-Type               C type   OBIType    Definition
-================   ======   ========   ==================
-Character string   ?        OBIStr_t   a character string
-================   ======   ========   ==================
-
-The taxid type
-..............
-
-====================   ======   ==========   ======================
-Type                   C type   OBIType      Definition
-====================   ======   ==========   ======================
-Taxonomic identifier   size_t   OBITaxid_t   a taxonomic identifier
-====================   ======   ==========   ======================

doc/source/guidelines.rst

@@ -1,131 +0,0 @@
-######################
-Programming guidelines
-######################
-
-
-***************
-Version control
-***************
-
-Version control is managed with `Git <http://git-scm.com/>`_. 
-Issue tracking and repository management are done using `GitLab <https://about.gitlab.com/>`_ 
-at http://git.metabarcoding.org/.
-
-
-Branching strategy
-==================
-
-
-Master branch
--------------
-
-The master branch should only contain functional scripts. 
-
-
-Topic branches
---------------
-
-Topic branches should correspond to development branches revolving around a topic corresponding 
-to the branch's name.
-
-
-Release branches
-----------------
-
-Release branches should start with duplicates of tags and be used to patch them. 
-
-
-Tags
-----
-
-Tags should never be committed to.
-
-
-Rebasing
---------
-
-Rebasing should be avoided on the distant server.
-
-
-Merging
--------
-
-Merging should never overwrite on a release branch or on a tag.
-
-
-Branching strategy diagram
---------------------------
-
-.. image:: ./images/version_control.png
-
-
-Issue tracking
-==============
-
-Issue tracking is done using `GitLab <https://about.gitlab.com/>`_ at http://git.metabarcoding.org/.
-Tickets should always be labeled with the branches for which they are relevant.
-
-
-*************
-Documentation
-*************
-
-C functions are documented in the header files for public functions, and in the source file for private functions.
-
-
-**************
-OBITools3 wiki
-**************
-
-The OBITools3 wiki is managed with GitLab.
-
-
-*********************
-Programming languages
-*********************
-
-C99 :
-	* All the low-level input/output functions (e.g. all the `OBIDMS <formats.html#the-obitools3-data-management-system-obidms>`_ functions)
-	* Computing-intensive code (e.g. alignment or pattern matching)
-	
-`Cython <cython.org>`_ :
-	* Object layer
-	* OBITools3 library
-
-`Python 3.5 <https://www.python.org/>`_ :
-	* Top layer code (scripts)	
-	
-For the documentation, `Sphinx <http://sphinx-doc.org/>`_ should be used for both the original 
-documentation and for the generation of documentation from the python code. `Doxygen <http://www.stack.nl/~dimitri/doxygen/>`_ 
-should be used for the generation of documentation from the C code, which should be then integrated 
-in the Sphinx documentation using `Breathe <https://breathe.readthedocs.org/en/latest/>`_.
-
-
-******************
-Naming conventions
-******************
-
-Struct, Enum:                                   ``Title_case``
-
-Enum members, macros, constants:                ``ALL_CAPS``
-
-Functions, local variables:                     ``lower_case``
-
-Public functions:                               ``obi_lower_case``
-
-Functions that shouldn't be called directly:    ``_lower_case`` (``_`` prefix)
-
-Global variables:                                ``g_lower_case`` (``g_`` prefix)
-
-Pointers:                                        ``pointer_ptr`` (``_ptr`` suffix)
-
-.. note::
-    Underscores are used to delimit 'words'.
-
-
-*****************
-Programming rules
-*****************
-
-    * 
-

doc/source/index.rst

@@ -1,21 +0,0 @@
-.. OBITools3 documentation master file, created by
-   sphinx-quickstart on Mon May  4 14:36:57 2015.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
-
-OBITools3 documentation
-==========================
-
-.. toctree::
-   :maxdepth: 2
-
-   Programming guidelines <guidelines>
-   Data structures <data>
-   Code documentation <code_doc/codedoc>
-
-Indices and tables
-------------------
-
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`

doc/source/specialvalues.rst

@@ -1,55 +0,0 @@
-==============
-Special values
-==============
-
-
-NA values
-=========
-
-All OBITypes have an associated NA (Not Available) value. 
-NA values are implemented by specifying an explicit NA value for each type, 
-corresponding to the R standards as much as possible:
-
-* For the type ``OBIInt_t``, the NA value is ``INT_MIN``.
-
-* For the type ``OBIBool_t``, the NA value is ``2``.
-
-* For the type ``OBIIdx_t`` and ``OBITaxid_t``, the NA value is ``SIZE_MAX``.
-
-* For the type ``OBIChar_t``: the NA value is ``\0``.
-
-* For the type ``OBIFloat_t``::
-
-    typedef union
-    {
-        double value;
-        unsigned int word[2];
-    } ieee_double;
-    
-    static double NA_value(void)
-    {
-        volatile ieee_double x;
-        x.word[hw] = 0x7ff00000;
-        x.word[lw] = 1954;
-        return x.value;
-    }
-
-
-Minimum and maximum values for ``OBIInt_t``
-===========================================
-
-* Maximum value : ``INT_MAX``
-* Minimum value : ``INT_MIN(-1?)``
-
-
-Infinity values for the type ``OBIFloat_t``
-===========================================
-
-* Positive infinity : ``INFINITY`` (should be defined in ``<math.h>``)
-* Negative infinity : ``-INFINITY``
-
-
-NaN value for the type ``OBIFloat_t``
-=====================================
-
-* NaN (Not a Number) value : ``NAN`` (should be defined in ``<math.h>`` but probably needs to be tested)

doc/source/types.rst

@@ -1,21 +0,0 @@
-********
-OBITypes
-********
-
-    
-.. image:: ./UML/OBITypes_UML.png
-
-:download:`html version of the OBITypes UML file <UML/OBITypes_UML.class.violet.html>`
-
-
-.. image:: ./UML/Obicolumn_classes_UML.png
-
-:download:`html version of the OBIDMS classes UML file <UML/Obicolumn_classes_UML.class.violet.html>`
-
-
-.. toctree::
-   :maxdepth: 2
-
-   The elementary types <elementary>
-   The containers <containers>
-   Special values <specialvalues>

obi_completion_script.bash

@@ -0,0 +1,25 @@
+
+_obi_comp ()
+{
+	local cur prev
+    cur="${COMP_WORDS[COMP_CWORD]}"
+    prev="${COMP_WORDS[COMP_CWORD-1]}"
+	if [ "${#COMP_WORDS[@]}" = "2" ]; then
+		COMPREPLY=($(compgen -W "align alignpairedend annotate build_ref_db clean_dms clean count ecopcr ecotag export grep head history import less ls ngsfilter sort stats tail test uniq" "${COMP_WORDS[1]}"))
+	else
+		if [[ "$cur" == *VIEWS* ]]; then
+			COMPREPLY=($(compgen -o plusdirs -f -X '!*.obiview' -- "${COMP_WORDS[COMP_CWORD]}"))
+		elif  [[ -d $cur.obidms ]]; then
+			COMPREPLY=($(compgen -o plusdirs -f $cur.obidms/VIEWS/ -- "${COMP_WORDS[COMP_CWORD]}"), $(compgen -o plusdirs -f -X '!*.obidms/' -- "${COMP_WORDS[COMP_CWORD]}"))
+		elif [[ "$cur" == *obidms* ]]; then
+			COMPREPLY=($(compgen -o plusdirs -f $cur/VIEWS/ -- "${COMP_WORDS[COMP_CWORD]}"))
+    	else
+			COMPREPLY=($(compgen -o plusdirs -f -X '!*.obidms/' -- "${COMP_WORDS[COMP_CWORD]}"))
+		fi
+		if [[ "$prev" == import ]]; then
+			COMPREPLY+=($(compgen -f -- "${COMP_WORDS[COMP_CWORD]}"))
+		fi
+	fi
+}
+
+complete -o nospace -F _obi_comp obi

python/.gitignore

@@ -0,0 +1,2 @@
+/.DS_Store
+/OBITools3.egg-info/

python/obitools3/.gitignore

@@ -0,0 +1 @@
+/.DS_Store

python/obitools3/apps/arguments.cfiles

@@ -1,110 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h
-
-
-
-
-
-
-

python/obitools3/apps/arguments.pyx

@@ -33,10 +33,6 @@ cpdef buildArgumentParser(str configname,
                                        default=None, 
                         help='Create a logfile')
 
-    parser.add_argument('--no-progress', dest='%s:progress' % configname, 
-                                       action='store_false', 
-                                       default=None, 
-                        help='Do not print the progress bar during analyzes')
 
     subparsers = parser.add_subparsers(title='subcommands',
                                        description='valid subcommands',

python/obitools3/apps/command.cfiles

@@ -1,110 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h
-
-
-
-
-
-
-

python/obitools3/apps/config.cfiles

@@ -1,110 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h
-
-
-
-
-
-
-

python/obitools3/apps/config.pyx

@@ -13,7 +13,7 @@ from .logging   cimport getLogger
 from .arguments cimport buildArgumentParser
 
 from ..version import version
-from _curses import version
+
 
 cdef dict __default_config__ = {}
 

python/obitools3/apps/logging.cfiles

@@ -1,110 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h
-
-
-
-
-
-
-

python/obitools3/apps/optiongroups/__init__.py

@@ -1,3 +1,9 @@
+import codecs
+
+def unescaped_str(arg_str):
+    return arg_str.encode('latin-1', 'backslashreplace').decode('unicode-escape')
+
+
 def __addInputOption(optionManager):
     
     optionManager.add_argument(
@@ -39,6 +45,30 @@ def __addImportInputOption(optionManager):
                      const=b'fastq',
                      help="Input file is in fastq format")
 
+    group.add_argument('--silva-input',
+                     action="store_const", dest="obi:inputformat",
+                     default=None,
+                     const=b'silva',
+                     help="Input file is in SILVA fasta format. If NCBI taxonomy provided with --taxonomy, taxid and scientific name will be added for each sequence.")
+
+    group.add_argument('--rdp-input',
+                     action="store_const", dest="obi:inputformat",
+                     default=None,
+                     const=b'rdp',
+                     help="Input file is in RDP training set fasta format. If NCBI taxonomy provided with --taxonomy, taxid and scientific name will be added for each sequence.")
+
+    group.add_argument('--unite-input',
+                     action="store_const", dest="obi:inputformat",
+                     default=None,
+                     const=b'unite',
+                     help="Input file is in UNITE fasta format. If NCBI taxonomy provided with --taxonomy, taxid and scientific name will be added for each sequence.")
+
+    group.add_argument('--sintax-input',
+                     action="store_const", dest="obi:inputformat",
+                     default=None,
+                     const=b'sintax',
+                     help="Input file is in SINTAX fasta format. If NCBI taxonomy provided with --taxonomy, taxid and scientific name will be added for each sequence.")
+
     group.add_argument('--embl-input',
                      action="store_const", dest="obi:inputformat",
                      default=None,
@@ -55,7 +85,7 @@ def __addImportInputOption(optionManager):
                      action="store_const", dest="obi:inputformat",
                      default=None,
                      const=b'ngsfilter',
-                     help="Input file is an ngsfilter file")
+                     help="Input file is an ngsfilter file. If not using tags, use ':' or 'None:None' or '-:-' or any combination")
 
     group.add_argument('--ecopcr-result-input',
                      action="store_const", dest="obi:inputformat",
@@ -115,20 +145,26 @@ def __addImportInputOption(optionManager):
                      type=str,
                      help="String associated with Non Available (NA) values in the input")
 
-  
-def __addTabularInputOption(optionManager):
-    group = optionManager.add_argument_group("Input format options for tabular files")
 
-    group.add_argument('--header',
-                     action="store_true", dest="obi:header",
-                     default=False,
-                     help="First line of tabular file contains column names")
+def __addTabularOption(optionManager):
+    group = optionManager.add_argument_group("Input and output format options for tabular files")
+
+    group.add_argument('--no-header',
+                     action="store_false", dest="obi:header",
+                     default=True,
+                     help="Don't print the header (first line with column names")
 
     group.add_argument('--sep',
                      action="store", dest="obi:sep",
-                     default=None,
-                     type=str,
+                     default="\t",
+                     type=unescaped_str,
                      help="Column separator")
+
+
+def __addTabularInputOption(optionManager):
+    group = optionManager.add_argument_group("Input format options for tabular files")
+    
+    __addTabularOption(optionManager)
     
     group.add_argument('--dec',
                      action="store", dest="obi:dec",
@@ -153,6 +189,16 @@ def __addTabularInputOption(optionManager):
                      help="Lines starting by this char are considered as comment")
 
 
+def __addTabularOutputOption(optionManager):
+    group = optionManager.add_argument_group("Output format options for tabular files")
+    
+    __addTabularOption(optionManager)
+
+    group.add_argument('--na-int-stay-na',
+                     action="store_false", dest="obi:na_int_to_0",
+                     help="NA (Non available) integer values should be exported as NA in tabular output (default: they are converted to 0 for tabular output).") # TODO
+
+
 def __addTaxdumpInputOption(optionManager):  # TODO maybe not the best way to do it
     group = optionManager.add_argument_group("Input format options for taxdump")
 
@@ -186,6 +232,10 @@ def addTabularInputOption(optionManager):
     __addTabularInputOption(optionManager)
 
 
+def addTabularOutputOption(optionManager):
+    __addTabularOutputOption(optionManager)
+
+
 def addTaxonomyOption(optionManager):
     __addTaxonomyOption(optionManager)
 
@@ -198,6 +248,7 @@ def addAllInputOption(optionManager):
     __addInputOption(optionManager)
     __addImportInputOption(optionManager)
     __addTabularInputOption(optionManager)
+    __addTabularOutputOption(optionManager)
     __addTaxonomyOption(optionManager)
     __addTaxdumpInputOption(optionManager)    
 
@@ -205,7 +256,7 @@ def addAllInputOption(optionManager):
 def __addOutputOption(optionManager):
     
     optionManager.add_argument(
-                    dest='obi:outputURI',  
+                    dest='obi:outputURI',
                     metavar='OUTPUT', 
                     help='Data destination URI')
 
@@ -216,12 +267,15 @@ def __addDMSOutputOption(optionManager):
     group.add_argument('--no-create-dms',
                  action="store_true", dest="obi:nocreatedms",
                  default=False,
-                 help="Don't create an output DMS is it is not existing")
+                 help="Don't create an output DMS if it does not already exist")
 
+
+def __addEltLimitOption(optionManager):
+    group = optionManager.add_argument_group("Option to limit the number of elements per line in columns")
     group.add_argument('--max-elts',
                  action="store", dest="obi:maxelts",
                  metavar='<N>',
-                 default=1000,
+                 default=1000000,
                  type=int,
                  help="Maximum number of elements per line in a column "
                       "(e.g. the number of different keys in a dictionary-type "
@@ -232,6 +286,11 @@ def __addDMSOutputOption(optionManager):
 def __addExportOutputOption(optionManager):
     group = optionManager.add_argument_group("Output format options for exported files")
 
+    group.add_argument('-o',
+                    dest='obi:outputURI',
+                    metavar='OUTPUT', 
+                    help='Data destination URI')
+
     group.add_argument('--fasta-output',
                      action="store_const", dest="obi:outputformat",
                      default=None,
@@ -244,6 +303,41 @@ def __addExportOutputOption(optionManager):
                      const=b'fastq',
                      help="Output file is in fastq format")
 
+    group.add_argument('--tab-output',
+                     action="store_const", dest="obi:outputformat",
+                     default=None,
+                     const=b'tabular',
+                     help="Output file is in tabular format")
+
+    group.add_argument('--metabaR-output',
+                     action="store_const", dest="obi:outputformat",
+                     default=None,
+                     const=b'metabaR',
+                     help="Export the files needed by the obifiles_to_metabarlist function of the metabaR package")
+
+    group.add_argument('--metabaR-prefix',
+                     action="store", dest="obi:metabarprefix",
+                     type=str,
+                     help="Prefix for the files when using --metabaR-output option")
+
+    group.add_argument('--metabaR-ngsfilter',
+                     action="store", dest="obi:metabarngsfilter",
+                     type=str,
+                     default=None,
+                     help="URI to the ngsfilter view when using --metabaR-output option (if not provided, it is not exported)")
+
+    group.add_argument('--metabaR-samples',
+                     action="store", dest="obi:metabarsamples",
+                     type=str,
+                     default=None,
+                     help="URI to the sample metadata view when using --metabaR-output option (if not provided, it is built as just a list of the sample names)")
+
+    group.add_argument('--only-keys',
+                       action="append", dest="obi:only_keys",
+                       type=str,
+                       default=[],
+                       help="Only export the given keys (columns).")
+
     group.add_argument('--print-na',
                      action="store_true", dest="obi:printna",
                      default=False,
@@ -256,17 +350,40 @@ def __addExportOutputOption(optionManager):
                      help="String associated with Non Available (NA) values in the output")
 
 
+def __addNoProgressBarOption(optionManager):
+    group = optionManager.add_argument_group("Option to deactivate the display of the progress bar")
+
+    group.add_argument('--no-progress-bar',
+                     action="store_true", dest="obi:noprogressbar",
+                     default=False,
+                     help="Do not display progress bar")
+
+
 def addMinimalOutputOption(optionManager):
     __addOutputOption(optionManager)
     __addDMSOutputOption(optionManager)
 
+
+def addTabularOutputOption(optionManager):
+    __addTabularOption(optionManager)
+
+
 def addExportOutputOption(optionManager):
-    __addOutputOption(optionManager)
     __addExportOutputOption(optionManager)
+    __addTabularOutputOption(optionManager)
 
 
 def addAllOutputOption(optionManager):
     __addOutputOption(optionManager)
     __addDMSOutputOption(optionManager)
     __addExportOutputOption(optionManager)
+    __addTabularOutputOption(optionManager)
 
+
+def addNoProgressBarOption(optionManager):
+    __addNoProgressBarOption(optionManager)
+    
+def addEltLimitOption(optionManager):
+    __addEltLimitOption(optionManager)
+    
+    

python/obitools3/apps/progress.cfiles

@@ -1,110 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h
-
-
-
-
-
-
-

python/obitools3/apps/progress.pyx

@@ -30,12 +30,12 @@ cdef class ProgressBar:
                  off_t maxi,
                  dict  config={},
                  str head="",
-                 double seconde=0.1,
+                 double seconds=5,
                  cut=False):
         
         self.starttime = self.clock()
         self.lasttime  = self.starttime
-        self.tickcount = <clock_t> (seconde * CLOCKS_PER_SEC)
+        self.tickcount = <clock_t> (seconds * CLOCKS_PER_SEC)
         self.freq      = 1
         self.cycle     = 0
         self.arrow     = 0

python/obitools3/apps/temp.cfiles

@@ -1,110 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h
-
-
-
-
-
-
-

python/obitools3/apps/temp.pxd

@@ -6,3 +6,5 @@ Created on 28 juillet 2017
 @author: coissac
 '''
 
+from obitools3.dms.dms cimport DMS
+from obitools3.utils cimport tobytes,tostr

python/obitools3/apps/temp.pyx

@@ -17,9 +17,6 @@ from obitools3.dms.dms import DMS
 from obitools3.apps.config import getConfiguration
 from obitools3.apps.config import logger
 
-from obitools3.dms.dms cimport DMS
-from obitools3.utils cimport tobytes,tostr
-
 cpdef get_temp_dir():
     """
     Returns a temporary directory object specific of this instance of obitools.

python/obitools3/commands/.gitignore

@@ -0,0 +1 @@
+/.DS_Store

python/obitools3/commands/addtaxids.pyx

@@ -0,0 +1,231 @@
+#cython: language_level=3
+
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
+from obitools3.dms import DMS
+from obitools3.dms.view.view cimport View, Line_selection
+from obitools3.uri.decode import open_uri
+from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption, addNoProgressBarOption
+from obitools3.dms.view import RollbackException
+from obitools3.dms.column.column cimport Column
+from functools import reduce
+from obitools3.apps.config import logger
+from obitools3.utils cimport tobytes, str2bytes, tostr
+from io import BufferedWriter
+from obitools3.dms.capi.obiview cimport NUC_SEQUENCE_COLUMN, \
+                                        ID_COLUMN, \
+                                        DEFINITION_COLUMN, \
+                                        QUALITY_COLUMN, \
+                                        COUNT_COLUMN, \
+                                        TAXID_COLUMN
+from obitools3.dms.capi.obitypes cimport OBI_INT
+from obitools3.dms.capi.obitaxonomy cimport MIN_LOCAL_TAXID
+import time
+import math 
+import sys
+
+from cpython.exc cimport PyErr_CheckSignals
+
+ 
+__title__="Annotate sequences with their corresponding NCBI taxid found from the taxon scientific name"
+
+
+ 
+def addOptions(parser):
+    
+    addMinimalInputOption(parser)
+    addTaxonomyOption(parser)
+    addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
+
+    group=parser.add_argument_group('obi addtaxids specific options')
+
+    group.add_argument('-t', '--taxid-tag', 
+                       action="store", 
+                       dest="addtaxids:taxid_tag",
+                       metavar="<TAXID_TAG>",
+                       default=b"TAXID",
+                       help="Name of the tag to store the found taxid "
+                            "(default: 'TAXID').")
+
+    group.add_argument('-n', '--taxon-name-tag', 
+                       action="store", 
+                       dest="addtaxids:taxon_name_tag",
+                       metavar="<SCIENTIFIC_NAME_TAG>",
+                       default=b"SCIENTIFIC_NAME",
+                       help="Name of the tag giving the scientific name of the taxon "
+                            "(default: 'SCIENTIFIC_NAME').")
+
+    group.add_argument('-g', '--try-genus-match',
+                     action="store_true", dest="addtaxids:try_genus_match",
+                     default=False,
+                     help="Try matching the first word of <SCIENTIFIC_NAME_TAG> when can't find corresponding taxid for a taxon. "
+                          "If there is a match it is added in the 'parent_taxid' tag. (Can be used by 'obi taxonomy' to add the taxon under that taxid).")
+
+    group.add_argument('-a', '--restricting-ancestor', 
+                       action="store", 
+                       dest="addtaxids:restricting_ancestor",
+                       metavar="<RESTRICTING_ANCESTOR>",
+                       default=None,
+                       help="Enables to restrict the search of taxids under an ancestor specified by its taxid.")
+
+    group.add_argument('-l', '--log-file', 
+                       action="store", 
+                       dest="addtaxids:log_file",
+                       metavar="<LOG_FILE>",
+                       default='',
+                       help="Path to a log file to write informations about not found taxids.")
+
+
+def run(config):
+     
+    DMS.obi_atexit()
+    
+    logger("info", "obi addtaxids")
+
+    # Open the input
+    input = open_uri(config['obi']['inputURI'])
+    if input is None:
+        raise Exception("Could not read input view")
+    i_dms = input[0]
+    i_view = input[1]
+    i_view_name = input[1].name
+
+    # Open the output: only the DMS, as the output view is going to be created by cloning the input view
+    # (could eventually be done via an open_uri() argument)
+    output = open_uri(config['obi']['outputURI'],
+                      input=False,
+                      dms_only=True)
+    if output is None:
+        raise Exception("Could not create output view")
+    o_dms = output[0]
+    output_0 = output[0]
+    o_view_name = output[1]
+
+    # stdout output: create temporary view
+    if type(output_0)==BufferedWriter:
+        o_dms = i_dms
+        i=0
+        o_view_name = b"temp"
+        while o_view_name in i_dms: # Making sure view name is unique in output DMS
+            o_view_name = o_view_name+b"_"+str2bytes(str(i))
+            i+=1
+        imported_view_name = o_view_name
+
+    # If the input and output DMS are not the same, import the input view in the output DMS before cloning it to modify it
+    # (could be the other way around: clone and modify in the input DMS then import the new view in the output DMS)
+    if i_dms != o_dms:
+        imported_view_name = i_view_name
+        i=0
+        while imported_view_name in o_dms:  # Making sure view name is unique in output DMS
+            imported_view_name = i_view_name+b"_"+str2bytes(str(i))
+            i+=1
+        View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], i_view_name, imported_view_name)
+        i_view = o_dms[imported_view_name]
+
+    # Clone output view from input view
+    o_view = i_view.clone(o_view_name)
+    if o_view is None:
+        raise Exception("Couldn't create output view")
+    i_view.close()
+
+    # Open taxonomy
+    taxo_uri = open_uri(config['obi']['taxoURI'])
+    if taxo_uri is None or taxo_uri[2] == bytes:
+        raise Exception("Couldn't open taxonomy")
+    taxo = taxo_uri[1]
+
+    # Initialize the progress bar
+    if config['obi']['noprogressbar'] == False:
+        pb = ProgressBar(len(o_view), config)
+    else:
+        pb = None
+        
+    try:
+        if config['addtaxids']['log_file']:
+            logfile = open(config['addtaxids']['log_file'], 'w')
+        else:
+            logfile = None
+        if config['addtaxids']['try_genus_match']:
+            try_genus = True
+        else:
+            try_genus = False
+        if 'restricting_ancestor' in config['addtaxids']:
+            res_anc = int(config['addtaxids']['restricting_ancestor'])
+        else:
+            res_anc = None
+        taxid_column_name = config['addtaxids']['taxid_tag']
+        parent_taxid_column_name = "PARENT_TAXID" # TODO macro
+        taxon_name_column_name = config['addtaxids']['taxon_name_tag']
+        taxid_column = Column.new_column(o_view, taxid_column_name, OBI_INT)
+        parent_taxid_column = Column.new_column(o_view, parent_taxid_column_name, OBI_INT)
+        taxon_name_column = o_view[taxon_name_column_name]
+        
+        found_count = 0
+        not_found_count = 0
+        parent_found_count = 0
+        
+        for i in range(len(o_view)):
+            PyErr_CheckSignals()
+            if pb is not None:
+                pb(i)
+            taxon_name = taxon_name_column[i]
+            taxon = taxo.get_taxon_by_name(taxon_name, res_anc)
+            if taxon is not None:
+                taxid_column[i] = taxon.taxid
+                found_count+=1
+            elif try_genus: # try finding genus or other parent taxon from the first word
+                #print(i, o_view[i].id)
+                taxon_name_sp = taxon_name.split(b" ")
+                taxon = taxo.get_taxon_by_name(taxon_name_sp[0], res_anc)
+                if taxon is not None:
+                    parent_taxid_column[i] = taxon.taxid
+                    parent_found_count+=1
+                    if logfile:
+                        print("Found parent taxon for", tostr(taxon_name), file=logfile)
+                else:
+                    not_found_count+=1
+                    if logfile:
+                        print("No taxid found for", tostr(taxon_name), file=logfile)
+            else:
+                not_found_count+=1
+                if logfile:
+                    print("No taxid found for", tostr(taxon_name), file=logfile)
+
+    except Exception, e:
+        raise RollbackException("obi addtaxids error, rollbacking view: "+str(e), o_view)
+
+    if pb is not None:
+        pb(i, force=True)
+        print("", file=sys.stderr)
+    
+    logger("info", "\nTaxids found: "+str(found_count)+"/"+str(len(o_view))+" ("+str(round(found_count*100.0/len(o_view), 2))+"%)")
+    if config['addtaxids']['try_genus_match']:
+        logger("info", "\nParent taxids found: "+str(parent_found_count)+"/"+str(len(o_view))+" ("+str(round(parent_found_count*100.0/len(o_view), 2))+"%)")
+    logger("info", "\nTaxids not found: "+str(not_found_count)+"/"+str(len(o_view))+" ("+str(round(not_found_count*100.0/len(o_view), 2))+"%)") 
+    
+    # Save command config in View and DMS comments
+    command_line = " ".join(sys.argv[1:])
+    input_dms_name=[input[0].name]
+    input_view_name=[i_view_name]
+    if 'taxoURI' in config['obi'] and config['obi']['taxoURI'] is not None:
+        input_dms_name.append(config['obi']['taxoURI'].split("/")[-3])
+        input_view_name.append("taxonomy/"+config['obi']['taxoURI'].split("/")[-1])
+    o_view.write_config(config, "addtaxids", command_line, input_dms_name=input_dms_name, input_view_name=input_view_name)
+    o_dms.record_command_line(command_line)
+
+    #print("\n\nOutput view:\n````````````", file=sys.stderr)
+    #print(repr(o_view), file=sys.stderr)
+
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
+    # If the input and the output DMS are different or if stdout output, delete the temporary imported view used to create the final view
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
+        View.delete_view(o_dms, imported_view_name)
+        o_dms.close(force=True)
+    i_dms.close(force=True)
+
+    logger("info", "Done.")

python/obitools3/commands/align.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/align.pyx

@@ -4,7 +4,7 @@ from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms import DMS
 from obitools3.dms.view.view cimport View
 from obitools3.uri.decode import open_uri
-from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.dms.view import RollbackException
 from obitools3.apps.config import logger
 from obitools3.utils cimport tobytes, str2bytes
@@ -12,17 +12,21 @@ from obitools3.utils cimport tobytes, str2bytes
 from obitools3.dms.capi.obilcsalign cimport obi_lcs_align_one_column, \
                                             obi_lcs_align_two_columns
 
+from io import BufferedWriter
+from cpython.exc cimport PyErr_CheckSignals
+
 import time
 import sys
 
 
-__title__="Aligns one sequence column with itself or two sequence columns"
+__title__="Align one sequence column with itself or two sequence columns"
 
 
 def addOptions(parser):
    
    addMinimalInputOption(parser)
    addMinimalOutputOption(parser)
+   addNoProgressBarOption(parser)
 
    group=parser.add_argument_group('obi align specific options')
 
@@ -154,7 +158,7 @@ def run(config):
     i_view_name = i_uri.split(b"/")[0]
     i_column_name = b""
     i_element_name = b""
-    if len(i_uri.split(b"/")) == 2:
+    if len(i_uri.split(b"/")) >= 2:
         i_column_name = i_uri.split(b"/")[1]
     if len(i_uri.split(b"/")) == 3:
         i_element_name = i_uri.split(b"/")[2]
@@ -177,7 +181,7 @@ def run(config):
         i_dms_name_2 = i_dms_2.name
         i_uri_2 = input_2[1]
         original_i_view_name_2 = i_uri_2.split(b"/")[0]
-        if len(i_uri_2.split(b"/")) == 2:
+        if len(i_uri_2.split(b"/")) >= 2:
             i_column_name_2 = i_uri_2.split(b"/")[1]
         if len(i_uri_2.split(b"/")) == 3:
             i_element_name_2 = i_uri_2.split(b"/")[2]
@@ -201,20 +205,20 @@ def run(config):
     if output is None:
         raise Exception("Could not create output")
     o_dms = output[0]
+    output_0 = output[0]
     o_dms_name = o_dms.name 
     final_o_view_name = output[1]
+    o_view_name = final_o_view_name
 
-    # If the input and output DMS are not the same, align creating a temporary view in the input dms that will be exported to 
+    # If stdout output or the input and output DMS are not the same, align creating a temporary view in the input dms that will be exported to 
     # the right DMS and deleted in the other afterwards.
-    if i_dms != o_dms:
-        temporary_view_name = final_o_view_name
-        i=0
-        while temporary_view_name in i_dms:  # Making sure view name is unique in input DMS
-            temporary_view_name = final_o_view_name+b"_"+str2bytes(str(i))
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
+        if type(output_0)==BufferedWriter:
+            o_dms = i_dms
+        o_view_name = b"temp"
+        while o_view_name in i_dms:   # Making sure view name is unique in input DMS
+            o_view_name = o_view_name+b"_"+str2bytes(str(i))
             i+=1
-        o_view_name = temporary_view_name
-    else:
-        o_view_name = final_o_view_name
         
     # Save command config in View comments
     command_line = " ".join(sys.argv[1:])
@@ -229,7 +233,7 @@ def run(config):
     
     # Call cython alignment function
       # Using default ID columns of the view. TODO discuss adding option
-    align_columns(i_dms_name,  \
+    align_columns(i_dms.name_with_full_path,  \
                   i_view_name,  \
                   o_view_name,  \
                   input_view_2_n   = i_view_name_2,  \
@@ -263,12 +267,19 @@ def run(config):
         View.delete_view(i_dms, i_view_name_2)
         i_dms_2.close()
 
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view = o_dms[o_view_name]
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
     # If the input and the output DMS are different, delete the temporary result view in the input DMS
-    if i_dms != o_dms:
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
         View.delete_view(i_dms, o_view_name)
-        o_dms.close()
+        o_dms.close(force=True)
     
-    i_dms.close()
+    i_dms.close(force=True)
     
     logger("info", "Done.")
   

python/obitools3/commands/alignpairedend.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/alignpairedend.pyx

@@ -2,12 +2,11 @@
 
 from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms import DMS
-from obitools3.dms.view import RollbackException
 from obitools3.dms.view.typed_view.view_NUC_SEQS cimport View_NUC_SEQS
 from obitools3.dms.column.column cimport Column
 from obitools3.dms.capi.obiview cimport QUALITY_COLUMN
 from obitools3.dms.capi.obitypes cimport OBI_QUAL
-from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.uri.decode import open_uri
 from obitools3.apps.config import logger
 from obitools3.libalign._qsassemble import QSolexaReverseAssemble
@@ -15,13 +14,16 @@ from obitools3.libalign._qsrassemble import QSolexaRightReverseAssemble
 from obitools3.libalign._solexapairend import buildConsensus, buildJoinedSequence
 from obitools3.dms.obiseq cimport Nuc_Seq
 from obitools3.libalign.shifted_ali cimport Kmer_similarity, Ali_shifted
-from obitools3.commands.ngsfilter import REVERSE_SEQ_COLUMN_NAME, REVERSE_QUALITY_COLUMN_NAME
+from obitools3.dms.capi.obiview cimport REVERSE_SEQUENCE_COLUMN, REVERSE_QUALITY_COLUMN
+from obitools3.utils cimport str2bytes
 
+from io import BufferedWriter
 import sys
 import os
 
+from cpython.exc cimport PyErr_CheckSignals
 
-__title__="Aligns paired-ended reads"
+__title__="Align paired-ended reads"
 
 
 
@@ -29,6 +31,7 @@ def addOptions(parser):
 
     addMinimalInputOption(parser)
     addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
  
     group = parser.add_argument_group('obi alignpairedend specific options')
 
@@ -39,24 +42,25 @@ def addOptions(parser):
                      type=str,
                      help="URI to the reverse reads if they are in a different view than the forward reads")
 
-    group.add_argument('--score-min',
-                     action="store", dest="alignpairedend:smin",
-                     metavar="#.###",
-                     default=None,
-                     type=float,
-                     help="Minimum score for keeping alignments")
+#     group.add_argument('--score-min',
+#                      action="store", dest="alignpairedend:smin",
+#                      metavar="#.###",
+#                      default=None,
+#                      type=float,
+#                      help="Minimum score for keeping alignments. " 
+#                           "(for kmer alignment) The score is an approximation of the number of nucleotides matching in the overlap of the alignment.")
 
-    group.add_argument('-A', '--true-ali',
-                       action="store_true", dest="alignpairedend:trueali",
-                       default=False,
-                       help="Performs gap free end alignment of sequences instead of using kmers to compute alignments (slower).")
+#     group.add_argument('-A', '--true-ali',
+#                        action="store_true", dest="alignpairedend:trueali",
+#                        default=False,
+#                        help="Performs gap free end alignment of sequences instead of using kmers to compute alignments (slower).")
 
     group.add_argument('-k', '--kmer-size',
                        action="store", dest="alignpairedend:kmersize",
                        metavar="#",
                        default=3,
                        type=int,
-                       help="K-mer size for kmer comparisons, between 1 and 4 (not when using -A option; default: 3)")
+                       help="K-mer size for kmer comparisons, between 1 and 4 (default: 3)")
 
 
 la = QSolexaReverseAssemble()
@@ -96,12 +100,13 @@ def alignmentIterator(entries, aligner):
         entries_len = len(entries)
     
     for i in range(entries_len):
+                
         if two_views:
             seqF = forward[i]
             seqR = reverse[i]
         else:
             seqF = Nuc_Seq.new_from_stored(entries[i])
-            seqR = Nuc_Seq(seqF.id, seqF[REVERSE_SEQ_COLUMN_NAME], quality=seqF[REVERSE_QUALITY_COLUMN_NAME])
+            seqR = Nuc_Seq(seqF.id, seqF[REVERSE_SEQUENCE_COLUMN], quality=seqF[REVERSE_QUALITY_COLUMN])
             seqR.index = i
         
         ali = aligner(seqF, seqR)
@@ -169,81 +174,120 @@ def run(config):
     if output is None:
         raise Exception("Could not create output view")
 
-    view = output[1]
+    output_0 = output[0]
+    o_dms = output[0]
     
-    Column.new_column(view, QUALITY_COLUMN, OBI_QUAL)   #TODO output URI quality option?
-
-    if 'smin' in config['alignpairedend']:
-        smin = config['alignpairedend']['smin']
+    # stdout output: create temporary view
+    if type(output_0)==BufferedWriter:
+        i_dms = forward.dms   # using any dms
+        o_dms = i_dms
+        i=0
+        o_view_name = b"temp"
+        while o_view_name in i_dms:   # Making sure view name is unique in input DMS
+            o_view_name = o_view_name+b"_"+str2bytes(str(i))
+            i+=1
+        o_view = View_NUC_SEQS.new(o_dms, o_view_name, quality=True)
     else:
-        smin = 0
+        o_view = output[1]
+        Column.new_column(o_view, QUALITY_COLUMN, OBI_QUAL)
 
     # Initialize the progress bar
-    pb = ProgressBar(entries_len, config, seconde=5)
-    
-    if config['alignpairedend']['trueali']:
-        kmer_ali = False
-        aligner = buildAlignment
-    else :
-        kmer_ali = True
-        if type(entries) == list:
-            forward = entries[0]
-            reverse = entries[1]
-            aligner = Kmer_similarity(forward, view2=reverse, kmer_size=config['alignpairedend']['kmersize'])
+    if config['obi']['noprogressbar'] == False:
+        pb = ProgressBar(entries_len, config)
+    else:
+        pb = None
+         
+    #if config['alignpairedend']['trueali']:
+    #    kmer_ali = False
+    #    aligner = buildAlignment
+    #else :
+    kmer_ali = True
+    if type(entries) == list:
+        forward = entries[0]
+        reverse = entries[1]
+        if len(forward) == 0 or len(reverse) == 0:
+            aligner = None
         else:
-            aligner = Kmer_similarity(entries, column2=entries[REVERSE_SEQ_COLUMN_NAME], qual_column2=entries[REVERSE_QUALITY_COLUMN_NAME], kmer_size=config['alignpairedend']['kmersize'])
+            aligner = Kmer_similarity(forward, \
+                                      view2=reverse, \
+                                      kmer_size=config['alignpairedend']['kmersize'], \
+                                      reversed_column=None)
+    else:
+        if len(entries) == 0:
+            aligner = None
+        else:
+            aligner = Kmer_similarity(entries, \
+                                      column2=entries[REVERSE_SEQUENCE_COLUMN], \
+                                      qual_column2=entries[REVERSE_QUALITY_COLUMN], \
+                                      kmer_size=config['alignpairedend']['kmersize'], \
+                                      reversed_column=entries[b'reversed'])  # column created by the ngsfilter tool
         
     ba = alignmentIterator(entries, aligner)
-
+    
     i = 0
     for ali in ba:
         
-        pb(i)
+        if pb is not None:
+            pb(i)
         
-        consensus = view[i]
+        PyErr_CheckSignals()
+
+        consensus = o_view[i]
         
+        if two_views:
+            consensus[b"R1_parent"] = forward[i].id
+            consensus[b"R2_parent"] = reverse[i].id
+               
         if not two_views:
             seqF = entries[i]
         else:
             seqF = forward[i]
-                
-        if smin > 0:
-            if (ali.score > smin) :
-                buildConsensus(ali, consensus, seqF)
-            else:
-                if not two_views:
-                    seqR = Nuc_Seq(seqF.id, seqF[REVERSE_SEQ_COLUMN_NAME], quality = seqF[REVERSE_QUALITY_COLUMN_NAME])
-                else:
-                    seqR = reverse[i]
-                buildJoinedSequence(ali, seqR, consensus, forward=seqF)
-                
-            consensus[b"smin"] = smin
-        else:
+                        
+        if ali.overlap_len > 0 :
             buildConsensus(ali, consensus, seqF)
-                
+        else:
+            if not two_views:
+                seqR = Nuc_Seq(seqF.id, seqF[REVERSE_SEQUENCE_COLUMN], quality = seqF[REVERSE_QUALITY_COLUMN])
+            else:
+                seqR = reverse[i]
+            buildJoinedSequence(ali, seqR, consensus, forward=seqF)
+                   
         if kmer_ali :
             ali.free()
     
         i+=1
 
-    pb(i, force=True)
-    print("", file=sys.stderr)
+    if pb is not None:
+        pb(i, force=True)
+        print("", file=sys.stderr)
 
-    if kmer_ali :
+    if kmer_ali and aligner is not None:
         aligner.free()
 
     # Save command config in View and DMS comments
     command_line = " ".join(sys.argv[1:])
-    view.write_config(config, "alignpairedend", command_line, input_dms_name=input_dms_name, input_view_name=input_view_name)
-    output[0].record_command_line(command_line)
+    o_view.write_config(config, "alignpairedend", command_line, input_dms_name=input_dms_name, input_view_name=input_view_name)
+    o_dms.record_command_line(command_line)
 
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(view), file=sys.stderr)
-    
-    input[0].close()
+
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
+    # If stdout output, delete the temporary imported view used to create the final file
+    if type(output_0)==BufferedWriter:
+        View_NUC_SEQS.delete_view(o_dms, o_view_name)
+        output_0.close()
+
+    # Close all DMS
+    input[0].close(force=True)
     if two_views:
-        rinput[0].close()
-    output[0].close()
+        rinput[0].close(force=True)
+    o_dms.close(force=True)
 
     logger("info", "Done.")
     

python/obitools3/commands/annotate.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/annotate.pyx

@@ -4,21 +4,27 @@ from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms import DMS
 from obitools3.dms.view.view cimport View, Line_selection
 from obitools3.uri.decode import open_uri
-from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.dms.view import RollbackException
 from functools import reduce
 from obitools3.apps.config import logger
 from obitools3.utils cimport tobytes, str2bytes
+from io import BufferedWriter
 from obitools3.dms.capi.obiview cimport NUC_SEQUENCE_COLUMN, \
                                         ID_COLUMN, \
                                         DEFINITION_COLUMN, \
                                         QUALITY_COLUMN, \
-                                        COUNT_COLUMN
+                                        COUNT_COLUMN, \
+                                        TAXID_COLUMN
+from obitools3.dms.capi.obitypes cimport OBI_STR
+from obitools3.dms.column.column cimport Column
 
 import time
 import math 
 import sys
 
+from cpython.exc cimport PyErr_CheckSignals
+
  
 __title__="Annotate views with new tags and edit existing annotations"
 
@@ -31,6 +37,7 @@ def addOptions(parser):
     addMinimalInputOption(parser)
     addTaxonomyOption(parser)
     addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
 
     group=parser.add_argument_group('obi annotate specific options')
  
@@ -173,8 +180,8 @@ def sequenceTaggerGenerator(config, taxo=None):
             counter[0]+=1
                             
         for rank in annoteRank:
-            if 'taxid' in seq:
-                taxid = seq['taxid']
+            if TAXID_COLUMN in seq:
+                taxid = seq[TAXID_COLUMN]
                 if taxid is not None:
                     rtaxid = taxo.get_taxon_at_rank(taxid, rank)
                     if rtaxid is not None:
@@ -182,64 +189,58 @@ def sequenceTaggerGenerator(config, taxo=None):
                     else:
                         scn=None
                     seq[rank]=rtaxid
+                    if "%s_name"%rank not in seq.view:
+                        Column.new_column(seq.view, "%s_name"%rank, OBI_STR)
                     seq["%s_name"%rank]=scn
              
         if add_rank:
             seq['seq_rank']=counter[0]
     
         for i,v in toSet:
-            #try:
-            if taxo is not None:
-                environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
-            else:
-                environ = {'sequence':seq, 'counter':counter[0], 'math':math}
-            val = eval(v, environ, seq)
-            #except Exception,e:       # TODO discuss usefulness of this
-            #    if options.onlyValid:
-            #        raise e
-            #    val = v
+            try:
+                if taxo is not None:
+                    environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
+                else:
+                    environ = {'sequence':seq, 'counter':counter[0], 'math':math}
+                val = eval(v, environ, seq)
+            except Exception:       # set string if not a valid expression
+                val = v
             seq[i]=val
 
         if length:
             seq['seq_length']=len(seq)
 
         if newId is not None:
-#            try:
-            if taxo is not None:
-                environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
-            else:
-                environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
-            val = eval(newId, environ, seq)
-#            except Exception,e:
-#                if options.onlyValid:
-#                    raise e
-#                val = newId
+            try:
+                if taxo is not None:
+                    environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
+                else:
+                    environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
+                val = eval(newId, environ, seq)
+            except Exception:      # set string if not a valid expression
+                val = newId
             seq.id=val
 
         if newDef is not None:
-#            try:
-            if taxo is not None:
-                environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
-            else:
-                environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
-            val = eval(newDef, environ, seq)
-#            except Exception,e:
-#                if options.onlyValid:
-#                    raise e
-#                val = newDef
+            try:
+                if taxo is not None:
+                    environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
+                else:
+                    environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
+                val = eval(newDef, environ, seq)
+            except Exception:    # set string if not a valid expression
+                val = newDef
             seq.definition=val
-#             
+             
         if newSeq is not None:
-#            try:
-            if taxo is not None:
-                environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
-            else:
-                environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
-            val = eval(newSeq, environ, seq)
-#            except Exception,e:
-#                if options.onlyValid:
-#                    raise e
-#                val = newSeq
+            try:
+                if taxo is not None:
+                    environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
+                else:
+                    environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
+                val = eval(newSeq, environ, seq)
+            except Exception:    # set string if not a valid expression
+                val = newSeq
             seq.seq=val
             if 'seq_length' in seq:
                 seq['seq_length']=len(seq)
@@ -249,15 +250,14 @@ def sequenceTaggerGenerator(config, taxo=None):
                 seq.view.delete_column(QUALITY_COLUMN)
                     
         if run is not None:
-#            try:
-            if taxo is not None:
-                environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
-            else:
-                environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
-            eval(run, environ, seq)
-#            except Exception,e:
-#                if options.onlyValid:
-#                    raise e
+            try:
+                if taxo is not None:
+                    environ = {'taxonomy' : taxo, 'sequence':seq, 'counter':counter[0], 'math':math}
+                else:
+                    environ = {'sequence':seq, 'counter':counter[0], 'math':math}     
+                eval(run, environ, seq)
+            except Exception,e:
+                raise e
 
     return sequenceTagger
 
@@ -284,8 +284,19 @@ def run(config):
     if output is None:
         raise Exception("Could not create output view")
     o_dms = output[0]
+    output_0 = output[0]
     o_view_name = output[1]
-    
+
+    # stdout output: create temporary view
+    if type(output_0)==BufferedWriter:
+        o_dms = i_dms
+        i=0
+        o_view_name = b"temp"
+        while o_view_name in i_dms: # Making sure view name is unique in output DMS
+            o_view_name = o_view_name+b"_"+str2bytes(str(i))
+            i+=1
+        imported_view_name = o_view_name
+
     # If the input and output DMS are not the same, import the input view in the output DMS before cloning it to modify it
     # (could be the other way around: clone and modify in the input DMS then import the new view in the output DMS)
     if i_dms != o_dms:
@@ -296,7 +307,7 @@ def run(config):
             i+=1
         View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], i_view_name, imported_view_name)
         i_view = o_dms[imported_view_name]
-        
+
     # Clone output view from input view
     o_view = i_view.clone(o_view_name)
     if o_view is None:
@@ -306,14 +317,17 @@ def run(config):
     # Open taxonomy if there is one
     if 'taxoURI' in config['obi'] and config['obi']['taxoURI'] is not None:
         taxo_uri = open_uri(config['obi']['taxoURI'])
-        if taxo_uri is None:
+        if taxo_uri is None or taxo_uri[2] == bytes:
             raise Exception("Couldn't open taxonomy")
         taxo = taxo_uri[1]
     else :
         taxo = None
 
     # Initialize the progress bar
-    pb = ProgressBar(len(o_view), config, seconde=5)
+    if config['obi']['noprogressbar'] == False:
+        pb = ProgressBar(len(o_view), config)
+    else:
+        pb = None
 
     try:
 
@@ -351,14 +365,17 @@ def run(config):
         # Editions at line level
         sequenceTagger = sequenceTaggerGenerator(config, taxo=taxo)
         for i in range(len(o_view)):
-            pb(i)
+            PyErr_CheckSignals()
+            if pb is not None:
+                pb(i)
             sequenceTagger(o_view[i])
 
     except Exception, e:
         raise RollbackException("obi annotate error, rollbacking view: "+str(e), o_view)
 
-    pb(i, force=True)
-    print("", file=sys.stderr)
+    if pb is not None:
+        pb(i, force=True)
+        print("", file=sys.stderr)
     
     # Save command config in View and DMS comments
     command_line = " ".join(sys.argv[1:])
@@ -368,15 +385,21 @@ def run(config):
         input_dms_name.append(config['obi']['taxoURI'].split("/")[-3])
         input_view_name.append("taxonomy/"+config['obi']['taxoURI'].split("/")[-1])
     o_view.write_config(config, "annotate", command_line, input_dms_name=input_dms_name, input_view_name=input_view_name)
-    output[0].record_command_line(command_line)
+    o_dms.record_command_line(command_line)
 
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(o_view), file=sys.stderr)
 
-    # If the input and the output DMS are different, delete the temporary imported view used to create the final view
-    if i_dms != o_dms:
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
+    # If the input and the output DMS are different or if stdout output, delete the temporary imported view used to create the final view
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
         View.delete_view(o_dms, imported_view_name)
-        o_dms.close()
-    i_dms.close()
+        o_dms.close(force=True)
+    i_dms.close(force=True)
 
     logger("info", "Done.")

python/obitools3/commands/build_ref_db.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/build_ref_db.pyx

@@ -4,17 +4,19 @@ from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms.dms cimport DMS
 from obitools3.dms.view import RollbackException
 from obitools3.dms.capi.build_reference_db cimport build_reference_db
-from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.uri.decode import open_uri
 from obitools3.apps.config import logger
 from obitools3.utils cimport tobytes, str2bytes
 from obitools3.dms.view.view cimport View
 from obitools3.dms.view.typed_view.view_NUC_SEQS cimport View_NUC_SEQS
 
+from io import BufferedWriter
 import sys
+from cpython.exc cimport PyErr_CheckSignals
 
 
-__title__="Tag a set of sequences for PCR and sequencing errors identification"
+__title__="Build a reference database for ecotag"
 
 
 def addOptions(parser):
@@ -22,16 +24,16 @@ def addOptions(parser):
     addMinimalInputOption(parser)
     addTaxonomyOption(parser)
     addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
  
     group = parser.add_argument_group('obi build_ref_db specific options')
 
     group.add_argument('--threshold','-t',
                       action="store", dest="build_ref_db:threshold",
                       metavar='<THRESHOLD>',
-                      default=0.0,
+                      default=0.99,
                       type=float,
-                      help="Score threshold as a normalized identity, e.g. 0.95 for an identity of 95%%. Default: 0.00"
-                           " (no threshold).")
+                      help="Score threshold as a normalized identity, e.g. 0.95 for an identity of 95%%. Default: 0.99.")
 
 
 def run(config):
@@ -56,17 +58,20 @@ def run(config):
     if output is None:
         raise Exception("Could not create output")
     o_dms = output[0]
+    output_0 = output[0]
     final_o_view_name = output[1]
     
-    # If the input and output DMS are not the same, build the database creating a temporary view that will be exported to 
+    # If stdout output or the input and output DMS are not the same, build the database creating a temporary view that will be exported to 
     # the right DMS and deleted in the other afterwards.
-    if i_dms != o_dms:
-        temporary_view_name = final_o_view_name
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
+        temporary_view_name = b"temp"
         i=0
         while temporary_view_name in i_dms:  # Making sure view name is unique in input DMS
-            temporary_view_name = final_o_view_name+b"_"+str2bytes(str(i))
+            temporary_view_name = temporary_view_name+b"_"+str2bytes(str(i))
             i+=1
         o_view_name = temporary_view_name
+        if type(output_0)==BufferedWriter:
+            o_dms = i_dms
     else:
         o_view_name = final_o_view_name
 
@@ -80,26 +85,33 @@ def run(config):
     input_dms_name.append(config['obi']['taxoURI'].split("/")[-3])
     input_view_name.append("taxonomy/"+config['obi']['taxoURI'].split("/")[-1])
     comments = View.print_config(config, "build_ref_db", command_line, input_dms_name=input_dms_name, input_view_name=input_view_name)
-
-    if build_reference_db(tobytes(i_dms_name), tobytes(i_view_name), tobytes(taxonomy_name), tobytes(o_view_name), comments, config['build_ref_db']['threshold']) < 0:
+    
+    if build_reference_db(i_dms.name_with_full_path, tobytes(i_view_name), tobytes(taxonomy_name), tobytes(o_view_name), comments, config['build_ref_db']['threshold']) < 0:
         raise Exception("Error building a reference database")
     
     # If the input and output DMS are not the same, export result view to output DMS
     if i_dms != o_dms:
         View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], o_view_name, final_o_view_name)
-    
+
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view = o_dms[o_view_name]
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
     # Save command config in DMS comments
     o_dms.record_command_line(command_line)
 
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(o_dms[final_o_view_name]), file=sys.stderr)
 
-    # If the input and the output DMS are different, delete the temporary result view in the input DMS
-    if i_dms != o_dms:
+    # If the input and the output DMS are different or if stdout output, delete the temporary imported view used to create the final view
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
         View.delete_view(i_dms, o_view_name)
-        o_dms.close()
+        o_dms.close(force=True)
     
-    i_dms.close()
+    i_dms.close(force=True)
 
     logger("info", "Done.")
     

python/obitools3/commands/cat.pyx

@@ -0,0 +1,168 @@
+#cython: language_level=3
+
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
+from obitools3.dms import DMS
+from obitools3.dms.view.view cimport View
+from obitools3.uri.decode import open_uri
+from obitools3.apps.optiongroups import addMinimalOutputOption, addNoProgressBarOption
+from obitools3.dms.view import RollbackException
+from obitools3.apps.config import logger
+from obitools3.utils cimport str2bytes
+from obitools3.dms.view.typed_view.view_NUC_SEQS cimport View_NUC_SEQS
+from obitools3.dms.view.view cimport View
+from obitools3.dms.capi.obiview cimport NUC_SEQUENCE_COLUMN, REVERSE_SEQUENCE_COLUMN, \
+                                        QUALITY_COLUMN, REVERSE_QUALITY_COLUMN
+from obitools3.dms.capi.obitypes cimport OBI_SEQ, OBI_QUAL
+from obitools3.dms.column.column cimport Column
+
+from io import BufferedWriter
+import time
+import sys
+ 
+from cpython.exc cimport PyErr_CheckSignals
+
+
+__title__="Concatenate views"
+
+ 
+def addOptions(parser):
+    
+    addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
+
+    group=parser.add_argument_group('obi cat specific options')
+
+    group.add_argument("-c",
+                       action="append", dest="cat:views_to_cat",
+                       metavar="<VIEW_NAME>",
+                       default=[],
+                       type=str,
+                       help="URI of a view to concatenate. (e.g. 'my_dms/my_view'). "
+                            "Several -c options can be used on the same "
+                            "command line.")
+
+     
+def run(config):
+     
+    DMS.obi_atexit()
+    
+    logger("info", "obi cat")
+
+    # Check the views to concatenate
+    idms_list = []
+    iview_list = []
+    total_len = 0
+    remove_qual = False
+    remove_rev_qual = False
+    v_type = View_NUC_SEQS
+    for v_uri in config["cat"]["views_to_cat"]:
+        input = open_uri(v_uri)
+        if input is None:
+            raise Exception("Could not read input view")
+        i_dms = input[0]
+        i_view = input[1]
+        if input[2] != View_NUC_SEQS:  # Check view type (output view is nuc_seqs view if all input view are nuc_seqs view)
+            v_type = View
+        if QUALITY_COLUMN not in i_view: # Check if keep quality column in output view (if all input views have it)
+            remove_qual = True
+        if REVERSE_QUALITY_COLUMN not in i_view: # same as above for reverse quality
+            remove_rev_qual = True
+        total_len += len(i_view)
+        idms_list.append(i_dms)
+        iview_list.append(i_view.name)
+        i_view.close()
+
+    # Open the output: only the DMS
+    output = open_uri(config['obi']['outputURI'],
+                      input=False, 
+                      newviewtype=v_type)
+    if output is None:
+        raise Exception("Could not create output view")
+    o_dms = output[0]
+    output_0 = output[0]
+    o_view = output[1]
+    
+    # stdout output
+    if type(output_0)==BufferedWriter: 
+        o_dms = i_dms
+    
+    # Initialize quality columns and their associated sequence columns if needed
+    if type(output_0) != BufferedWriter: 
+        if not remove_qual:
+            if NUC_SEQUENCE_COLUMN not in o_view:
+                Column.new_column(o_view, NUC_SEQUENCE_COLUMN, OBI_SEQ)
+            Column.new_column(o_view, QUALITY_COLUMN, OBI_QUAL, associated_column_name=NUC_SEQUENCE_COLUMN, associated_column_version=o_view[NUC_SEQUENCE_COLUMN].version)    
+        if not remove_rev_qual:
+            Column.new_column(o_view, REVERSE_SEQUENCE_COLUMN, OBI_SEQ)
+            Column.new_column(o_view, REVERSE_QUALITY_COLUMN, OBI_QUAL, associated_column_name=REVERSE_SEQUENCE_COLUMN, associated_column_version=o_view[REVERSE_SEQUENCE_COLUMN].version)
+        
+    # Initialize multiple elements columns
+    if type(output_0)!=BufferedWriter: 
+        dict_cols = {}
+        for v_uri in config["cat"]["views_to_cat"]:
+            v = open_uri(v_uri)[1]
+            for coln in v.keys():
+                col = v[coln]
+                if v[coln].nb_elements_per_line > 1:
+                    if coln not in dict_cols:
+                        dict_cols[coln] = {}
+                        dict_cols[coln]['eltnames'] = set(v[coln].elements_names)
+                        dict_cols[coln]['nbelts'] = v[coln].nb_elements_per_line
+                        dict_cols[coln]['obitype'] = v[coln].data_type_int
+                    else:
+                        dict_cols[coln]['eltnames'] = set(v[coln].elements_names + list(dict_cols[coln]['eltnames']))
+                        dict_cols[coln]['nbelts'] = len(dict_cols[coln]['eltnames'])
+            v.close()            
+        for coln in dict_cols:
+            Column.new_column(o_view, coln, dict_cols[coln]['obitype'], 
+                              nb_elements_per_line=dict_cols[coln]['nbelts'], elements_names=list(dict_cols[coln]['eltnames']), dict_column=True)
+    
+    # Initialize the progress bar
+    if not config['obi']['noprogressbar']:
+        pb = ProgressBar(total_len, config)
+    else:
+        pb = None
+    
+    i = 0
+    for v_uri in config["cat"]["views_to_cat"]:
+        v = open_uri(v_uri)[1]
+        for entry in v:
+            PyErr_CheckSignals()
+            if pb is not None:
+                pb(i)
+            if type(output_0)==BufferedWriter: 
+                rep = repr(entry)
+                output_0.write(str2bytes(rep)+b"\n")
+            else:
+                try:
+                    o_view[i] = entry
+                except:
+                    print("\nError with entry:", repr(entry))
+                    print(repr(o_view))
+            i+=1
+        v.close()
+
+    # Deletes quality columns if needed
+    if type(output_0)!=BufferedWriter: 
+        if QUALITY_COLUMN in o_view and remove_qual :
+            o_view.delete_column(QUALITY_COLUMN)
+        if REVERSE_QUALITY_COLUMN in o_view and remove_rev_qual :
+            o_view.delete_column(REVERSE_QUALITY_COLUMN)
+    
+    if pb is not None:
+        pb(i, force=True)
+        print("", file=sys.stderr)
+    
+    # Save command config in DMS comments
+    command_line = " ".join(sys.argv[1:])
+    o_view.write_config(config, "cat", command_line, input_dms_name=[d.name for d in idms_list], input_view_name=[vname for vname in iview_list])
+    o_dms.record_command_line(command_line)
+
+    #print("\n\nOutput view:\n````````````", file=sys.stderr)
+    #print(repr(view), file=sys.stderr)
+
+    for d in idms_list:
+        d.close(force=True)
+    o_dms.close(force=True)
+    
+    logger("info", "Done.")

python/obitools3/commands/clean.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/clean.pyx

@@ -4,13 +4,14 @@ from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms.dms cimport DMS
 from obitools3.dms.view import RollbackException
 from obitools3.dms.capi.obiclean cimport obi_clean
-from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.uri.decode import open_uri
 from obitools3.apps.config import logger
 from obitools3.utils cimport tobytes, str2bytes
 from obitools3.dms.view.view cimport View
 from obitools3.dms.view.typed_view.view_NUC_SEQS cimport View_NUC_SEQS
 
+from io import BufferedWriter
 import sys
 
 
@@ -21,7 +22,8 @@ def addOptions(parser):
 
     addMinimalInputOption(parser)
     addMinimalOutputOption(parser)
- 
+    addNoProgressBarOption(parser)
+    
     group = parser.add_argument_group('obi clean specific options')
 
     group.add_argument('--distance', '-d',
@@ -36,8 +38,7 @@ def addOptions(parser):
                        dest="clean:sample-tag-name",
                        metavar="<SAMPLE TAG NAME>",
                        type=str,
-                       default="merged_sample",
-                       help="Name of the tag where sample counts are kept.")
+                       help="Name of the tag where merged sample count informations are kept (typically generated by obi uniq, usually MERGED_sample, default: None).")
     
     group.add_argument('--ratio', '-r',
                        action="store", dest="clean:ratio",
@@ -53,11 +54,18 @@ def addOptions(parser):
                        default=False,
                        help="Only sequences labeled as heads are kept in the output. Default: False")
 
-    group.add_argument('--cluster-tags', '-C',
-                       action="store_true", 
-                       dest="clean:cluster-tags",
-                       default=False,
-                       help="Adds tags for each sequence giving its cluster's head and weight for each sample.")
+#    group.add_argument('--cluster-tags', '-C',
+#                       action="store_true", 
+#                       dest="clean:cluster-tags",
+#                       default=False,
+#                       help="Adds tags for each sequence giving its cluster's head and weight for each sample.")
+
+    group.add_argument('--thread-count','-p',   # TODO should probably be in a specific option group
+                       action="store", dest="clean:thread-count",
+                       metavar='<THREAD COUNT>',
+                       default=-1,
+                       type=int,
+                       help="Number of threads to use for the computation. Default: the maximum available.")
 
 
 def run(config):
@@ -82,17 +90,20 @@ def run(config):
     if output is None:
         raise Exception("Could not create output")
     o_dms = output[0]
+    output_0 = output[0]
     final_o_view_name = output[1]
     
-    # If the input and output DMS are not the same, run obiclean creating a temporary view that will be exported to 
+    # If stdout output or the input and output DMS are not the same, create a temporary view that will be exported to 
     # the right DMS and deleted in the other afterwards.
-    if i_dms != o_dms:
-        temporary_view_name = final_o_view_name
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
+        temporary_view_name = b"temp"
         i=0
         while temporary_view_name in i_dms:  # Making sure view name is unique in input DMS
-            temporary_view_name = final_o_view_name+b"_"+str2bytes(str(i))
+            temporary_view_name = temporary_view_name+b"_"+str2bytes(str(i))
             i+=1
         o_view_name = temporary_view_name
+        if type(output_0)==BufferedWriter:
+            o_dms = i_dms
     else:
         o_view_name = final_o_view_name
         
@@ -100,25 +111,36 @@ def run(config):
     command_line = " ".join(sys.argv[1:])
     comments = View.print_config(config, "clean", command_line, input_dms_name=[i_dms_name], input_view_name=[i_view_name])
 
-    if obi_clean(tobytes(i_dms_name), tobytes(i_view_name), tobytes(config['clean']['sample-tag-name']), tobytes(o_view_name), comments, \
-              config['clean']['distance'], config['clean']['ratio'], config['clean']['heads-only'], 1) < 0:
+    if 'sample-tag-name' not in config['clean']:
+        config['clean']['sample-tag-name'] = ""
+        
+    if obi_clean(i_dms.name_with_full_path, tobytes(i_view_name), tobytes(config['clean']['sample-tag-name']), tobytes(o_view_name), comments, \
+              config['clean']['distance'], config['clean']['ratio'], config['clean']['heads-only'], config['clean']['thread-count']) < 0:
         raise Exception("Error running obiclean")
     
     # If the input and output DMS are not the same, export result view to output DMS
     if i_dms != o_dms:
         View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], o_view_name, final_o_view_name)
-    
+
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view = o_dms[o_view_name]
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
     # Save command config in DMS comments
     o_dms.record_command_line(command_line)
 
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(o_dms[final_o_view_name]), file=sys.stderr)
 
-    # If the input and the output DMS are different, delete the temporary result view in the input DMS
-    if i_dms != o_dms:
+    # If the input and the output DMS are different or if stdout output, delete the temporary imported view used to create the final view
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
         View.delete_view(i_dms, o_view_name)
-        o_dms.close()
+        o_dms.close(force=True)
     
-    i_dms.close()
+    i_dms.close(force=True)
 
-    logger("info", "Done.")
+    logger("info", "Done.")
+    

python/obitools3/commands/clean_dms.pyx

@@ -0,0 +1,30 @@
+#cython: language_level=3
+
+from obitools3.apps.optiongroups import addMinimalInputOption
+from obitools3.uri.decode import open_uri
+from obitools3.dms import DMS
+from obitools3.dms.capi.obidms cimport obi_clean_dms
+from obitools3.apps.config import logger
+from obitools3.utils cimport tobytes
+
+ 
+__title__="Clean a DMS from unfinished views and columns"
+ 
+
+def addOptions(parser):
+    addMinimalInputOption(parser)
+  
+
+def run(config):
+    
+    DMS.obi_atexit()
+    
+    logger("info", "obi clean_dms")
+    
+    dms_path = tobytes(config['obi']['inputURI'])
+    if b'.obidms' in dms_path:
+        dms_path = dms_path.split(b'.obidms')[0]
+    if obi_clean_dms(dms_path) < 0 :
+        raise Exception("Error cleaning DMS", config['obi']['inputURI'])
+
+    logger("info", "Done.")

python/obitools3/commands/count.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/count.pyx

@@ -7,8 +7,10 @@ from obitools3.dms import DMS
 from obitools3.apps.optiongroups import addMinimalInputOption
 from obitools3.dms.capi.obiview cimport COUNT_COLUMN
 
+from cpython.exc cimport PyErr_CheckSignals
 
-__title__="Counts sequence records"
+
+__title__="Count sequence records"
 
  
 def addOptions(parser):
@@ -20,13 +22,19 @@ def addOptions(parser):
     group.add_argument('-s','--sequence',
                         action="store_true", dest="count:sequence",
                         default=False,
-                        help="Prints only the number of sequence records.")
+                        help="Prints only the number of sequence records (much faster, default: False).")
  
     group.add_argument('-a','--all',
                         action="store_true", dest="count:all",
                         default=False,
                         help="Prints only the total count of sequence records (if a sequence has no `count` attribute, its default count is 1) (default: False).")
 
+    group.add_argument('-c','--count-tag',
+                        action="store", dest="count:countcol",
+                        default='COUNT',
+                        type=str,
+                        help="Name of the tag/column associated with the count information (default: COUNT).")
+
     
 def run(config):
     
@@ -39,17 +47,22 @@ def run(config):
     if input is None:
         raise Exception("Could not read input")
     entries = input[1]
-
+    
+    countcol = config['count']['countcol'] 
+    
     count1 = len(entries)
     count2 = 0
     
-    if COUNT_COLUMN in entries and ((config['count']['sequence'] == config['count']['all']) or (config['count']['all'])) :
+    if countcol in entries and ((config['count']['sequence'] == config['count']['all']) or (config['count']['all'])) :
         for e in entries:
-            count2+=e[COUNT_COLUMN]
+            PyErr_CheckSignals()
+            count2+=e[countcol]
 
-    if COUNT_COLUMN in entries and (config['count']['sequence'] == config['count']['all']):
+    if countcol in entries and (config['count']['sequence'] == config['count']['all']):
         print(count1,count2)
-    elif COUNT_COLUMN in entries and config['count']['all']:
+    elif countcol in entries and config['count']['all']:
         print(count2)
     else:
         print(count1)
+    
+    input[0].close(force=True)

python/obitools3/commands/ecopcr.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/ecopcr.pyx

@@ -5,10 +5,10 @@ from obitools3.dms.dms cimport DMS
 from obitools3.dms.capi.obidms cimport OBIDMS_p
 from obitools3.dms.view import RollbackException
 from obitools3.dms.capi.obiecopcr cimport obi_ecopcr
-from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption, addTaxonomyOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption, addTaxonomyOption, addNoProgressBarOption
 from obitools3.uri.decode import open_uri
 from obitools3.apps.config import logger
-from obitools3.utils cimport tobytes
+from obitools3.utils cimport tobytes, str2bytes
 from obitools3.dms.view.typed_view.view_NUC_SEQS cimport View_NUC_SEQS
 from obitools3.dms.view import View
 
@@ -16,6 +16,7 @@ from libc.stdlib  cimport malloc, free
 from libc.stdint  cimport int32_t
 
 import sys
+from io import BufferedWriter
 
 
 __title__="in silico PCR"
@@ -27,6 +28,7 @@ def addOptions(parser):
     addMinimalInputOption(parser)
     addTaxonomyOption(parser)
     addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
 
  
     group = parser.add_argument_group('obi ecopcr specific options')
@@ -35,13 +37,15 @@ def addOptions(parser):
                        action="store", dest="ecopcr:primer1",
                        metavar='<PRIMER>',
                        type=str,
-                       help="Forward primer.")
+                       required=True,
+                       help="Forward primer, length must be less than or equal to 32")
 
     group.add_argument('--primer2', '-R',
                        action="store", dest="ecopcr:primer2",
                        metavar='<PRIMER>',
                        type=str,
-                       help="Reverse primer.")
+                       required=True,
+                       help="Reverse primer, length must be less than or equal to 32")
 
     group.add_argument('--error', '-e',
                        action="store", dest="ecopcr:error",
@@ -107,14 +111,20 @@ def addOptions(parser):
                        help="Defines the method used for estimating the Tm (melting temperature) between the primers and their corresponding "
                             "target sequences. SANTALUCIA: 1, or OWCZARZY: 2. Default: 1.")
 
+    group.add_argument('--keep-primers', '-p',
+                       action="store_true", 
+                       dest="ecopcr:keep-primers",
+                       default=False,
+                       help="Whether to keep the primers attached to the output sequences (default: the primers are cut out).")
+
     group.add_argument('--keep-nucs', '-D',
                        action="store", 
                        dest="ecopcr:keep-nucs",
-                       metavar="<INTEGER>",
+                       metavar="<N>",
                        type=int,
                        default=0,
-                       help="Keeps the specified number of nucleotides on each side of the in silico amplified sequences, "
-                            "(already including the amplified DNA fragment plus the two target sequences of the primers).")
+                       help="Keeps N nucleotides on each side of the in silico amplified sequences, "
+                            "not including the primers (implying that primers are automatically kept if N > 0).")
 
     group.add_argument('--kingdom-mode', '-k',
                        action="store_true", 
@@ -161,11 +171,28 @@ def run(config):
     if output is None:
         raise Exception("Could not create output")
     o_dms = output[0]
+    output_0 = output[0]
     o_dms_name = output[0].name
     o_view_name = output[1]
 
+    # Open the taxonomy DMS
+    taxdms = open_uri(config['obi']['taxoURI'],
+                     dms_only=True)
+    if taxdms is None:
+        raise Exception("Could not open taxonomy DMS")
+    tax_dms = taxdms[0]
+    tax_dms_name = taxdms[0].name
+    
     # Read taxonomy name    
     taxonomy_name = config['obi']['taxoURI'].split("/")[-1]   # Robust in theory
+
+    # If stdout output create a temporary view in the input dms that will be deleted afterwards.
+    if type(output_0)==BufferedWriter:
+        o_dms = i_dms
+        o_view_name = b"temp"
+        while o_view_name in i_dms:   # Making sure view name is unique in input DMS
+            o_view_name = o_view_name+b"_"+str2bytes(str(i))
+            i+=1
     
     # Save command config in View comments
     command_line = " ".join(sys.argv[1:])
@@ -178,14 +205,15 @@ def run(config):
 
     # TODO: primers in comments?
 
-    if obi_ecopcr(tobytes(i_dms_name), tobytes(i_view_name), tobytes(taxonomy_name), \
-                  tobytes(o_dms_name), tobytes(o_view_name), comments, \
+    if obi_ecopcr(i_dms.name_with_full_path, tobytes(i_view_name), 
+                  tax_dms.name_with_full_path, tobytes(taxonomy_name), \
+                  o_dms.name_with_full_path, tobytes(o_view_name), comments, \
                   tobytes(config['ecopcr']['primer1']), tobytes(config['ecopcr']['primer2']), \
                   config['ecopcr']['error'], \
                   config['ecopcr']['min-length'], config['ecopcr']['max-length'], \
                   restrict_to_taxids_p, ignore_taxids_p, \
                   config['ecopcr']['circular'], config['ecopcr']['salt-concentration'], config['ecopcr']['salt-correction-method'], \
-                  config['ecopcr']['keep-nucs'], config['ecopcr']['kingdom-mode']) < 0:
+                  config['ecopcr']['keep-nucs'], config['ecopcr']['keep-primers'], config['ecopcr']['kingdom-mode']) < 0:
         raise Exception("Error running ecopcr")
 
     # Save command config in DMS comments
@@ -193,10 +221,22 @@ def run(config):
 
     free(restrict_to_taxids_p)
     free(ignore_taxids_p)
-    
+
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view = o_dms[o_view_name]
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(o_dms[o_view_name]), file=sys.stderr)
 
-    o_dms.close()
+    # If stdout output, delete the temporary result view in the input DMS
+    if type(output_0)==BufferedWriter:
+        View.delete_view(i_dms, o_view_name)
+
+    i_dms.close(force=True)
+    o_dms.close(force=True)
 
     logger("info", "Done.")

python/obitools3/commands/ecotag.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/ecotag.pyx

@@ -4,7 +4,7 @@ from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms.dms cimport DMS
 from obitools3.dms.view import RollbackException
 from obitools3.dms.capi.obiecotag cimport obi_ecotag
-from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.uri.decode import open_uri
 from obitools3.apps.config import logger
 from obitools3.utils cimport tobytes, str2bytes
@@ -12,6 +12,7 @@ from obitools3.dms.view.view cimport View
 from obitools3.dms.view.typed_view.view_NUC_SEQS cimport View_NUC_SEQS
 
 import sys
+from io import BufferedWriter
 
 
 __title__="Taxonomic assignment of sequences"
@@ -22,6 +23,7 @@ def addOptions(parser):
     addMinimalInputOption(parser)
     addTaxonomyOption(parser)
     addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
  
     group = parser.add_argument_group('obi ecotag specific options')
 
@@ -39,6 +41,17 @@ def addOptions(parser):
                       help="Minimum identity to consider for assignment, as a normalized identity, e.g. 0.95 for an identity of 95%%. "
                            "Default: 0.00 (no threshold).")
 
+    group.add_argument('--minimum-circle','-c',
+                      action="store", dest="ecotag:bubble_threshold",
+                      metavar='<CIRCLE_THRESHOLD>',
+                      default=0.99,
+                      type=float,
+                      help="Minimum identity considered for the assignment circle "
+                           "(sequence is assigned to the LCA of all sequences within a similarity circle of the best matches; "
+                           "the threshold for this circle is the highest value between <CIRCLE_THRESHOLD> and the best assignment score found for the query sequence). "
+                           "Give value as a normalized identity, e.g. 0.95 for an identity of 95%%. "
+                           "Default: 0.99.")
+
 def run(config):
         
     DMS.obi_atexit()
@@ -63,6 +76,10 @@ def run(config):
     ref_dms_name = ref[0].name
     ref_view_name = ref[1]
 
+    # Check that the threshold demanded is greater than or equal to the threshold used to build the reference database
+    if config['ecotag']['bubble_threshold'] < eval(ref_dms[ref_view_name].comments["ref_db_threshold"]) :
+        raise Exception(f"Error: The threshold demanded ({config['ecotag']['bubble_threshold']}) is lower than the threshold used to build the reference database ({float(ref_dms[ref_view_name].comments['ref_db_threshold'])}).")
+    
     # Open the output: only the DMS
     output = open_uri(config['obi']['outputURI'],
                       input=False,
@@ -70,17 +87,19 @@ def run(config):
     if output is None:
         raise Exception("Could not create output")
     o_dms = output[0]
+    output_0 = output[0]
     final_o_view_name = output[1]
     
-    # If the input and output DMS are not the same, run ecotag creating a temporary view that will be exported to 
-    # the right DMS and deleted in the other afterwards.
-    if i_dms != o_dms:
-        temporary_view_name = final_o_view_name
+    # If stdout output or the input and output DMS are not the same, create a temporary view that will be exported and deleted.
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
+        temporary_view_name = b"temp"
         i=0
         while temporary_view_name in i_dms:  # Making sure view name is unique in input DMS
-            temporary_view_name = final_o_view_name+b"_"+str2bytes(str(i))
+            temporary_view_name = temporary_view_name+b"_"+str2bytes(str(i))
             i+=1
         o_view_name = temporary_view_name
+        if type(output_0)==BufferedWriter:
+            o_dms = i_dms
     else:
         o_view_name = final_o_view_name
 
@@ -101,11 +120,12 @@ def run(config):
     input_view_name.append("taxonomy/"+config['obi']['taxoURI'].split("/")[-1])
     comments = View.print_config(config, "ecotag", command_line, input_dms_name=input_dms_name, input_view_name=input_view_name)
 
-    if obi_ecotag(tobytes(i_dms_name), tobytes(i_view_name), \
-                  tobytes(ref_dms_name), tobytes(ref_view_name), \
-                  tobytes(taxo_dms_name), tobytes(taxonomy_name), \
-                  tobytes(o_view_name), comments, 
-                  config['ecotag']['threshold']) < 0:
+    if obi_ecotag(i_dms.name_with_full_path, tobytes(i_view_name), \
+                  ref_dms.name_with_full_path, tobytes(ref_view_name), \
+                  taxo_dms.name_with_full_path, tobytes(taxonomy_name), \
+                  tobytes(o_view_name), comments, \
+                  config['ecotag']['threshold'], \
+                  config['ecotag']['bubble_threshold']) < 0:
         raise Exception("Error running ecotag")
     
     # If the input and output DMS are not the same, export result view to output DMS
@@ -115,15 +135,24 @@ def run(config):
     # Save command config in DMS comments
     o_dms.record_command_line(command_line)
 
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view = o_dms[o_view_name]
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
+
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(o_dms[final_o_view_name]), file=sys.stderr)
 
-    # If the input and the output DMS are different, delete the temporary result view in the input DMS
-    if i_dms != o_dms:
+    # If the input and the output DMS are different or if stdout output, delete the temporary imported view used to create the final view
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
         View.delete_view(i_dms, o_view_name)
-        o_dms.close()
+        o_dms.close(force=True)
     
-    i_dms.close()
+    taxo_dms.close(force=True)
+    ref_dms.close(force=True)
+    i_dms.close(force=True)
     
     logger("info", "Done.")
 

python/obitools3/commands/export.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/export.pyx

@@ -5,11 +5,19 @@ from obitools3.uri.decode import open_uri
 from obitools3.apps.config import logger
 from obitools3.dms import DMS
 from obitools3.dms.obiseq import Nuc_Seq
+from obitools3.dms.capi.obiview cimport QUALITY_COLUMN
+from obitools3.writers.tab import TabWriter
+from obitools3.format.tab import TabFormat
+from obitools3.utils cimport tobytes, tostr
 
 from obitools3.apps.optiongroups import addMinimalInputOption, \
-                                        addExportOutputOption
+                                        addExportOutputOption, \
+                                        addNoProgressBarOption
 
 import sys
+import io
+
+from cpython.exc cimport PyErr_CheckSignals
 
 __title__="Export a view to a different file format"
 
@@ -18,6 +26,7 @@ def addOptions(parser):
     
     addMinimalInputOption(parser)
     addExportOutputOption(parser)
+    addNoProgressBarOption(parser)
 
 
 def run(config):
@@ -31,7 +40,16 @@ def run(config):
     if input is None:
         raise Exception("Could not read input")
     iview = input[1]
-            
+    
+    if 'outputformat' not in config['obi']:
+        if iview.type == b"NUC_SEQS_VIEW":
+            if QUALITY_COLUMN in iview:
+                config['obi']['outputformat'] = b'fastq'
+            else:
+                config['obi']['outputformat'] = b'fasta'
+        else:
+            config['obi']['outputformat'] = b'tabular'
+    
     # Open the output
     output = open_uri(config['obi']['outputURI'],
                       input=False)
@@ -44,26 +62,128 @@ def run(config):
      # Check that the input view has the type NUC_SEQS if needed    # TODO discuss, maybe bool property
     if (output[2] == Nuc_Seq) and (iview.type != b"NUC_SEQS_VIEW") :  # Nuc_Seq_Stored? TODO
         raise Exception("Error: the view to export in fasta or fastq format is not a NUC_SEQS view")
-   
-    # Initialize the progress bar
-    pb = ProgressBar(len(iview), config, seconde=5)
     
+    if config['obi']['only'] is not None:
+        withoutskip = min(input[4], config['obi']['only'])
+    else:
+        withoutskip = input[4]
+     
+    if config['obi']['skip'] is not None:
+        skip = min(input[4], config['obi']['skip'])
+    else:
+        skip = 0
+    
+    # Initialize the progress bar
+    if config['obi']['noprogressbar']:
+        pb = None
+    else:
+        pb = ProgressBar(withoutskip - skip, config)
+
+    if config['obi']['outputformat'] == b'metabaR':
+        # Check prefix
+        if "metabarprefix" not in config["obi"]:
+            raise Exception("Prefix needed when exporting for metabaR (--metabaR-prefix option)")
+        else:
+            metabaRprefix = config["obi"]["metabarprefix"]
+        
     i=0
     for seq in iview :
-        pb(i)
+        PyErr_CheckSignals()
+        if pb is not None:
+            pb(i)
         try:
             writer(seq)
-        except StopIteration:
+        except (StopIteration, BrokenPipeError, IOError):
             break
         i+=1
 
-    pb(i, force=True)
-    print("", file=sys.stderr)
+    if pb is not None:
+        pb(i, force=True)
+        print("", file=sys.stderr)
+
+    if config['obi']['outputformat'] == b'metabaR':
+        
+        # Export ngsfilter file if view provided
+        if 'metabarngsfilter' in config['obi']:
+            ngsfilter_input = open_uri(config['obi']['metabarngsfilter'])
+            if ngsfilter_input is None:
+                raise Exception("Could not read ngsfilter view for metabaR output")
+            ngsfilter_view = ngsfilter_input[1]
+    
+            ngsfilter_output = open(config['obi']['metabarprefix']+'.ngsfilter', 'w')
+            
+            for line in ngsfilter_view:
+
+                line_to_print = b""                
+                line_to_print += line[b'experiment']
+                line_to_print += b"\t"
+                line_to_print += line[b'sample']
+                line_to_print += b"\t"
+                line_to_print += line[b'forward_tag']
+                line_to_print += b":"
+                line_to_print += line[b'reverse_tag']
+                line_to_print += b"\t"
+                line_to_print += line[b'forward_primer']
+                line_to_print += b"\t"
+                line_to_print += line[b'reverse_primer']
+                line_to_print += b"\t"
+                line_to_print += line[b'additional_info']
+                
+                print(tostr(line_to_print), file=ngsfilter_output)
+            
+            if ngsfilter_input[0] != input[0]:
+                ngsfilter_input[0].close()
+            ngsfilter_output.close()
+            
+        # Export sample metadata
+        samples_output = open(config['obi']['metabarprefix']+'_samples.csv', 'w')
+
+        # Export sample metadata file if view provided
+        if 'metabarsamples' in config['obi']:
+            samples_input = open_uri(config['obi']['metabarsamples'])
+            if samples_input is None:
+                raise Exception("Could not read sample view for metabaR output")
+            samples_view = samples_input[1]
+        
+            # Export with tab formatter
+            TabWriter(TabFormat(header=True, sep='\t',), 
+                      samples_output,
+                      header=True)
+            
+            if samples_input[0] != input[0]:
+                samples_input[0].close()
+        
+        # Else export just sample names from main view
+        else:
+            
+            sample_list = []
+            if 'MERGED_sample' in iview:
+                sample_list = iview['MERGED_sample'].keys()
+            elif 'sample' not in iview:
+                for seq in iview:
+                    sample = seq['sample']
+                    if sample not in sample_list:
+                        sample_list.append(sample)
+            else:
+                logger("warning", "Can not read sample list from main view for metabaR sample list export")            
+            
+            print("sample_id", file=samples_output)
+            for sample in sample_list:
+                line_to_print = b""                
+                line_to_print += sample
+                line_to_print += b"\t"
+                print(tostr(line_to_print), file=samples_output)
+        
+        samples_output.close()
 
     # TODO save command in input dms?
     
-    output_object.close()
+    if not BrokenPipeError and not IOError:
+        output_object.close()
     iview.close()
-    input[0].close()
+    input[0].close(force=True)
 
     logger("info", "Done.")
+
+    if BrokenPipeError or IOError:
+        sys.stderr.close()

python/obitools3/commands/grep.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/grep.pyx

@@ -4,7 +4,7 @@ from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms import DMS
 from obitools3.dms.view.view cimport View, Line_selection
 from obitools3.uri.decode import open_uri
-from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addTaxonomyOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.dms.view import RollbackException
 from obitools3.apps.config import logger
 from obitools3.utils cimport tobytes, str2bytes
@@ -13,6 +13,9 @@ from functools import reduce
 import time
 import re
 import sys
+import ast
+from io import BufferedWriter
+from cpython.exc cimport PyErr_CheckSignals
 
  
 __title__="Grep view lines that match the given predicates"
@@ -26,21 +29,22 @@ def addOptions(parser):
     addMinimalInputOption(parser)
     addTaxonomyOption(parser)
     addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
 
     group=parser.add_argument_group("obi grep specific options")
  
     group.add_argument("--predicate", "-p",
                        action="append", dest="grep:grep_predicates",
                        metavar="<PREDICATE>",
-                       default=None,
+                       default=[],
                        type=str,
                        help="Python boolean expression to be evaluated in the "
                             "sequence/line context. The attribute name can be "
-                            "used in the expression as a variable name."
-                            "An extra variable named 'sequence' or 'line' refers"
+                            "used in the expression as a variable name. "
+                            "An extra variable named 'sequence' or 'line' refers "
                             "to the sequence or line object itself. "
                             "Several -p options can be used on the same "
-                            "commande line.")
+                            "command line.")
  
     group.add_argument("-S", "--sequence",
                        action="store", dest="grep:seq_pattern",
@@ -87,7 +91,7 @@ def addOptions(parser):
                        metavar="<ATTRIBUTE_NAME>",
                        help="Select records with the attribute <ATTRIBUTE_NAME> "
                             "defined (not set to NA value). "
-                            "Several -a options can be used on the same "
+                            "Several -A options can be used on the same "
                             "command line.")
     
     group.add_argument("-L", "--lmax",
@@ -138,16 +142,39 @@ def addOptions(parser):
                           "the sequences having at least one of them are ignored.")
 
 
-def Filter_generator(options, tax_filter):
-    #taxfilter = taxonomyFilterGenerator(options)
+def obi_compile_eval(str expr):
     
+    class MyVisitor(ast.NodeTransformer):
+        def visit_Str(self, node: ast.Str):
+            result = ast.Bytes(s = node.s.encode('utf-8'))
+            return ast.copy_location(result, node)
+   
+    expr = "obi_eval_result="+expr
+    tree = ast.parse(expr)
+    optimizer = MyVisitor()
+    tree = optimizer.visit(tree)
+    return compile(tree, filename="<ast>", mode="exec")
+
+    
+def obi_eval(compiled_expr, loc_env, line):
+
+    exec(compiled_expr, {}, loc_env)
+    obi_eval_result = loc_env["obi_eval_result"]
+    return obi_eval_result
+    
+
+def Filter_generator(options, tax_filter, i_view):
+
     # Initialize conditions
     predicates = None
-    if "predicates" in options:
-        predicates = options["predicates"]
+    if "grep_predicates" in options:
+        predicates = [obi_compile_eval(p) for p in options["grep_predicates"]]
     attributes = None
-    if "attributes" in options:
+    if "attributes" in options and len(options["attributes"]) > 0:
         attributes = options["attributes"]
+        for attribute in attributes:
+            if attribute not in i_view:
+                return None
     lmax = None
     if "lmax" in options:
         lmax = options["lmax"]
@@ -157,7 +184,7 @@ def Filter_generator(options, tax_filter):
     invert_selection = options["invert_selection"]
     id_set = None
     if "id_list" in options:
-        id_set = set(x.strip() for x in open(options["id_list"]))
+        id_set = set(x.strip() for x in open(options["id_list"], 'rb'))
 
     # Initialize the regular expression patterns
     seq_pattern = None
@@ -170,9 +197,11 @@ def Filter_generator(options, tax_filter):
     if "def_pattern" in options:
         def_pattern = re.compile(tobytes(options["def_pattern"]))
     attribute_patterns={}
-    if "attribute_patterns" in options:
+    if "attribute_patterns" in options and len(options["attribute_patterns"]) > 0:
         for p in options["attribute_patterns"]:
             attribute, pattern = p.split(":", 1)
+            if attribute not in i_view:
+                return None
             attribute_patterns[tobytes(attribute)] = re.compile(tobytes(pattern))
     
     def filter(line, loc_env):
@@ -197,7 +226,7 @@ def Filter_generator(options, tax_filter):
              
         if good and attribute_patterns:
             good = (reduce(lambda bint x, bint y : x and y, 
-                        (line[attribute] is not None for attribute in attributes),
+                        (line[attribute] is not None for attribute in attribute_patterns),
                         True)
                     and
                     reduce(lambda bint x, bint y: x and y,
@@ -205,10 +234,10 @@ def Filter_generator(options, tax_filter):
                         for attribute in attribute_patterns), 
                         True)
                    )
-             
+                     
         if good and predicates:
             good = (reduce(lambda bint x, bint y: x and y,
-                    (bool(eval(p, loc_env, line))
+                    (bool(obi_eval(p, loc_env, line))
                     for p in predicates), True))
 
         if good and lmin:
@@ -229,6 +258,13 @@ def Filter_generator(options, tax_filter):
 
 
 def Taxonomy_filter_generator(taxo, options):
+    
+    if (("required_ranks" in options and options["required_ranks"]) or \
+       ("required_taxids" in options and options["required_taxids"]) or \
+       ("ignored_taxids" in options and options["ignored_taxids"])) and \
+       (taxo is None):
+        raise RollbackException("obi grep error: can't use taxonomy options without providing a taxonomy. Rollbacking view")
+    
     if taxo is not None:
         def tax_filter(seq):
             good = True
@@ -277,52 +313,74 @@ def run(config):
     if output is None:
         raise Exception("Could not create output view")
     o_dms = output[0]
-    o_view_name_final = output[1]
-    o_view_name = o_view_name_final
-    
-    # If the input and output DMS are not the same, create output view in input DMS first, then export it
-    # to output DMS, making sure the temporary view name is unique in the input DMS 
-    if i_dms != o_dms:
+    output_0 = output[0]
+    final_o_view_name = output[1]
+
+    # If stdout output or the input and output DMS are not the same, create a temporary view that will be exported and deleted afterwards.
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
+        temporary_view_name = b"temp"
         i=0
-        while o_view_name in i_dms:
-            o_view_name = o_view_name_final+b"_"+str2bytes(str(i))
-            i+=1        
+        while temporary_view_name in i_dms:  # Making sure view name is unique in input DMS
+            temporary_view_name = temporary_view_name+b"_"+str2bytes(str(i))
+            i+=1
+        o_view_name = temporary_view_name
+        if type(output_0)==BufferedWriter:
+            o_dms = i_dms
+    else:
+        o_view_name = final_o_view_name
 
     if 'taxoURI' in config['obi'] and config['obi']['taxoURI'] is not None:
         taxo_uri = open_uri(config["obi"]["taxoURI"])
-        if taxo_uri is None:
+        if taxo_uri is None or taxo_uri[2] == bytes:
             raise Exception("Couldn't open taxonomy")
         taxo = taxo_uri[1]
     else :
         taxo = None
 
     # Initialize the progress bar
-    pb = ProgressBar(len(i_view), config, seconde=5)
+    if config['obi']['noprogressbar'] == False:
+        pb = ProgressBar(len(i_view), config)
+    else:
+        pb = None
  
     # Apply filter
     tax_filter = Taxonomy_filter_generator(taxo, config["grep"])
-    filter = Filter_generator(config["grep"], tax_filter)
+    filter = Filter_generator(config["grep"], tax_filter, i_view)        
     selection = Line_selection(i_view)
-    for i in range(len(i_view)):
-        pb(i)
-        line = i_view[i]
-                 
-        loc_env = {"sequence": line, "line": line, "taxonomy": taxo}
-        
-        good = filter(line, loc_env)
- 
-        if good :
+    
+    if filter is None and config["grep"]["invert_selection"]: # all sequences are selected: filter is None if no line will be selected because some columns don't exist
+        for i in range(len(i_view)):
+            PyErr_CheckSignals()
+            if pb is not None:
+                pb(i)
             selection.append(i)
-
-    pb(i, force=True)
-    print("", file=sys.stderr)
+    
+    elif filter is not None :   # filter is None if no line will be selected because some columns don't exist
+        for i in range(len(i_view)):
+            PyErr_CheckSignals()
+            if pb is not None:
+                pb(i)
+            line = i_view[i]
+            
+            loc_env = {"sequence": line, "line": line, "taxonomy": taxo, "obi_eval_result": False}
+            
+            good = filter(line, loc_env)
+     
+            if good :
+                selection.append(i)
+   
+    if pb is not None:
+        pb(len(i_view), force=True)
+        print("", file=sys.stderr)
 
     # Create output view with the line selection
     try:
         o_view = selection.materialize(o_view_name)
     except Exception, e:
         raise RollbackException("obi grep error, rollbacking view: "+str(e), o_view)
-    
+
+    logger("info", "Grepped %d entries" % len(o_view))
+
     # Save command config in View and DMS comments
     command_line = " ".join(sys.argv[1:])
     input_dms_name=[input[0].name]
@@ -337,16 +395,22 @@ def run(config):
     # and delete the temporary view in the input DMS
     if i_dms != o_dms:
         o_view.close()
-        View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], o_view_name, o_view_name_final)
-        o_view = o_dms[o_view_name_final]
+        View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], o_view_name, final_o_view_name)
+        o_view = o_dms[final_o_view_name]
+    
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
 
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(o_view), file=sys.stderr)
 
-    # If the input and the output DMS are different, delete the temporary imported view used to create the final view
-    if i_dms != o_dms:
+    # If the input and the output DMS are different or if stdout output, delete the temporary imported view used to create the final view
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
         View.delete_view(i_dms, o_view_name)
-        o_dms.close()
-    i_dms.close()
+        o_dms.close(force=True)
+    i_dms.close(force=True)
 
     logger("info", "Done.")

python/obitools3/commands/head.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/head.pyx

@@ -4,22 +4,28 @@ from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms import DMS
 from obitools3.dms.view.view cimport View, Line_selection
 from obitools3.uri.decode import open_uri
-from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption
+from obitools3.apps.optiongroups import addMinimalInputOption, addMinimalOutputOption, addNoProgressBarOption
 from obitools3.dms.view import RollbackException
 from obitools3.apps.config import logger
 from obitools3.utils cimport str2bytes
+from obitools3.apps.optiongroups import addExportOutputOption
 
 import time
 import sys
- 
+from io import BufferedWriter
 
-__title__="Keep the N first lines of a view."
+from cpython.exc cimport PyErr_CheckSignals
+
+
+__title__="Keep the N first lines of a view"
 
  
 def addOptions(parser):
     
     addMinimalInputOption(parser)
     addMinimalOutputOption(parser)
+    addExportOutputOption(parser)
+    addNoProgressBarOption(parser)
 
     group=parser.add_argument_group('obi head specific options')
 
@@ -51,30 +57,41 @@ def run(config):
     if output is None:
         raise Exception("Could not create output view")
     o_dms = output[0]
-    o_view_name_final = output[1]
-    o_view_name = o_view_name_final
+    output_0 = output[0]
+    final_o_view_name = output[1]
     
-    # If the input and output DMS are not the same, create output view in input DMS first, then export it
-    # to output DMS, making sure the temporary view name is unique in the input DMS 
-    if i_dms != o_dms:
+    # If stdout output or the input and output DMS are not the same, create a temporary view that will be exported and deleted.
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
+        temporary_view_name = b"temp"
         i=0
-        while o_view_name in i_dms:
-            o_view_name = o_view_name_final+b"_"+str2bytes(str(i))
-            i+=1        
+        while temporary_view_name in i_dms:  # Making sure view name is unique in input DMS
+            temporary_view_name = temporary_view_name+b"_"+str2bytes(str(i))
+            i+=1
+        o_view_name = temporary_view_name
+        if type(output_0)==BufferedWriter:
+            o_dms = i_dms
+    else:
+        o_view_name = final_o_view_name
     
     n = min(config['head']['count'], len(i_view))
 
     # Initialize the progress bar
-    pb = ProgressBar(n, config, seconde=5)
+    if config['obi']['noprogressbar'] == False:
+        pb = ProgressBar(n, config)
+    else:
+        pb = None
         
     selection = Line_selection(i_view)
     
     for i in range(n):
-        pb(i)
+        PyErr_CheckSignals()
+        if pb is not None:
+            pb(i)
         selection.append(i)
 
-    pb(i, force=True)
-    print("", file=sys.stderr)
+    if pb is not None:
+        pb(i, force=True)
+        print("", file=sys.stderr)
 
     # Create output view with the line selection
     try:
@@ -91,16 +108,22 @@ def run(config):
     # and delete the temporary view in the input DMS
     if i_dms != o_dms:
         o_view.close()
-        View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], o_view_name, o_view_name_final)
-        o_view = o_dms[o_view_name_final]
+        View.import_view(i_dms.full_path[:-7], o_dms.full_path[:-7], o_view_name, final_o_view_name)
+        o_view = o_dms[final_o_view_name]
+
+    # stdout output: write to buffer
+    if type(output_0)==BufferedWriter:
+        logger("info", "Printing to output...")
+        o_view.print_to_output(output_0, noprogressbar=config['obi']['noprogressbar'])
+        o_view.close()
 
     #print("\n\nOutput view:\n````````````", file=sys.stderr)
     #print(repr(view), file=sys.stderr)
 
-    # If the input and the output DMS are different, delete the temporary imported view used to create the final view
-    if i_dms != o_dms:
+    # If the input and the output DMS are different or if stdout output, delete the temporary imported view used to create the final view
+    if i_dms != o_dms or type(output_0)==BufferedWriter:
         View.delete_view(i_dms, o_view_name)
-        o_dms.close()
-    i_dms.close()
+        o_dms.close(force=True)
+    i_dms.close(force=True)
     
     logger("info", "Done.")

python/obitools3/commands/history.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/history.pyx

@@ -51,7 +51,8 @@ def run(config):
         print(bytes2str(entries.dot_history_graph))
     elif config['history']['format'] == "ascii" :
         if isinstance(entries, View):
-            print(bytes2str(entries.ascii_history_graph))
+            print(bytes2str(entries.ascii_history))
         else:
             raise Exception("ASCII history only available for views")
-            
+    
+    input[0].close(force=True)

python/obitools3/commands/import.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/import.pyx

@@ -2,6 +2,7 @@
 
 import sys
 import os
+import re
 
 from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.dms.view.view cimport View
@@ -11,9 +12,11 @@ from obitools3.dms.column.column cimport Column
 from obitools3.dms.obiseq cimport Nuc_Seq
 from obitools3.dms import DMS
 from obitools3.dms.taxo.taxo cimport Taxonomy
+from obitools3.files.uncompress cimport CompressedFile
 
 
 from obitools3.utils cimport tobytes, \
+                             tostr, \
                              get_obitype, \
                              update_obitype
 
@@ -23,24 +26,38 @@ from obitools3.dms.capi.obiview cimport VIEW_TYPE_NUC_SEQS, \
                                         DEFINITION_COLUMN, \
                                         QUALITY_COLUMN, \
                                         COUNT_COLUMN, \
-                                        TAXID_COLUMN
+                                        TAXID_COLUMN, \
+                                        MERGED_PREFIX, \
+                                        SCIENTIFIC_NAME_COLUMN
                                         
+from obitools3.dms.capi.obidms cimport obi_import_view
+
 from obitools3.dms.capi.obitypes cimport obitype_t, \
                                          OBI_VOID, \
-                                         OBI_QUAL
+                                         OBI_QUAL, \
+                                         OBI_STR, \
+                                         OBI_INT
 
 from obitools3.dms.capi.obierrno cimport obi_errno
 
 from obitools3.apps.optiongroups import addImportInputOption, \
                                         addTabularInputOption, \
                                         addTaxdumpInputOption, \
-                                        addMinimalOutputOption
+                                        addMinimalOutputOption, \
+                                        addNoProgressBarOption, \
+                                        addTaxonomyOption
 
 from obitools3.uri.decode import open_uri
 
 from obitools3.apps.config import logger
 
-__title__="Imports sequences from different formats into a DMS"
+from cpython.exc cimport PyErr_CheckSignals
+
+from io import BufferedWriter
+import ast
+
+
+__title__="Import sequences from different formats into a DMS"
  
  
 default_config = {   'destview'     : None,
@@ -57,8 +74,27 @@ def addOptions(parser):
     addImportInputOption(parser)
     addTabularInputOption(parser)
     addTaxdumpInputOption(parser)
+    addTaxonomyOption(parser)
     addMinimalOutputOption(parser)
+    addNoProgressBarOption(parser)
 
+    group = parser.add_argument_group('obi import specific options')
+
+    group.add_argument('--preread',
+                     action="store_true", dest="import:preread",
+                     default=False,
+                     help="Do a first readthrough of the dataset if it contains huge dictionaries (more than 100 keys) for "
+                          "a much faster import. This option is not recommended and will slow down the import in any other case.")
+
+    group.add_argument('--space-priority',
+                     action="store_true", dest="import:space_priority",
+                     default=False,
+                     help="If importing a view into another DMS, do it by importing each line, saving disk space if the original view "
+                          "has a line selection associated.")
+
+#    group.add_argument('--only-id',
+#                     action="store", dest="import:onlyid",
+#                     help="only id")
 
 def run(config):
     
@@ -71,6 +107,10 @@ def run(config):
     cdef   obitype_t   new_type
     cdef   bint        get_quality
     cdef   bint        NUC_SEQS_view
+    cdef   bint        silva
+    cdef   bint        rdp
+    cdef   bint        unite
+    cdef   bint        sintax
     cdef   int         nb_elts
     cdef   object      d
     cdef   View        view
@@ -81,6 +121,8 @@ def run(config):
     cdef   Column      seq_col
     cdef   Column      qual_col
     cdef   Column      old_column
+    cdef   Column      sci_name_col
+    cdef   bytes       sci_name
     cdef   bint        rewrite
     cdef   dict        dcols
     cdef   int         skipping
@@ -99,44 +141,86 @@ def run(config):
     logger("info", "obi import: imports an object (file(s), obiview, taxonomy...) into a DMS")
     
     entry_count = -1
+    pb = None
     
     if not config['obi']['taxdump']:
         input = open_uri(config['obi']['inputURI'])
         if input is None:  # TODO check for bytes instead now?
             raise Exception("Could not open input URI")
         
-        entry_count = input[4]
-        logger("info", "Importing %d entries", entry_count)
+        if config['obi']['only'] is not None:
+            entry_count = min(input[4], config['obi']['only'])
+        else:
+            entry_count = input[4]
+        
+        if entry_count > 0:
+            logger("info", "Importing %d entries", entry_count)
+        else:
+            logger("info", "Importing an unknown number of entries")
         
         # TODO a bit dirty?
-        if input[2]==Nuc_Seq:
+        if input[2]==Nuc_Seq or input[2]==View_NUC_SEQS:
             v = View_NUC_SEQS
         else:
             v = View 
     else:
         v = None
-
+    
+    if config['obi']['taxdump'] or (isinstance(input[1], View) and not config['import']['space_priority']): 
+        dms_only=True
+    else:
+        dms_only=False
+    
     output = open_uri(config['obi']['outputURI'],
                       input=False,
-                      newviewtype=v)
+                      newviewtype=v,
+                      dms_only=dms_only)
     if output is None:
-        raise Exception("Could not create output view")
-        
+        raise Exception("Could not open output")
+    
+    o_dms = output[0]
+    
     # Read taxdump
     if config['obi']['taxdump']:  # The input is a taxdump to import in a DMS
-        taxo = Taxonomy.open_taxdump(output[0], config['obi']['inputURI'])
-        taxo.write(output[1])
+        # Check if taxonomy name isn't already taken
+        taxo_name = output[1].split(b'/')[1]
+        if Taxonomy.exists(o_dms, taxo_name):
+            raise Exception("Taxonomy name already exists in this DMS")
+        taxo = Taxonomy.open_taxdump(o_dms, config['obi']['inputURI'])
+        taxo.write(taxo_name)
         taxo.close()
-        output[0].record_command_line(" ".join(sys.argv[1:]))
-        output[0].close()
+        o_dms.record_command_line(" ".join(sys.argv[1:]))
+        o_dms.close(force=True)
+        logger("info", "Done.")
         return
-    
-    if entry_count >= 0:
-        pb = ProgressBar(entry_count, config, seconde=5)
+
+    # Open taxonomy if there is one
+    if 'taxoURI' in config['obi'] and config['obi']['taxoURI'] is not None:
+        taxo_uri = open_uri(config['obi']['taxoURI'])
+        if taxo_uri is None or taxo_uri[2] == bytes:
+            raise Exception("Couldn't open taxonomy")
+        taxo = taxo_uri[1]
+
+    else :
+        taxo = None
+
+    # If importing a view between two DMS and not wanting to save space if line selection in original view, use C API
+    if isinstance(input[1], View) and not config['import']['space_priority']:
+        if obi_import_view(input[0].name_with_full_path, o_dms.name_with_full_path, input[1].name, tobytes((config['obi']['outputURI'].split('/'))[-1])) < 0 :
+            input[0].close(force=True)
+            output[0].close(force=True)
+            raise Exception("Error importing a view in a DMS")
+        o_dms.record_command_line(" ".join(sys.argv[1:]))
+        input[0].close(force=True)
+        output[0].close(force=True)
+        logger("info", "Done.")
+        return
+
+    # Reinitialize the progress bar
+    if entry_count >= 0 and config['obi']['noprogressbar'] == False:
+        pb = ProgressBar(entry_count, config)
     else:
         pb = None
-    
-    entries = input[1]
         
     NUC_SEQS_view = False
     if isinstance(output[1], View) :
@@ -145,141 +229,308 @@ def run(config):
             NUC_SEQS_view = True
     else: 
         raise NotImplementedError()
-    
+
     # Save basic columns in variables for optimization
     if NUC_SEQS_view :
         id_col = view[ID_COLUMN]
         def_col = view[DEFINITION_COLUMN]
         seq_col = view[NUC_SEQUENCE_COLUMN]
-        
+
+    # Prepare taxon scientific name and taxid refs if RDP/SILVA/UNITE/SINTAX formats
+    silva = False
+    rdp = False
+    unite = False
+    sintax=False
+    if 'inputformat' in config['obi'] and (config['obi']['inputformat'] == b"silva" or \
+                                           config['obi']['inputformat'] == b"rdp" or \
+                                           config['obi']['inputformat'] == b"unite" or \
+                                           config['obi']['inputformat'] == b"sintax"):
+        #if taxo is None:
+        #    raise Exception("A taxonomy (as built by 'obi import --taxdump') must be provided for SILVA and RDP files")
+        if config['obi']['inputformat'] == b"silva":
+            silva = True
+        elif config['obi']['inputformat'] == b"rdp":
+            rdp = True
+        elif config['obi']['inputformat'] == b"unite":
+            unite = True
+        elif config['obi']['inputformat'] == b"sintax":
+            sintax = True
+        sci_name_col = Column.new_column(view, SCIENTIFIC_NAME_COLUMN, OBI_STR)
+        if taxo is not None:
+            taxid_col = Column.new_column(view, TAXID_COLUMN, OBI_INT)
+
     dcols = {}
+
+    # First read through the entries to prepare columns with dictionaries as they are very time-expensive to rewrite
+    if config['import']['preread']:
+        logger("info", "First readthrough...")
+        entries = input[1]
+        i = 0
+        dict_dict = {}
+        for entry in entries:
+            PyErr_CheckSignals()
         
+            if entry is None:  # error or exception handled at lower level, not raised because Python generators can't resume after any exception is raised
+                if config['obi']['skiperror']:
+                    i-=1
+                    continue
+                else:
+                    raise Exception("obi import error in first readthrough")
+            
+            if pb is not None:
+                pb(i)
+            elif not i%50000:
+                logger("info", "Read %d entries", i)
+    
+            for tag in entry :
+                newtag = tag
+                if tag[:7] == b"merged_":
+                    newtag = MERGED_PREFIX+tag[7:]
+                if type(entry[tag]) == dict :
+                    if tag in dict_dict:
+                        dict_dict[newtag][0].update(entry[tag].keys())
+                    else:
+                        dict_dict[newtag] = [set(entry[tag].keys()), get_obitype(entry[tag])]
+            i+=1
+        
+        if pb is not None:
+            pb(i, force=True)
+            print("", file=sys.stderr)
+       
+        for tag in dict_dict:
+            dcols[tag] = (Column.new_column(view, tag, dict_dict[tag][1], \
+                              nb_elements_per_line=len(dict_dict[tag][0]), \
+                              elements_names=list(dict_dict[tag][0]), \
+                              dict_column=True), \
+                          dict_dict[tag][1])
+    
+        
+        # Reinitialize the input
+        if isinstance(input[0], CompressedFile):
+            input_is_file = True
+        
+        # Reinitialize the progress bar
+        if entry_count >= 0 and config['obi']['noprogressbar'] == False:
+            pb = ProgressBar(entry_count, config)
+        else:
+            pb = None
+
+        try:
+            input[0].close()
+        except AttributeError:
+            pass
+        input = open_uri(config['obi']['inputURI'], force_file=input_is_file)
+        if input is None:
+            raise Exception("Could not open input URI")
+    
+#    if 'onlyid' in config['import']:
+#        onlyid = tobytes(config['import']['onlyid'])
+#    else:
+#        onlyid = None
+    
+    entries = input[1]
     i = 0
     for entry in entries :
         
+        PyErr_CheckSignals()
+
         if entry is None:  # error or exception handled at lower level, not raised because Python generators can't resume after any exception is raised
             if config['obi']['skiperror']:
-                i-=1
                 continue
             else:
                 raise RollbackException("obi import error, rollbacking view", view)
         
         if pb is not None:
             pb(i)
-                        
-        if NUC_SEQS_view: 
-            id_col[i] = entry.id
-            def_col[i] = entry.definition
-            seq_col[i] = entry.seq
-            # Check if there is a sequencing quality associated by checking the first entry    # TODO haven't found a more robust solution yet
-            if i == 0:
-                get_quality = QUALITY_COLUMN in entry
+        elif not i%50000:
+            logger("info", "Imported %d entries", i)
+        
+#        if onlyid is not None and entry.id != onlyid:
+#            continue
+        
+        try:
+             
+            if NUC_SEQS_view: 
+                id_col[i] = entry.id           
+                def_col[i] = entry.definition
+                seq_col[i] = entry.seq
+                # Check if there is a sequencing quality associated by checking the first entry    # TODO haven't found a more robust solution yet
+                if i == 0:
+                    get_quality = QUALITY_COLUMN in entry
+                    if get_quality:
+                        Column.new_column(view, QUALITY_COLUMN, OBI_QUAL)
+                        qual_col = view[QUALITY_COLUMN]
                 if get_quality:
-                    Column.new_column(view, QUALITY_COLUMN, OBI_QUAL)
-                    qual_col = view[QUALITY_COLUMN]
-            if get_quality:
-                qual_col[i] = entry.quality
-         
-        for tag in entry :
-            
-            if tag != ID_COLUMN and tag != DEFINITION_COLUMN and tag != NUC_SEQUENCE_COLUMN and tag != QUALITY_COLUMN :  # TODO dirty 
-                                
-                value = entry[tag]
-                if tag == b"taxid":
-                    tag = TAXID_COLUMN
-                if tag == b"count":
-                    tag = COUNT_COLUMN
+                    qual_col[i] = entry.quality
                     
-                if tag not in dcols :
-                     
-                    value_type = type(value)
-                    nb_elts = 1
-                    value_obitype = OBI_VOID
-                     
-                    if value_type == dict or value_type == list :
-                        nb_elts = len(value)
-                        elt_names = list(value)
-                    else :
-                        nb_elts = 1
-                        elt_names = None
-                     
-                    value_obitype = get_obitype(value)
-                     
-                    if value_obitype != OBI_VOID :
-                        dcols[tag] = (Column.new_column(view, tag, value_obitype, nb_elements_per_line=nb_elts, elements_names=elt_names), value_obitype)
-                                                 
-                        # Fill value
-                        dcols[tag][0][i] = value
-                     
-                    # TODO else log error?
- 
-                else :
-         
-                    rewrite = False
- 
-                    # Check type adequation
-                    old_type = dcols[tag][1]
-                    new_type = OBI_VOID
-                    new_type = update_obitype(old_type, value)
-                    if old_type != new_type :
-                        rewrite = True
- 
-                    try:
-                        # Fill value
-                        dcols[tag][0][i] = value
-                     
-                    except IndexError :
-                                                 
-                        value_type = type(value)
-                        old_column = dcols[tag][0]
-                        old_nb_elements_per_line = old_column.nb_elements_per_line
-                        new_nb_elements_per_line = 0
-                        old_elements_names = old_column.elements_names
-                        new_elements_names = None
-     
-                        #####################################################################
-                         
-                        # Check the length and keys of column lines if needed
-                        if value_type == dict :    # Check dictionary keys
-                            for k in value :
-                                if k not in old_elements_names :
-                                    new_elements_names = list(set(old_elements_names+[tobytes(k) for k in value]))
-                                    rewrite = True
+                # Parse taxon scientific name if RDP or Silva or Unite file
+                if (rdp or silva or unite or sintax):
+                    if rdp or silva:
+                        sci_names = entry.definition.split(b";")
+                        sci_name_col[i] = sci_names[-1]
+                    elif unite:
+                        sci_names = entry.id.split(b'|')[-1].split(b';')
+                        sci_name_col[i] = re.sub(b'[a-zA-Z]__', b'', sci_names[-1])
+                    elif sintax:
+                        reconstructed_line = entry.id+b' '+entry.definition[:-1]
+                        splitted_reconstructed_line = reconstructed_line.split(b';')
+                        taxa = splitted_reconstructed_line[1].split(b'=')[1]
+                        taxa = splitted_reconstructed_line[1].split(b',')
+                        sci_names = []
+                        for t in taxa:
+                            tf = t.split(b':')[1]
+                            sci_names.append(tf)
+                        sci_name_col[i] = sci_names[-1]
+                        id_col[i] = reconstructed_line.split(b';')[0]    
+                        def_col[i] = reconstructed_line
+                        
+                    # Fond taxid if taxonomy provided
+                    if taxo is not None :
+                        for sci_name in reversed(sci_names):
+                            if unite:
+                                sci_name = re.sub(b'[a-zA-Z]__', b'', sci_name)
+                            if sci_name.split()[0] != b'unidentified' and sci_name.split()[0] != b'uncultured' and sci_name.split()[0] != b'metagenome':
+                                taxon = taxo.get_taxon_by_name(sci_name)
+                                if taxon is not None:
+                                    sci_name_col[i] = taxon.name
+                                    taxid_col[i] = taxon.taxid
+                                    #print(taxid_col[i], sci_name_col[i])
                                     break
-                         
-                        elif value_type == list or value_type == tuple :  # Check vector length
-                            if old_nb_elements_per_line < len(value) :
-                                new_nb_elements_per_line = len(value)
-                                rewrite = True
-                         
-                        #####################################################################
-                         
-                        if rewrite :
-                            if new_nb_elements_per_line == 0 and new_elements_names is not None :
-                                new_nb_elements_per_line = len(new_elements_names)
-                             
-                            # Reset obierrno 
-                            obi_errno = 0
- 
-                            dcols[tag] = (view.rewrite_column_with_diff_attributes(old_column.name, 
-                                                                                   new_data_type=new_type, 
-                                                                                   new_nb_elements_per_line=new_nb_elements_per_line,
-                                                                                   new_elements_names=new_elements_names,
-                                                                                   rewrite_last_line=False), 
-                                          value_obitype)
-                             
-                            # Update the dictionary:
-                            for t in dcols :
-                                dcols[t] = (view[t], dcols[t][1])
-                             
+
+            for tag in entry :
+                
+                if tag != ID_COLUMN and tag != DEFINITION_COLUMN and tag != NUC_SEQUENCE_COLUMN and tag != QUALITY_COLUMN :  # TODO dirty 
+                
+                    value = entry[tag]
+                    if tag == b"taxid":
+                        tag = TAXID_COLUMN
+                    if tag == b"count":
+                        tag = COUNT_COLUMN
+                    if tag == b"scientific_name":
+                        tag = SCIENTIFIC_NAME_COLUMN
+                    if tag[:7] == b"merged_":
+                        tag = MERGED_PREFIX+tag[7:]
+
+                    if type(value) == bytes and value[:1]==b"[" :
+                        try:
+                            if type(eval(value)) == list:
+                                value = eval(value)
+                                #print(value)
+                        except:
+                            pass
+                    
+                    if tag not in dcols :
+                    
+                        value_type = type(value)
+                        nb_elts = 1
+                        value_obitype = OBI_VOID
+                        dict_col = False
+                        
+                        if value_type == dict :
+                            nb_elts = len(value)
+                            elt_names = list(value)
+                            dict_col = True
+                        else :
+                            nb_elts = 1
+                            elt_names = None
+                        
+                        if value_type == list :
+                            tuples = True
+                        else:
+                            tuples = False
+                        
+                        value_obitype = get_obitype(value)
+                                                
+                        if value_obitype != OBI_VOID :
+                            dcols[tag] = (Column.new_column(view, tag, value_obitype, nb_elements_per_line=nb_elts, elements_names=elt_names, dict_column=dict_col, tuples=tuples), value_obitype)
+                                                     
                             # Fill value
                             dcols[tag][0][i] = value
-                                    
+                         
+                        # TODO else log error?
+     
+                    else :
+             
+                        rewrite = False
+     
+                        # Check type adequation
+                        old_type = dcols[tag][1]
+                        new_type = OBI_VOID
+                        new_type = update_obitype(old_type, value)
+                        if old_type != new_type :
+                            rewrite = True
+     
+                        try:
+                            # Check that it's not the case where the first entry contained a dict of length 1 and now there is a new key                        
+                            if type(value) == dict and \
+                                dcols[tag][0].nb_elements_per_line == 1 \
+                                and set(dcols[tag][0].elements_names) != set(value.keys()) :
+                                raise IndexError  # trigger column rewrite
+                            
+                            # Fill value
+                            dcols[tag][0][i] = value
+                         
+                        except (IndexError, OverflowError):
+                                                                                 
+                            value_type = type(value)
+                            old_column = dcols[tag][0]
+                            old_nb_elements_per_line = old_column.nb_elements_per_line
+                            new_nb_elements_per_line = 0
+                            old_elements_names = old_column.elements_names
+                            new_elements_names = None
+         
+                            #####################################################################
+                             
+                            # Check the length and keys of column lines if needed
+                            if value_type == dict :    # Check dictionary keys
+                                for k in value :
+                                    if k not in old_elements_names :
+                                        new_elements_names = list(set(old_elements_names+[tobytes(k) for k in value]))
+                                        rewrite = True
+                                        break
+                             
+                            elif value_type == list or value_type == tuple :  # Check vector length
+                                if old_nb_elements_per_line < len(value) :
+                                    new_nb_elements_per_line = len(value)
+                                    rewrite = True
+                             
+                            #####################################################################
+                             
+                            if rewrite :
+                                if new_nb_elements_per_line == 0 and new_elements_names is not None :
+                                    new_nb_elements_per_line = len(new_elements_names)
+                                 
+                                # Reset obierrno 
+                                obi_errno = 0
+     
+                                dcols[tag] = (view.rewrite_column_with_diff_attributes(old_column.name, 
+                                                                                       new_data_type=new_type, 
+                                                                                       new_nb_elements_per_line=new_nb_elements_per_line,
+                                                                                       new_elements_names=new_elements_names,
+                                                                                       rewrite_last_line=False), 
+                                              new_type)
+                                 
+                                # Update the dictionary:
+                                for t in dcols :
+                                    dcols[t] = (view[t], dcols[t][1])
+                                 
+                                # Fill value
+                                dcols[tag][0][i] = value
+        
+        except Exception as e:
+            print("\nCould not import sequence:\n", repr(entry), "\nError raised:", e, "\n/!\ Check if '--input-na-string' option needs to be set")
+            if 'skiperror' in config['obi'] and not config['obi']['skiperror']:
+                raise e
+            else:
+                pass
+                i-=1  # overwrite problematic entry
+        
         i+=1
  
     if pb is not None:
         pb(i, force=True)
         print("", file=sys.stderr)
+    logger("info", "Imported %d entries", len(view))
     
     # Save command config in View and DMS comments
     command_line = " ".join(sys.argv[1:])
@@ -294,7 +545,7 @@ def run(config):
     except AttributeError:
         pass
     try:
-        output[0].close()
+        output[0].close(force=True)
     except AttributeError:
         pass
 

python/obitools3/commands/less.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h

python/obitools3/commands/less.pyx

@@ -1,44 +1,50 @@
 #cython: language_level=3
 
-from obitools3.apps.optiongroups import addMinimalInputOption
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
 from obitools3.uri.decode import open_uri
 from obitools3.dms import DMS
+from obitools3.utils cimport tobytes
+from obitools3.dms.capi.obiview cimport QUALITY_COLUMN
+from obitools3.apps.optiongroups import addMinimalInputOption
+
+import sys
+import io
+from subprocess import Popen, PIPE
+
+from cpython.exc cimport PyErr_CheckSignals
 
- 
 __title__="Less equivalent"
- 
+
 
 def addOptions(parser):
-
+    
     addMinimalInputOption(parser)
- 
-    group=parser.add_argument_group('obi less specific options')
- 
-    group.add_argument('--print', '-n',
-                     action="store", dest="less:print",
-                     metavar='<N>',
-                     default=10,
-                     type=int,
-                     help="Print N entries (default: 10)")
- 
+
 
 def run(config):
-    
-    cdef object entries
-    cdef int    n
-    
-    DMS.obi_atexit()
-    
-    input = open_uri(config['obi']['inputURI'])
-       
-    entries = input[1]
-    
-    if config['less']['print'] > len(entries) :
-        n = len(entries)
-    else :
-        n = config['less']['print']
         
-    # Print
-    for i in range(n) :
-        print(repr(entries[i]))
+    DMS.obi_atexit()
+        
+    # Open the input
+    input = open_uri(config['obi']['inputURI'])
+    if input is None:
+        raise Exception("Could not read input")
+    iview = input[1]
+    
+    process = Popen(["less"], stdin=PIPE)
+    
+    for seq in iview :
+        PyErr_CheckSignals()
+        try:
+            process.stdin.write(tobytes(repr(seq)))
+            process.stdin.write(b"\n")
+        except (StopIteration, BrokenPipeError, IOError):
+            break
+
+    sys.stderr.close()
+    process.stdin.close()
+    process.wait()
+
+    iview.close()
+    input[0].close(force=True)
 

python/obitools3/commands/ls.cfiles

@@ -1,103 +0,0 @@
-../../../src/obi_lcs.h
-../../../src/obi_lcs.c
-../../../src/obierrno.h
-../../../src/obierrno.c
-../../../src/upperband.h
-../../../src/upperband.c
-../../../src/sse_banded_LCS_alignment.h
-../../../src/sse_banded_LCS_alignment.c
-../../../src/obiblob.h
-../../../src/obiblob.c
-../../../src/utils.h
-../../../src/utils.c
-../../../src/obidms.h
-../../../src/obidms.c
-../../../src/libjson/json_utils.h
-../../../src/libjson/json_utils.c
-../../../src/libjson/cJSON.h
-../../../src/libjson/cJSON.c
-../../../src/obiavl.h
-../../../src/obiavl.c
-../../../src/bloom.h
-../../../src/bloom.c
-../../../src/crc64.h
-../../../src/crc64.c
-../../../src/murmurhash2.h
-../../../src/murmurhash2.c
-../../../src/obidmscolumn.h
-../../../src/obidmscolumn.c
-../../../src/obitypes.h
-../../../src/obitypes.c
-../../../src/obidmscolumndir.h
-../../../src/obidmscolumndir.c
-../../../src/obiblob_indexer.h
-../../../src/obiblob_indexer.c
-../../../src/obiview.h
-../../../src/obiview.c
-../../../src/hashtable.h
-../../../src/hashtable.c
-../../../src/linked_list.h
-../../../src/linked_list.c
-../../../src/obidmscolumn_array.h
-../../../src/obidmscolumn_array.c
-../../../src/obidmscolumn_blob.h
-../../../src/obidmscolumn_blob.c
-../../../src/obidmscolumn_idx.h
-../../../src/obidmscolumn_idx.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_qual.h
-../../../src/obidmscolumn_qual.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_str.h
-../../../src/obidmscolumn_str.c
-../../../src/array_indexer.h
-../../../src/array_indexer.c
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
-../../../src/dna_seq_indexer.h
-../../../src/dna_seq_indexer.c
-../../../src/encode.c
-../../../src/encode.h
-../../../src/uint8_indexer.c
-../../../src/uint8_indexer.h
-../../../src/build_reference_db.c
-../../../src/build_reference_db.h
-../../../src/kmer_similarity.c
-../../../src/kmer_similarity.h
-../../../src/obi_clean.c
-../../../src/obi_clean.h
-../../../src/obi_ecopcr.c
-../../../src/obi_ecopcr.h
-../../../src/obi_ecotag.c
-../../../src/obi_ecotag.h
-../../../src/obidms_taxonomy.c
-../../../src/obidms_taxonomy.h
-../../../src/obilittlebigman.c
-../../../src/obilittlebigman.h
-../../../src/_sse.h
-../../../src/obidebug.h
-../../../src/libecoPCR/libapat/CODES/dft_code.h
-../../../src/libecoPCR/libapat/CODES/dna_code.h
-../../../src/libecoPCR/libapat/CODES/prot_code.h
-../../../src/libecoPCR/libapat/apat_parse.c
-../../../src/libecoPCR/libapat/apat_search.c
-../../../src/libecoPCR/libapat/apat.h
-../../../src/libecoPCR/libapat/Gmach.h
-../../../src/libecoPCR/libapat/Gtypes.h
-../../../src/libecoPCR/libapat/libstki.c
-../../../src/libecoPCR/libapat/libstki.h
-../../../src/libecoPCR/libthermo/nnparams.h
-../../../src/libecoPCR/libthermo/nnparams.c
-../../../src/libecoPCR/ecoapat.c
-../../../src/libecoPCR/ecodna.c
-../../../src/libecoPCR/ecoError.c
-../../../src/libecoPCR/ecoMalloc.c
-../../../src/libecoPCR/ecoPCR.h