Simplify the code

Conflicts:
	python/obitools3/obidms/_obidmscolumn_seq.pyx

LICENSE

@@ -0,0 +1,518 @@
+
+  CeCILL FREE SOFTWARE LICENSE AGREEMENT
+
+Version 2.1 dated 2013-06-21
+
+
+    Notice
+
+This Agreement is a Free Software license agreement that is the result
+of discussions between its authors in order to ensure compliance with
+the two main principles guiding its drafting:
+
+  * firstly, compliance with the principles governing the distribution
+    of Free Software: access to source code, broad rights granted to users,
+  * secondly, the election of a governing law, French law, with which it
+    is conformant, both as regards the law of torts and intellectual
+    property law, and the protection that it offers to both authors and
+    holders of the economic rights over software.
+
+The authors of the CeCILL (for Ce[a] C[nrs] I[nria] L[ogiciel] L[ibre]) 
+license are: 
+
+Commissariat à l'énergie atomique et aux énergies alternatives - CEA, a
+public scientific, technical and industrial research establishment,
+having its principal place of business at 25 rue Leblanc, immeuble Le
+Ponant D, 75015 Paris, France.
+
+Centre National de la Recherche Scientifique - CNRS, a public scientific
+and technological establishment, having its principal place of business
+at 3 rue Michel-Ange, 75794 Paris cedex 16, France.
+
+Institut National de Recherche en Informatique et en Automatique -
+Inria, a public scientific and technological establishment, having its
+principal place of business at Domaine de Voluceau, Rocquencourt, BP
+105, 78153 Le Chesnay cedex, France.
+
+
+    Preamble
+
+The purpose of this Free Software license agreement is to grant users
+the right to modify and redistribute the software governed by this
+license within the framework of an open source distribution model.
+
+The exercising of this right is conditional upon certain obligations for
+users so as to preserve this status for all subsequent redistributions.
+
+In consideration of access to the source code and the rights to copy,
+modify and redistribute granted by the license, users are provided only
+with a limited warranty and the software's author, the holder of the
+economic rights, and the successive licensors only have limited liability.
+
+In this respect, the risks associated with loading, using, modifying
+and/or developing or reproducing the software by the user are brought to
+the user's attention, given its Free Software status, which may make it
+complicated to use, with the result that its use is reserved for
+developers and experienced professionals having in-depth computer
+knowledge. Users are therefore encouraged to load and test the
+suitability of the software as regards their requirements in conditions
+enabling the security of their systems and/or data to be ensured and,
+more generally, to use and operate it in the same conditions of
+security. This Agreement may be freely reproduced and published,
+provided it is not altered, and that no provisions are either added or
+removed herefrom.
+
+This Agreement may apply to any or all software for which the holder of
+the economic rights decides to submit the use thereof to its provisions.
+
+Frequently asked questions can be found on the official website of the
+CeCILL licenses family (http://www.cecill.info/index.en.html) for any 
+necessary clarification.
+
+
+    Article 1 - DEFINITIONS
+
+For the purpose of this Agreement, when the following expressions
+commence with a capital letter, they shall have the following meaning:
+
+Agreement: means this license agreement, and its possible subsequent
+versions and annexes.
+
+Software: means the software in its Object Code and/or Source Code form
+and, where applicable, its documentation, "as is" when the Licensee
+accepts the Agreement.
+
+Initial Software: means the Software in its Source Code and possibly its
+Object Code form and, where applicable, its documentation, "as is" when
+it is first distributed under the terms and conditions of the Agreement.
+
+Modified Software: means the Software modified by at least one
+Contribution.
+
+Source Code: means all the Software's instructions and program lines to
+which access is required so as to modify the Software.
+
+Object Code: means the binary files originating from the compilation of
+the Source Code.
+
+Holder: means the holder(s) of the economic rights over the Initial
+Software.
+
+Licensee: means the Software user(s) having accepted the Agreement.
+
+Contributor: means a Licensee having made at least one Contribution.
+
+Licensor: means the Holder, or any other individual or legal entity, who
+distributes the Software under the Agreement.
+
+Contribution: means any or all modifications, corrections, translations,
+adaptations and/or new functions integrated into the Software by any or
+all Contributors, as well as any or all Internal Modules.
+
+Module: means a set of sources files including their documentation that
+enables supplementary functions or services in addition to those offered
+by the Software.
+
+External Module: means any or all Modules, not derived from the
+Software, so that this Module and the Software run in separate address
+spaces, with one calling the other when they are run.
+
+Internal Module: means any or all Module, connected to the Software so
+that they both execute in the same address space.
+
+GNU GPL: means the GNU General Public License version 2 or any
+subsequent version, as published by the Free Software Foundation Inc.
+
+GNU Affero GPL: means the GNU Affero General Public License version 3 or
+any subsequent version, as published by the Free Software Foundation Inc.
+
+EUPL: means the European Union Public License version 1.1 or any
+subsequent version, as published by the European Commission.
+
+Parties: mean both the Licensee and the Licensor.
+
+These expressions may be used both in singular and plural form.
+
+
+    Article 2 - PURPOSE
+
+The purpose of the Agreement is the grant by the Licensor to the
+Licensee of a non-exclusive, transferable and worldwide license for the
+Software as set forth in Article 5 <#scope> hereinafter for the whole
+term of the protection granted by the rights over said Software.
+
+
+    Article 3 - ACCEPTANCE
+
+3.1 The Licensee shall be deemed as having accepted the terms and
+conditions of this Agreement upon the occurrence of the first of the
+following events:
+
+  * (i) loading the Software by any or all means, notably, by
+    downloading from a remote server, or by loading from a physical medium;
+  * (ii) the first time the Licensee exercises any of the rights granted
+    hereunder.
+
+3.2 One copy of the Agreement, containing a notice relating to the
+characteristics of the Software, to the limited warranty, and to the
+fact that its use is restricted to experienced users has been provided
+to the Licensee prior to its acceptance as set forth in Article 3.1
+<#accepting> hereinabove, and the Licensee hereby acknowledges that it
+has read and understood it.
+
+
+    Article 4 - EFFECTIVE DATE AND TERM
+
+
+      4.1 EFFECTIVE DATE
+
+The Agreement shall become effective on the date when it is accepted by
+the Licensee as set forth in Article 3.1 <#accepting>.
+
+
+      4.2 TERM
+
+The Agreement shall remain in force for the entire legal term of
+protection of the economic rights over the Software.
+
+
+    Article 5 - SCOPE OF RIGHTS GRANTED
+
+The Licensor hereby grants to the Licensee, who accepts, the following
+rights over the Software for any or all use, and for the term of the
+Agreement, on the basis of the terms and conditions set forth hereinafter.
+
+Besides, if the Licensor owns or comes to own one or more patents
+protecting all or part of the functions of the Software or of its
+components, the Licensor undertakes not to enforce the rights granted by
+these patents against successive Licensees using, exploiting or
+modifying the Software. If these patents are transferred, the Licensor
+undertakes to have the transferees subscribe to the obligations set
+forth in this paragraph.
+
+
+      5.1 RIGHT OF USE
+
+The Licensee is authorized to use the Software, without any limitation
+as to its fields of application, with it being hereinafter specified
+that this comprises:
+
+ 1. permanent or temporary reproduction of all or part of the Software
+    by any or all means and in any or all form.
+
+ 2. loading, displaying, running, or storing the Software on any or all
+    medium.
+
+ 3. entitlement to observe, study or test its operation so as to
+    determine the ideas and principles behind any or all constituent
+    elements of said Software. This shall apply when the Licensee
+    carries out any or all loading, displaying, running, transmission or
+    storage operation as regards the Software, that it is entitled to
+    carry out hereunder.
+
+
+      5.2 ENTITLEMENT TO MAKE CONTRIBUTIONS
+
+The right to make Contributions includes the right to translate, adapt,
+arrange, or make any or all modifications to the Software, and the right
+to reproduce the resulting software.
+
+The Licensee is authorized to make any or all Contributions to the
+Software provided that it includes an explicit notice that it is the
+author of said Contribution and indicates the date of the creation thereof.
+
+
+      5.3 RIGHT OF DISTRIBUTION
+
+In particular, the right of distribution includes the right to publish,
+transmit and communicate the Software to the general public on any or
+all medium, and by any or all means, and the right to market, either in
+consideration of a fee, or free of charge, one or more copies of the
+Software by any means.
+
+The Licensee is further authorized to distribute copies of the modified
+or unmodified Software to third parties according to the terms and
+conditions set forth hereinafter.
+
+
+        5.3.1 DISTRIBUTION OF SOFTWARE WITHOUT MODIFICATION
+
+The Licensee is authorized to distribute true copies of the Software in
+Source Code or Object Code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and that, in the event that only the Object Code of the Software is
+redistributed, the Licensee allows effective access to the full Source
+Code of the Software for a period of at least three years from the
+distribution of the Software, it being understood that the additional
+acquisition cost of the Source Code shall not exceed the cost of the
+data transfer.
+
+
+        5.3.2 DISTRIBUTION OF MODIFIED SOFTWARE
+
+When the Licensee makes a Contribution to the Software, the terms and
+conditions for the distribution of the resulting Modified Software
+become subject to all the provisions of this Agreement.
+
+The Licensee is authorized to distribute the Modified Software, in
+source code or object code form, provided that said distribution
+complies with all the provisions of the Agreement and is accompanied by:
+
+ 1. a copy of the Agreement,
+
+ 2. a notice relating to the limitation of both the Licensor's warranty
+    and liability as set forth in Articles 8 and 9,
+
+and, in the event that only the object code of the Modified Software is
+redistributed,
+
+ 3. a note stating the conditions of effective access to the full source
+    code of the Modified Software for a period of at least three years
+    from the distribution of the Modified Software, it being understood
+    that the additional acquisition cost of the source code shall not
+    exceed the cost of the data transfer.
+
+
+        5.3.3 DISTRIBUTION OF EXTERNAL MODULES
+
+When the Licensee has developed an External Module, the terms and
+conditions of this Agreement do not apply to said External Module, that
+may be distributed under a separate license agreement.
+
+
+        5.3.4 COMPATIBILITY WITH OTHER LICENSES
+
+The Licensee can include a code that is subject to the provisions of one
+of the versions of the GNU GPL, GNU Affero GPL and/or EUPL in the
+Modified or unmodified Software, and distribute that entire code under
+the terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+The Licensee can include the Modified or unmodified Software in a code
+that is subject to the provisions of one of the versions of the GNU GPL,
+GNU Affero GPL and/or EUPL and distribute that entire code under the
+terms of the same version of the GNU GPL, GNU Affero GPL and/or EUPL.
+
+
+    Article 6 - INTELLECTUAL PROPERTY
+
+
+      6.1 OVER THE INITIAL SOFTWARE
+
+The Holder owns the economic rights over the Initial Software. Any or
+all use of the Initial Software is subject to compliance with the terms
+and conditions under which the Holder has elected to distribute its work
+and no one shall be entitled to modify the terms and conditions for the
+distribution of said Initial Software.
+
+The Holder undertakes that the Initial Software will remain ruled at
+least by this Agreement, for the duration set forth in Article 4.2 <#term>.
+
+
+      6.2 OVER THE CONTRIBUTIONS
+
+The Licensee who develops a Contribution is the owner of the
+intellectual property rights over this Contribution as defined by
+applicable law.
+
+
+      6.3 OVER THE EXTERNAL MODULES
+
+The Licensee who develops an External Module is the owner of the
+intellectual property rights over this External Module as defined by
+applicable law and is free to choose the type of agreement that shall
+govern its distribution.
+
+
+      6.4 JOINT PROVISIONS
+
+The Licensee expressly undertakes:
+
+ 1. not to remove, or modify, in any manner, the intellectual property
+    notices attached to the Software;
+
+ 2. to reproduce said notices, in an identical manner, in the copies of
+    the Software modified or not.
+
+The Licensee undertakes not to directly or indirectly infringe the
+intellectual property rights on the Software of the Holder and/or
+Contributors, and to take, where applicable, vis-à-vis its staff, any
+and all measures required to ensure respect of said intellectual
+property rights of the Holder and/or Contributors.
+
+
+    Article 7 - RELATED SERVICES
+
+7.1 Under no circumstances shall the Agreement oblige the Licensor to
+provide technical assistance or maintenance services for the Software.
+
+However, the Licensor is entitled to offer this type of services. The
+terms and conditions of such technical assistance, and/or such
+maintenance, shall be set forth in a separate instrument. Only the
+Licensor offering said maintenance and/or technical assistance services
+shall incur liability therefor.
+
+7.2 Similarly, any Licensor is entitled to offer to its licensees, under
+its sole responsibility, a warranty, that shall only be binding upon
+itself, for the redistribution of the Software and/or the Modified
+Software, under terms and conditions that it is free to decide. Said
+warranty, and the financial terms and conditions of its application,
+shall be subject of a separate instrument executed between the Licensor
+and the Licensee.
+
+
+    Article 8 - LIABILITY
+
+8.1 Subject to the provisions of Article 8.2, the Licensee shall be
+entitled to claim compensation for any direct loss it may have suffered
+from the Software as a result of a fault on the part of the relevant
+Licensor, subject to providing evidence thereof.
+
+8.2 The Licensor's liability is limited to the commitments made under
+this Agreement and shall not be incurred as a result of in particular:
+(i) loss due the Licensee's total or partial failure to fulfill its
+obligations, (ii) direct or consequential loss that is suffered by the
+Licensee due to the use or performance of the Software, and (iii) more
+generally, any consequential loss. In particular the Parties expressly
+agree that any or all pecuniary or business loss (i.e. loss of data,
+loss of profits, operating loss, loss of customers or orders,
+opportunity cost, any disturbance to business activities) or any or all
+legal proceedings instituted against the Licensee by a third party,
+shall constitute consequential loss and shall not provide entitlement to
+any or all compensation from the Licensor.
+
+
+    Article 9 - WARRANTY
+
+9.1 The Licensee acknowledges that the scientific and technical
+state-of-the-art when the Software was distributed did not enable all
+possible uses to be tested and verified, nor for the presence of
+possible defects to be detected. In this respect, the Licensee's
+attention has been drawn to the risks associated with loading, using,
+modifying and/or developing and reproducing the Software which are
+reserved for experienced users.
+
+The Licensee shall be responsible for verifying, by any or all means,
+the suitability of the product for its requirements, its good working
+order, and for ensuring that it shall not cause damage to either persons
+or properties.
+
+9.2 The Licensor hereby represents, in good faith, that it is entitled
+to grant all the rights over the Software (including in particular the
+rights set forth in Article 5 <#scope>).
+
+9.3 The Licensee acknowledges that the Software is supplied "as is" by
+the Licensor without any other express or tacit warranty, other than
+that provided for in Article 9.2 <#good-faith> and, in particular,
+without any warranty as to its commercial value, its secured, safe,
+innovative or relevant nature.
+
+Specifically, the Licensor does not warrant that the Software is free
+from any error, that it will operate without interruption, that it will
+be compatible with the Licensee's own equipment and software
+configuration, nor that it will meet the Licensee's requirements.
+
+9.4 The Licensor does not either expressly or tacitly warrant that the
+Software does not infringe any third party intellectual property right
+relating to a patent, software or any other property right. Therefore,
+the Licensor disclaims any and all liability towards the Licensee
+arising out of any or all proceedings for infringement that may be
+instituted in respect of the use, modification and redistribution of the
+Software. Nevertheless, should such proceedings be instituted against
+the Licensee, the Licensor shall provide it with technical and legal
+expertise for its defense. Such technical and legal expertise shall be
+decided on a case-by-case basis between the relevant Licensor and the
+Licensee pursuant to a memorandum of understanding. The Licensor
+disclaims any and all liability as regards the Licensee's use of the
+name of the Software. No warranty is given as regards the existence of
+prior rights over the name of the Software or as regards the existence
+of a trademark.
+
+
+    Article 10 - TERMINATION
+
+10.1 In the event of a breach by the Licensee of its obligations
+hereunder, the Licensor may automatically terminate this Agreement
+thirty (30) days after notice has been sent to the Licensee and has
+remained ineffective.
+
+10.2 A Licensee whose Agreement is terminated shall no longer be
+authorized to use, modify or distribute the Software. However, any
+licenses that it may have granted prior to termination of the Agreement
+shall remain valid subject to their having been granted in compliance
+with the terms and conditions hereof.
+
+
+    Article 11 - MISCELLANEOUS
+
+
+      11.1 EXCUSABLE EVENTS
+
+Neither Party shall be liable for any or all delay, or failure to
+perform the Agreement, that may be attributable to an event of force
+majeure, an act of God or an outside cause, such as defective
+functioning or interruptions of the electricity or telecommunications
+networks, network paralysis following a virus attack, intervention by
+government authorities, natural disasters, water damage, earthquakes,
+fire, explosions, strikes and labor unrest, war, etc.
+
+11.2 Any failure by either Party, on one or more occasions, to invoke
+one or more of the provisions hereof, shall under no circumstances be
+interpreted as being a waiver by the interested Party of its right to
+invoke said provision(s) subsequently.
+
+11.3 The Agreement cancels and replaces any or all previous agreements,
+whether written or oral, between the Parties and having the same
+purpose, and constitutes the entirety of the agreement between said
+Parties concerning said purpose. No supplement or modification to the
+terms and conditions hereof shall be effective as between the Parties
+unless it is made in writing and signed by their duly authorized
+representatives.
+
+11.4 In the event that one or more of the provisions hereof were to
+conflict with a current or future applicable act or legislative text,
+said act or legislative text shall prevail, and the Parties shall make
+the necessary amendments so as to comply with said act or legislative
+text. All other provisions shall remain effective. Similarly, invalidity
+of a provision of the Agreement, for any reason whatsoever, shall not
+cause the Agreement as a whole to be invalid.
+
+
+      11.5 LANGUAGE
+
+The Agreement is drafted in both French and English and both versions
+are deemed authentic.
+
+
+    Article 12 - NEW VERSIONS OF THE AGREEMENT
+
+12.1 Any person is authorized to duplicate and distribute copies of this
+Agreement.
+
+12.2 So as to ensure coherence, the wording of this Agreement is
+protected and may only be modified by the authors of the License, who
+reserve the right to periodically publish updates or new versions of the
+Agreement, each with a separate number. These subsequent versions may
+address new issues encountered by Free Software.
+
+12.3 Any Software distributed under a given version of the Agreement may
+only be subsequently distributed under the same version of the Agreement
+or a subsequent version, subject to the provisions of Article 5.3.4
+<#compatibility>.
+
+
+    Article 13 - GOVERNING LAW AND JURISDICTION
+
+13.1 The Agreement is governed by French law. The Parties agree to
+endeavor to seek an amicable solution to any disagreements or disputes
+that may arise during the performance of the Agreement.
+
+13.2 Failing an amicable solution within two (2) months as from their
+occurrence, and unless emergency proceedings are necessary, the
+disagreements or disputes shall be referred to the Paris Courts having
+jurisdiction, by the more diligent Party.

README.md

@@ -0,0 +1,40 @@
+The `OBITools3`: A package for the management of analyses and data in DNA metabarcoding   
+---------------------------------------------
+
+
+
+DNA metabarcoding offers new perspectives for biodiversity research [1]. This approach of ecosystem studies relies heavily on the use of Next-Generation Sequencing (NGS), and consequently requires the ability to to treat large volumes of data. The `OBITools` package  satisfies this requirement thanks to a set of programs specifically designed for analyzing NGS data in a DNA metabarcoding context [2] - <http://metabarcoding.org/obitools>. Their capacity to filter and edit sequences while taking into account taxonomic annotation helps to setup tailored-made analysis pipelines for a broad range of DNA metabarcoding applications, including biodiversity surveys or diet analyses.   
+
+
+
+**The `OBITools3`.** This new version of the `OBITools` looks to significantly improve the storage efficiency and the data processing speed. To this end, the `OBITools3` rely on an ad hoc database system, inside which all the data that a DNA metabarcoding experiment must consider is stored: the sequences, the metadata (describing for instance the samples), the database containing the reference sequences used for the taxonomic annotation, as well as the taxonomic databases. Besides the gain in efficiency, this new structure allows an easier access to all the data associated with an experiment.   
+
+
+
+**Column-oriented storage.** An analysis pipeline corresponds to a succession of commands, each computing one step of the analysis, and where the result of the command *n* is used by the command *n+1*. DNA metabarcoding data can easily be represented in the form of tables, and each command can be regarded as an operation transforming one or several 'input' tables into one or several 'output' tables, which can be used by the next command. Many of the basic operations in a pipeline copy without modification an important part of the input tables to the result tables, and use for their calculations only a small part of the input data. In the original `OBITools`, those tables are kept in the form of annotated sequence files in the FASTA or FASTQ format. This has two consequences: i) keeping the transitional results of the analysis pipeline means using disk space for an important volume of redundant data, ii) The coding and decoding of informations that are not actually used represent an important part of the treatment process. The new database system used by the `OBITools3` (called DMS for Data Management System) relies on column-oriented storage. The columns are immutable and can be assembled in views representing the data tables. This way, the data not modified by a command in an input table can easily be associated to the result table without duplicating any information ; and the data not used at all by a command can be associated with the result table without being read. This strategy results in a gain in disk space efficiency by limiting data redundancy, as well as a gain in execution time by limiting data reading, writing and conversion operations. Finally, as a mean to optimize data access, each column is stored in a binary file directly mapped in memory for reading and writing operations.   
+
+
+
+**Storage optimization.** DNA metabarcoding data is intrinsically very redundant. For example, the same sequence corresponding to a species will be present several thousand times across all samples. In order to limit the disk space used and make comparison operations more efficient, data in the form of character strings is stored in columns using a complex indexing structure, efficient on millions of values, coupling hash functions, Bloom filters and AVL trees. Finally, DNA sequences are compressed by encoding each nucleotide on two or four bits depending on whether the sequences contain only the four nucleotides (A, C, G, T) or use the IUPAC codes.   
+
+
+
+**Saving the data processing history.** The totality of the informations used by the `OBITools3` is stored in immutable data structures in the DMS. If a command has to modify a column used as input to produce its result, a new version of that column is created, leaving the initial version intact. This storage system enables to keep, at minimal cost, the totality of the transitional results produced by the pipeline. The storage of metadata describing all the operations that have produced a view (a result table) in the DMS makes possible the creation of an oriented hypergraph, where each node corresponds to a view and each arrow to an operation. By retracing the dependency relationships in this hypergraph, it is possible to rebuild *a posteriori* the entirety of the process that has produced a result table.   
+
+
+
+**Tools.** The `OBITools3` offer the same tools as the original `OBITools`. Eventually, new versions of `ecoPrimers` (PCR primer design) [3], `ecoPCR` (*in silico* PCR) [4], as well as `Sumatra` (sequence alignment) and `Sumaclust` (sequence alignment and clustering) [5] will be added, taking advantage of the database structure developed for the `OBITools3`.    
+
+
+
+**Implementation and disponibility.** The lower layers managing the DMS as well as all the compute-intensive functions are coded in `C99` for efficiency reasons. A `Cython` (<http://www.cython.org>) object layer allows for a simple but efficient implementation of the `OBITools3` commands in `Python 3.5`. The `OBITools3` are still in development, and the first functional versions are expected for autumn 2016.    
+
+
+
+**References.**
+
+1. Taberlet P, Coissac E, Hajibabaei M, Rieseberg LH: Environmental DNA. Mol Ecol 2012:1789–1793.
+2. Boyer F, Mercier C, Bonin A, Le Bras Y, Taberlet P, Coissac E: OBITools: a Unix-inspired software package for DNA metabarcoding. Mol Ecol Resour 2015:n/a–n/a.
+3. Riaz T, Shehzad W, Viari A, Pompanon F, Taberlet P, Coissac E: ecoPrimers: inference of new DNA barcode markers from whole genome sequence analysis. Nucleic Acids Res 2011, 39:e145.
+4. Ficetola GF, Coissac E, Zundel S, Riaz T, Shehzad W, Bessière J, Taberlet P, Pompanon F: An in silico approach for the evaluation of DNA barcodes. BMC Genomics 2010, 11:434.
+5. Mercier C, Boyer F, Bonin A, Coissac E (2013) SUMATRA and SUMACLUST: fast and exact comparison and clustering of sequences. Available: <http://metabarcoding.org/sumatra> and <http://metabarcoding.org/sumaclust>

distutils.ext/obidistutils/command/build_ext.py

@@ -7,107 +7,109 @@ Created on 13 fevr. 2014
 from distutils import log
 import os
 
-from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
-
-from Cython.Compiler import Options as cython_options  # @UnresolvedImport
 
 from distutils.errors import DistutilsSetupError
 
-class build_ext(ori_build_ext):
-
+try:
+    from Cython.Distutils import build_ext  as ori_build_ext  # @UnresolvedImport
+    from Cython.Compiler import Options as cython_options  # @UnresolvedImport
+    class build_ext(ori_build_ext):
     
-    def modifyDocScripts(self):
-        build_dir_file=open("doc/sphinx/build_dir.txt","w")
-        print(self.build_lib,file=build_dir_file)
-        build_dir_file.close()
         
-    def initialize_options(self):
-        ori_build_ext.initialize_options(self)  # @UndefinedVariable
-        self.littlebigman = None
-        self.built_files = None
-
-    
-    def finalize_options(self):
-        ori_build_ext.finalize_options(self)  # @UndefinedVariable
-
-        self.set_undefined_options('littlebigman',
-                                   ('littlebigman',  'littlebigman'))
-        
-        self.set_undefined_options('build_files',
-                                   ('files',  'built_files'))
-
-        self.cython_c_in_temp = 1
-                   
-        if self.littlebigman =='-DLITTLE_END':
-            if self.define is None:
-                self.define=[('LITTLE_END',None)]
-            else:
-                self.define.append('LITTLE_END',None)
-        
-    def substitute_sources(self,exe_name,sources):
-        """
-        Substitutes source file name starting by an @ by the actual
-        name of the built file (see --> build_files)
-        """
-        sources = list(sources)
-        for i in range(len(sources)):
-            message = "%s :-> %s" % (exe_name,sources[i])
-            if sources[i][0]=='@':
-                try:
-                    filename = self.built_files[sources[i][1:]]
-                except KeyError:
-                    tmpfilename = os.path.join(self.build_temp,sources[i][1:])
-                    if os.path.isfile       (tmpfilename):
-                        filename = tmpfilename
-                    else:
-                        raise DistutilsSetupError(
-                             'The %s filename declared in the source '
-                             'files of the program %s have not been '
-                             'built by the installation process' % (sources[i],
-                                                                     exe_name))
-                sources[i]=filename
-                log.info("%s changed to %s",message,filename)
-            else:
-                log.info("%s ok",message)
-
-        return sources
-
-    def build_extensions(self):
-        # First, sanity-check the 'extensions' list
-        
-        for ext in self.extensions:
-            ext.sources = self.substitute_sources(ext.name,ext.sources)
+        def modifyDocScripts(self):
+            build_dir_file=open("doc/sphinx/build_dir.txt","w")
+            print(self.build_lib,file=build_dir_file)
+            build_dir_file.close()
             
-        self.check_extensions_list(self.extensions)
-
-        for ext in self.extensions:
-            log.info("%s :-> %s",ext.name,ext.sources)
-            ext.sources = self.cython_sources(ext.sources, ext)
-            self.build_extension(ext)
-
-        
-    def run(self):
-        self.modifyDocScripts()
-
-        for cmd_name in self.get_sub_commands():
-            self.run_command(cmd_name)
-
-        cython_options.annotate = True
-        ori_build_ext.run(self)  # @UndefinedVariable
-        
-
-    def has_files(self):
-        return self.distribution.has_files()
-
-    def has_executables(self):
-        return self.distribution.has_executables()
+        def initialize_options(self):
+            ori_build_ext.initialize_options(self)  # @UndefinedVariable
+            self.littlebigman = None
+            self.built_files = None
     
-    sub_commands = [('build_files',has_files),
-                    ('build_cexe', has_executables)
-                    ] + \
-                   ori_build_ext.sub_commands
-
         
+        def finalize_options(self):
+            ori_build_ext.finalize_options(self)  # @UndefinedVariable
+    
+            self.set_undefined_options('littlebigman',
+                                       ('littlebigman',  'littlebigman'))
+            
+            self.set_undefined_options('build_files',
+                                       ('files',  'built_files'))
+    
+            self.cython_c_in_temp = 1
+                       
+            if self.littlebigman =='-DLITTLE_END':
+                if self.define is None:
+                    self.define=[('LITTLE_END',None)]
+                else:
+                    self.define.append('LITTLE_END',None)
+            
+        def substitute_sources(self,exe_name,sources):
+            """
+            Substitutes source file name starting by an @ by the actual
+            name of the built file (see --> build_files)
+            """
+            sources = list(sources)
+            for i in range(len(sources)):
+                message = "%s :-> %s" % (exe_name,sources[i])
+                if sources[i][0]=='@':
+                    try:
+                        filename = self.built_files[sources[i][1:]]
+                    except KeyError:
+                        tmpfilename = os.path.join(self.build_temp,sources[i][1:])
+                        if os.path.isfile       (tmpfilename):
+                            filename = tmpfilename
+                        else:
+                            raise DistutilsSetupError(
+                                 'The %s filename declared in the source '
+                                 'files of the program %s have not been '
+                                 'built by the installation process' % (sources[i],
+                                                                         exe_name))
+                    sources[i]=filename
+                    log.info("%s changed to %s",message,filename)
+                else:
+                    log.info("%s ok",message)
+    
+            return sources
+    
+        def build_extensions(self):
+            # First, sanity-check the 'extensions' list
+            
+            for ext in self.extensions:
+                ext.sources = self.substitute_sources(ext.name,ext.sources)
+                
+            self.check_extensions_list(self.extensions)
+    
+            for ext in self.extensions:
+                log.info("%s :-> %s",ext.name,ext.sources)
+                ext.sources = self.cython_sources(ext.sources, ext)
+                self.build_extension(ext)
+    
+            
+        def run(self):
+            self.modifyDocScripts()
+    
+            for cmd_name in self.get_sub_commands():
+                self.run_command(cmd_name)
+    
+            cython_options.annotate = True
+            ori_build_ext.run(self)  # @UndefinedVariable
+            
+    
+        def has_files(self):
+            return self.distribution.has_files()
+    
+        def has_executables(self):
+            return self.distribution.has_executables()
+        
+        sub_commands = [('build_files',has_files),
+                        ('build_cexe', has_executables)
+                        ] + \
+                       ori_build_ext.sub_commands
+
+except ImportError:
+    from distutils.command import build_ext  # @UnusedImport
+           
 
 
         

distutils.ext/obidistutils/serenity/bootstrappip.py

@@ -0,0 +1,36 @@
+'''
+Created on 22 janv. 2016
+
+@author: coissac
+'''
+
+import sys
+from urllib import request
+import os.path
+
+from obidistutils.serenity.util import get_serenity_dir
+from obidistutils.serenity.rerun import rerun_with_anothe_python
+from obidistutils.serenity.checkpython import is_a_virtualenv_python
+
+getpipurl="https://bootstrap.pypa.io/get-pip.py"
+
+def bootstrap():
+
+    getpipfile=os.path.join(get_serenity_dir(),"get-pip.py")
+    
+    with request.urlopen(getpipurl) as getpip:
+        with open(getpipfile,"wb") as out:
+            for l in getpip:
+                out.write(l)
+                
+    python = sys.executable
+    
+    if is_a_virtualenv_python():
+        command= "%s %s" % (python,getpipfile)        
+    else:
+        command= "%s %s --user" % (python,getpipfile)
+    
+    os.system(command)
+    
+    rerun_with_anothe_python(python)
+    

distutils.ext/obidistutils/serenity/checkpackage.py

@@ -7,8 +7,13 @@ Created on 2 oct. 2014
 import re
 import os
 
-import pip                                              # @UnresolvedImport
-from pip.utils import get_installed_distributions       # @UnresolvedImport
+try:
+    import pip                                              # @UnresolvedImport
+    from pip.utils import get_installed_distributions       # @UnresolvedImport
+except ImportError:
+    from .bootstrappip import bootstrap
+    bootstrap()
+
 from distutils.version import StrictVersion             # @UnusedImport
 from distutils.errors import DistutilsError
 from distutils import log

distutils.ext/obidistutils/serenity/virtual.py

@@ -59,7 +59,7 @@ def serenity_virtualenv(envname,package,version,minversion='3.4',maxversion=None
                     clear=True, 
                     symlinks=False, 
                     with_pip=True)
-        
+                
         # check the newly created virtualenv
         return serenity_virtualenv(envname,package,version)
     

python/obi.py

@@ -16,30 +16,22 @@ It defines classes_and_methods
 @deffield    updated: Updated
 '''
 
-import sys
-import pkgutil
-import argparse
-import logging
-import json
 
-default_config = {
+default_config = { 'software'       : "The OBITools",
+                   'log'            : False,
+                   'loglevel'       : 'INFO',
+                   'progress'       : True,
+                   'defaultdms'     : None
+                  }
 
-    'obi' :        { 'log'            : True,
-                     'loglevel'       : 'INFO',
-                     'version'        : False,
-                     'progress'       : True
-                   }
-                 
-}
+root_config_name='obi'
 
-
-
-from obitools3 import command
+from obitools3.apps.config import getConfiguration     # @UnresolvedImport
 from obitools3.version import version
 
-__all__ = []
+__all__     = []
 __version__ = version
-__date__ = '2014-09-28'
+__date__    = '2014-09-28'
 __updated__ = '2014-09-28'
 
 DEBUG = 1
@@ -47,182 +39,12 @@ TESTRUN = 0
 PROFILE = 0
 
 
-
-def loadCommand(name,loader):
-    '''
-    Load a command module from its name and an ImpLoader
-    
-    This function is for internal use
-    
-    @param name:   name of the module
-    @type name: str 
-    @param loader: the module loader
-    @type loader: ImpLoader
-    
-    @return the loaded module
-    @rtype: module 
-    '''
-    
-    module = loader.find_module(name).load_module(name)
-    return module
-
-def getCommandsList():
-    '''
-    Returns the list of sub-commands available to the main `obi` command
-    
-    @return: a dict instance with key corresponding to each command and
-             value corresponding to the module
-             
-    @rtype: dict
-    '''
-    cmds = dict((x[1],loadCommand(x[1],x[0])) 
-                for x in pkgutil.iter_modules(command.__path__) 
-                if not x[2])
-    return cmds
-
-def getLogger(config):
-    '''
-    Returns the logger as defined by the command line option
-    or by the config file
-    :param config:
-    '''
- 
-    output = config['obi']['outputfilename'] 
-    level  = config['obi']['loglevel'] 
-    logfile= config['obi']['log'] 
-    
-    rootlogger   = logging.getLogger()
-    logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
-
-    stderrHandler = logging.StreamHandler(sys.stderr)
-    stderrHandler.setFormatter(logFormatter)
-
-    rootlogger.addHandler(stderrHandler)
-    
-    if logfile:
-        fileHandler = logging.FileHandler("%s.log" % output)
-        fileHandler.setFormatter(logFormatter)
-        rootlogger.addHandler(fileHandler)
-    
-    try:
-        loglevel = getattr(logging, level) 
-    except:
-        loglevel = logging.INFO
-        
-    rootlogger.setLevel(loglevel)
-    
-    config['obi']['logger']=rootlogger
-    
-    return rootlogger
-
-
-class ObiParser(argparse.ArgumentParser): 
-    def error(self, message):
-        sys.stderr.write('error: %s\n' % message)
-        self.print_help()
-        sys.exit(2)
-
-def buildArgumentParser():
-    parser = ObiParser()
-    
-    parser.add_argument('--version',   dest='obi:version', 
-                                       action='store_true', 
-                                       default=False, 
-                        help='Print the version of the OBITools')
-
-    parser.add_argument('--no-log',    dest='obi:log', 
-                                       action='store_false', 
-                                       default=None, 
-                        help='Do not create a logfile for the data analyze')
-
-    parser.add_argument('--no-progress', dest='obi:progress', 
-                                       action='store_false', 
-                                       default=None, 
-                        help='Do not print the progress bar during analyzes')
-
-    subparsers = parser.add_subparsers(title='subcommands',
-                                       description='valid subcommands',
-                                       help='additional help')
-    
-    commands = getCommandsList()
-    
-    for c in commands:
-        module = commands[c]
-        
-        if hasattr(module, "run"):
-            if hasattr(module, "__title__"):
-                sub = subparsers.add_parser(c,help=module.__title__)
-            else:
-                sub = subparsers.add_parser(c)
-    
-            if hasattr(module, "addOptions"):
-                module.addOptions(sub)
-            
-            sub.set_defaults(**{'obi:module' : module})
-                        
-    return parser
-
-def buildDefaultConfiguration():
-    global default_config
-    
-    commands = getCommandsList()
-    
-    for c in commands:
-        module = commands[c]
-    
-        assert hasattr(module, "run")
-        
-        if hasattr(module, 'default_config'):
-            default_config[c]=module.default_config
-        else:
-            default_config[c]={}
-                        
-    return default_config
-        
-
-def getConfiguration():
-    global default_config
-
-    if '__done__' in default_config:
-        return default_config
-    
-    parser = buildArgumentParser()
-    options = vars(parser.parse_args())
-    
-    config =  buildDefaultConfiguration()
-    
-    
-    for k in options:
-        section,key = k.split(':')
-        s = config[section]
-        if options[k] is not None:
-            s[key]=options[k]
-            
-    if config['obi']['version']:
-        print("The OBITools - Version %s" % __version__)
-        sys.exit(0)
-
-    if not 'module' in config['obi']:
-        print('\nError: No obi command specified',file=sys.stderr)
-        parser.print_help()
-        sys.exit(2)
-        
-            
-    if config['obi']['outputfilename'] is None:
-        config['obi']['outputfilename']=config['obi']['indexfilename']
-        
-    getLogger(config)
-    
-    config['__done__']=True
-            
-    return config
-    
-
 if __name__ =="__main__":
     
-    config = getConfiguration()    
+    config = getConfiguration(root_config_name,
+                              default_config)    
                 
-    config['obi']['module'].run(config)
+    config[root_config_name]['module'].run(config)
     
     
     

python/obitools3/apps/arguments.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef buildArgumentParser(str configname, str softname)

python/obitools3/apps/arguments.pyx

@@ -0,0 +1,61 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import argparse
+import sys
+
+
+from .command import getCommandsList
+
+class ObiParser(argparse.ArgumentParser): 
+    def error(self, message):
+        sys.stderr.write('error: %s\n' % message)
+        self.print_help()
+        sys.exit(2)
+
+cpdef buildArgumentParser(str configname, 
+                          str softname):
+    parser = ObiParser()
+    
+    parser.add_argument('--version',   dest='%s:version' % configname, 
+                                       action='store_true', 
+                                       default=False, 
+                        help='Print the version of %s' % softname)
+
+    parser.add_argument('--log',       dest='%s:log' % configname, 
+                                       action='store',
+                                       type=str,
+                                       default=None, 
+                        help='Create a logfile')
+
+    parser.add_argument('--no-progress', dest='%s:progress' % configname, 
+                                       action='store_false', 
+                                       default=None, 
+                        help='Do not print the progress bar during analyzes')
+
+    subparsers = parser.add_subparsers(title='subcommands',
+                                       description='valid subcommands',
+                                       help='additional help')
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+        
+        if hasattr(module, "run"):
+            if hasattr(module, "__title__"):
+                sub = subparsers.add_parser(c,help=module.__title__)
+            else:
+                sub = subparsers.add_parser(c)
+    
+            if hasattr(module, "addOptions"):
+                module.addOptions(sub)
+            
+            sub.set_defaults(**{'%s:module'  % configname : module})
+                        
+    return parser

python/obitools3/apps/command.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cdef object loadCommand(str name,loader)

python/obitools3/apps/command.pyx

@@ -0,0 +1,44 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import pkgutil
+
+from obitools3 import commands
+
+cdef object loadCommand(str name,loader):
+    '''
+    Load a command module from its name and an ImpLoader
+    
+    This function is for internal use
+    
+    @param name:   name of the module
+    @type name: str 
+    @param loader: the module loader
+    @type loader: ImpLoader
+    
+    @return the loaded module
+    @rtype: module 
+    '''
+    
+    module = loader.find_module(name).load_module(name)
+    return module
+
+def getCommandsList():
+    '''
+    Returns the list of sub-commands available to the main `obi` command
+    
+    @return: a dict instance with key corresponding to each command and
+             value corresponding to the module
+             
+    @rtype: dict
+    '''
+    
+    cdef dict cmds = dict((x[1],loadCommand(x[1],x[0])) 
+                           for x in pkgutil.iter_modules(commands.__path__) 
+                           if not x[2])
+    return cmds

python/obitools3/apps/config.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+cpdef str setRootConfigName(str rootname)
+cpdef str getRootConfigName()
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config)
+
+cpdef dict getConfiguration(str root_config_name=?,
+                                    dict  config=?)

python/obitools3/apps/config.pyx

@@ -0,0 +1,103 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+
+from .command   import  getCommandsList
+from .logging   cimport getLogger
+from .arguments cimport buildArgumentParser
+
+from ..version import version
+from _curses import version
+
+cdef dict __default_config__ = {}
+
+
+cpdef str setRootConfigName(str rootname):
+    global __default_config__
+    if '__root_config__' in __default_config__:
+        if __default_config__["__root_config__"] in __default_config__:
+            __default_config__[rootname]=__default_config__[__default_config__["__root_config__"]]
+            del __default_config__[__default_config__["__root_config__"]]
+    __default_config__['__root_config__']=rootname
+    return rootname
+
+cpdef str getRootConfigName():
+    global __default_config__
+    return __default_config__.get('__root_config__',None)
+
+cdef dict buildDefaultConfiguration(str root_config_name,
+                                    dict  config):
+    global __default_config__
+    
+    __default_config__.clear()
+    setRootConfigName(root_config_name)    
+
+    __default_config__[root_config_name]=config
+    
+    config['version']=version
+    
+    commands = getCommandsList()
+    
+    for c in commands:
+        module = commands[c]
+    
+        assert hasattr(module, "run")
+        
+        if hasattr(module, 'default_config'):
+            __default_config__[c]=module.default_config
+        else:
+            __default_config__[c]={}
+                        
+    return __default_config__
+        
+
+cpdef dict getConfiguration(str root_config_name="__default__",
+                            dict  config={}):
+    global __default_config__
+
+    if '__done__' in __default_config__:
+        return __default_config__
+    
+    if root_config_name=="__default__":
+        raise RuntimeError("No root_config_name specified")
+    
+    if not config:
+        raise RuntimeError("Base configuration is empty")
+    
+    
+    
+    config =  buildDefaultConfiguration(root_config_name,
+                                        config)
+    
+    parser = buildArgumentParser(root_config_name,
+                                 config[root_config_name]['software'])
+    
+    options = vars(parser.parse_args())
+
+    if options['%s:version' % root_config_name]:
+        print("%s - Version %s" % (config[root_config_name]['software'],
+                                   config[root_config_name]['version']))
+        sys.exit(0)
+    
+    for k in options:
+        section,key = k.split(':')
+        s = config[section]
+        if options[k] is not None:
+            s[key]=options[k]
+            
+    if not 'module' in config[root_config_name]:
+        print('\nError: No command specified',file=sys.stderr)
+        parser.print_help()
+        sys.exit(2)
+                
+    getLogger(config)
+    
+    config['__done__']=True
+            
+    return config

python/obitools3/apps/logging.pxd

@@ -0,0 +1,3 @@
+#cython: language_level=3
+
+cpdef getLogger(dict config)

python/obitools3/apps/logging.pyx

@@ -0,0 +1,46 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import logging
+import sys
+
+cpdef getLogger(dict config):
+    '''
+    Returns the logger as defined by the command line option
+    or by the config file
+    :param config:
+    '''
+    
+    root = config["__root_config__"]
+ 
+    level  = config[root]['loglevel'] 
+    logfile= config[root]['log'] 
+    
+    rootlogger   = logging.getLogger()
+    logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
+
+    stderrHandler = logging.StreamHandler(sys.stderr)
+    stderrHandler.setFormatter(logFormatter)
+
+    rootlogger.addHandler(stderrHandler)
+    
+    if logfile:
+        fileHandler = logging.FileHandler(logfile)
+        fileHandler.setFormatter(logFormatter)
+        rootlogger.addHandler(fileHandler)
+    
+    try:
+        loglevel = getattr(logging, level) 
+    except:
+        loglevel = logging.INFO
+        
+    rootlogger.setLevel(loglevel)
+    
+    config[root]['logger']=rootlogger
+    
+    return rootlogger

python/obitools3/apps/progress.pxd

@@ -0,0 +1,66 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+
+cdef extern from "stdio.h":
+    struct FILE
+    int fprintf(FILE *stream, char *format, ...)
+    int fputs(char *string, FILE *stream)
+    FILE* stderr
+    ctypedef unsigned int off_t "unsigned long long"
+    
+cdef extern from "unistd.h":
+    int fsync(int fd);
+
+cdef extern from "time.h":
+    struct tm :
+        int tm_yday 
+        int tm_hour
+        int tm_min
+        int tm_sec
+
+    enum: CLOCKS_PER_SEC
+    
+    ctypedef int time_t
+    ctypedef int clock_t
+    ctypedef int suseconds_t
+    
+    struct timeval:
+        time_t      tv_sec     #  seconds */
+        suseconds_t tv_usec    #  microseconds */
+
+
+    struct timezone :
+        int tz_minuteswest;    # minutes west of Greenwich
+        int tz_dsttime;        # type of DST correction 
+
+        
+    int gettimeofday(timeval *tv, timezone *tz)
+
+
+    tm *gmtime_r(time_t *clock, tm *result)
+    time_t time(time_t *tloc)
+    clock_t clock()
+
+cdef class ProgressBar:
+    cdef off_t   maxi
+    cdef clock_t starttime
+    cdef clock_t lasttime
+    cdef clock_t tickcount
+    cdef int freq
+    cdef int cycle
+    cdef int arrow
+    cdef int lastlog
+    cdef bint ontty
+    cdef int fd
+    
+    cdef bytes head
+    cdef char *chead
+    
+    cdef object logger
+
+    cdef char  *wheel
+    cdef char  *spaces
+    cdef char*  diese 
+    
+    cdef clock_t clock(self)

python/obitools3/apps/progress.pyx

@@ -0,0 +1,138 @@
+#cython: language_level=3
+
+'''
+Created on 27 mars 2016
+
+@author: coissac
+'''
+
+import sys
+from ..utils cimport bytes2str
+
+cdef class ProgressBar:
+    cdef clock_t clock(self):
+        cdef clock_t t
+        cdef timeval tp
+        cdef clock_t s
+        
+        <void> gettimeofday(&tp,NULL)
+        s = <clock_t> (<double> tp.tv_usec * 1.e-6 * <double> CLOCKS_PER_SEC)
+        t = tp.tv_sec * CLOCKS_PER_SEC + s 
+        
+        return t
+
+    def __init__(self,
+                 off_t maxi,
+                 dict  config,
+                 str head="",
+                 double seconde=0.1):
+        self.starttime = self.clock()
+        self.lasttime  = self.starttime
+        self.tickcount = <clock_t> (seconde * CLOCKS_PER_SEC)
+        self.freq      = 1
+        self.cycle     = 0
+        self.arrow     = 0
+        self.lastlog   = 0
+        
+        self.ontty = sys.stderr.isatty()
+        
+        if (maxi<=0):
+            maxi=1
+            
+        self.maxi = maxi
+        self.head = str2bytes(head)
+        self.chead= self.head 
+        
+        
+        self.logger=config[config["__root_config__"]]["logger"]
+        self.wheel =  '|/-\\'
+        self.spaces='          ' \
+                    '          ' \
+                    '          ' \
+                    '          ' \
+                    '          '
+        self.diese ='##########' \
+                    '##########' \
+                    '##########' \
+                    '##########' \
+                    '##########'  
+                           
+    def __call__(self,object pos):
+        cdef off_t    ipos
+        cdef clock_t  elapsed
+        cdef clock_t  newtime
+        cdef clock_t  delta
+        cdef clock_t  more 
+        cdef double   percent 
+        cdef tm remain
+        cdef int days,hour,minu,sec
+        cdef off_t fraction
+        cdef int twentyth
+        
+        self.cycle+=1
+    
+        if self.cycle % self.freq == 0:
+            self.cycle=1
+            newtime  = self.clock()
+            delta         = newtime - self.lasttime
+            self.lasttime = newtime
+            elapsed       = newtime - self.starttime
+#            print(" ",delta,elapsed,elapsed/CLOCKS_PER_SEC,self.tickcount)
+            
+            if   delta < self.tickcount / 5 :
+                self.freq*=2
+            elif delta > self.tickcount * 5 and self.freq>1:
+                self.freq/=2
+                
+            
+            if callable(pos):
+                ipos=pos()
+            else:
+                ipos=pos
+                
+            if ipos==0:
+                ipos=1                
+
+            percent = <double>ipos/<double>self.maxi
+            more = <time_t>((<double>elapsed / percent * (1. - percent))/CLOCKS_PER_SEC)
+            <void>gmtime_r(&more, &remain)
+            days  = remain.tm_yday 
+            hour  = remain.tm_hour
+            minu  = remain.tm_min
+            sec   = remain.tm_sec
+                
+            if self.ontty:
+                fraction=<int>(percent * 50.)
+                self.arrow=(self.arrow+1) % 4
+                self.diese[fraction]=0
+                self.spaces[50 - fraction]=0
+            
+                if days:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %d days %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    days,hour,minu,sec)
+                else:
+                    <void>fprintf(stderr,b'\r%s %5.1f %% |%s%c%s] remain : %02d:%02d:%02d',
+                                    self.chead,
+                                    percent*100.,
+                                    self.diese,self.wheel[self.arrow],self.spaces,
+                                    hour,minu,sec)
+                self.diese[fraction]=b'#'
+                self.spaces[50 - fraction]=b' '
+
+            twentyth = int(percent * 20)
+            if twentyth != self.lastlog:
+                
+                if self.ontty:
+                    <void>fputs(b'\n',stderr)
+                    
+                self.logger.info('%s %5.1f %% remain : %02d:%02d:%02d' % (
+                                        bytes2str(self.head),
+                                        percent*100.,
+                                        hour,minu,sec))
+                self.lastlog=twentyth
+        else:
+            self.cycle+=1
+

python/obitools3/commands/count.py

@@ -4,6 +4,9 @@ Created on 8 mars 2016
 @author: coissac
 '''
 
+from obitools3.apps.progress import ProgressBar  # @UnresolvedImport
+import time
+
 __title__="Counts sequences in a sequence set"
 
 
@@ -16,14 +19,14 @@ def addOptions(parser):
                                               default=None,
                         help='input data set' )
 
-    group=parser.add_option_group('Obicount specific options')
-    group.add_option('-s','--sequence',
+    group=parser.add_argument_group('Obicount specific options')
+    group.add_argument('-s','--sequence',
                              action="store_true", dest="count:sequence",
                              default=False,
                              help="Prints only the number of sequence records."
                              )
  
-    group.add_option('-a','--all',
+    group.add_argument('-a','--all',
                              action="store_true", dest="count:all",
                              default=False,
                              help="Prints only the total count of sequence records (if a sequence has no `count` attribute, its default count is 1) (default: False)."
@@ -33,4 +36,9 @@ def addOptions(parser):
     
 def run(config):
     # The code of my command
-    pass
+    pb = ProgressBar(1000,config,seconde=1)
+    
+    for i in range(1,1001):
+        pb(i)
+        time.sleep(0.01)
+        

python/obitools3/commands/import.pyx

@@ -0,0 +1,139 @@
+from obitools3.apps.progress cimport ProgressBar  # @UnresolvedImport
+from obitools3.files.universalopener cimport uopen
+from obitools3.parsers.fasta import fastaIterator
+from obitools3.parsers.fastq import fastqIterator
+from obitools3.obidms._obidms import OBIDMS
+
+
+import time
+
+__title__="Counts sequences in a sequence set"
+
+
+default_config = {   'destview'     : None,
+                     'skip'         : 0,
+                     'only'         : None,
+                     'skiperror'    : False,
+                     'seqinformat'  : None,
+                     'moltype'      : 'nuc',
+                     'filename'     : None
+                 }
+
+def addOptions(parser):
+    parser.add_argument(dest='import:filename',     
+                        metavar='<FILENAME>', 
+                        nargs='?', 
+                        default=None,
+                        help='sequence file name to be imported' )
+
+    group=parser.add_argument_group('obi import specific options')
+
+    group.add_argument('--default-dms','-d',
+                     action="store", dest="obi:defaultdms",
+                     metavar='<DMS NAME>',
+                     default=None,
+                     type=str,
+                     help="Name of the default DMS for reading and writing data")
+    
+    group.add_argument('--destination-view','-v',
+                     action="store", dest="import:destview",
+                     metavar='<VIEW NAME>',
+                     default=None,
+                     type=str,
+                     required=True,
+                     help="Name of the default DMS for reading and writing data")
+
+    group=parser.add_argument_group('obi import specific options')
+    
+    group.add_argument('--skip',
+                     action="store", dest="import:skip",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="skip the N first sequences")
+
+    group.add_argument('--only',
+                     action="store", dest="import:only",
+                     metavar='<N>',
+                     default=None,
+                     type=int,
+                     help="treat only N sequences")
+
+    group.add_argument('--skip-on-error',
+                     action="store_true", dest="import:skiperror",
+                     default=None,
+                     help="Skip sequence entries with parse error")
+    
+    group.add_argument('--fasta',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fasta',
+                     help="Input file is in fasta nucleic format (including obitools fasta extentions)")
+
+    group.add_argument('--fastq',
+                     action="store_const", dest="import:seqinformat",
+                     default=None,
+                     const='fastq',
+                     help="Input file is in sanger fastq nucleic format (standard fastq)")
+
+    group.add_argument('--nuc',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='nuc',
+                     help="Input file contains nucleic sequences")
+    
+    group.add_argument('--prot',
+                     action="store_const", dest="import:moltype",
+                     default=None,
+                     const='pep',
+                     help="Input file contains protein sequences")
+
+    
+    
+def run(config):
+    pb = ProgressBar(35000000,config,seconde=5)
+      
+    inputs = uopen(config['import']['filename'])
+    
+    get_quality = False
+    if config['import']['seqinformat']=='fasta':
+        iseq = fastaIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+    elif config['import']['seqinformat']=='fastq':
+        iseq = fastqIterator(inputs)
+        view_type="NUC_SEQS_VIEW"
+        get_quality = True
+    else:
+        raise RuntimeError('No file format specified')
+    
+    # Temporary way to handle NA values
+    #NA_list = ["nan"]
+    
+    # Create DMS
+    d = OBIDMS(config['obi']['defaultdms'])
+    
+    # Create view
+    view = d.new_view(config['import']['destview'], view_type=view_type)
+    
+    i = 0
+    for seq in iseq:
+        pb(i)
+        view[i].set_id(seq['id'])
+        view[i].set_definition(seq['definition'])
+        view[i].set_sequence(seq['sequence'])
+        if get_quality :
+            view[i].set_quality(seq['quality'])
+        for tag in seq['tags'] :
+            #print(tag, seq['tags'][tag])
+            #if seq['tags'][tag] not in NA_list :
+            view[i][tag] = seq['tags'][tag]
+        i+=1
+
+    #print(i)
+    print(view.__repr__())
+
+    view.save_and_close()
+    d.close()
+    
+    print("Done.")
+        

python/obitools3/files/linebuffer.pxd

@@ -0,0 +1,6 @@
+#cython: language_level=3
+
+
+cdef class LineBuffer:
+    cdef object fileobj
+    cdef int    size

python/obitools3/files/linebuffer.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+cdef class LineBuffer:
+
+    def __init__(self,object fileobj,int size=100000000):
+        self.fileobj=fileobj 
+        self.size=size
+    
+    def __iter__(self):
+        cdef list buff = self.fileobj.readlines(self.size)
+        cdef str  l
+        
+        while buff:
+            for l in buff:
+                yield l
+            buff = self.fileobj.readlines(self.size)
+    

python/obitools3/files/uncompress.pxd

@@ -0,0 +1,18 @@
+#cython: language_level=3
+
+cdef class MagicKeyFile:
+    cdef object stream
+    cdef str    stream_mode
+    cdef object binary
+    cdef bytes  key 
+    cdef int    keylength 
+    cdef int    pos
+    
+    cpdef bytes read(self,int size=?)
+    cpdef bytes read1(self,int size=?)
+    cpdef int tell(self)
+
+    
+cdef class CompressedFile:
+    cdef object accessor
+    

python/obitools3/files/uncompress.pyx

@@ -0,0 +1,117 @@
+#cython: language_level=3
+
+'''
+Created on 28 mars 2016
+
+@author: coissac
+'''
+
+import zipfile
+import bz2
+import gzip
+
+import io
+
+cdef class MagicKeyFile:
+    def __init__(self,stream,length=2):
+        
+        binary=stream
+        self.stream = stream
+        self.stream_mode = None 
+        if hasattr(stream, "mode"):
+            self.stream_mode = stream.mode
+            if (not 'b' in stream.mode and 
+                hasattr(stream, "buffer") and 
+                'b' in stream.buffer.mode):
+                binary=stream.buffer
+
+        if (self.stream_mode is None and 
+            not (hasattr(stream, 'headers') and 
+                 hasattr(stream.headers, "keys") and
+                 'Content-type' in stream.headers)):
+            raise TypeError("stream does not present the good interface")
+        
+        self.binary=binary
+        self.key=binary.read(length)
+        self.keylength=length
+        self.pos=0
+        
+    cpdef bytes read(self,int size=-1):
+        cdef bytes r
+        
+        if self.pos < self.keylength:
+            if size > (self.keylength - self.pos):
+                size = size - self.keylength + self.pos
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+            elif size >=0 :
+                r = self.key[self.pos:(self.pos+size)]
+                self.pos+=size
+            else:
+                r = self.key[self.pos:] + self.binary.read(size)
+                self.pos=self.keylength + 1
+        else:
+            r = self.binary.read(size)
+            
+        return r
+
+    cpdef bytes read1(self,int size=-1):
+        return self.read(size)
+
+    cpdef int tell(self):
+        cdef int p
+        
+        if self.pos < self.keylength:
+            p = self.pos 
+        else:
+            p = self.binary.tell()
+        
+        return p
+    
+    def __getattr__(self,name):
+        return getattr(self.binary, name)
+        
+
+    
+cdef class CompressedFile:
+    
+    
+    def __init__(self,stream):
+        cdef int            keylength
+        cdef MagicKeyFile   magic
+        cdef str            compressor
+        cdef bytes          k 
+        cdef object         c
+        
+        cdef dict compress = { 'zip' : (b'\x50\x4b\x03\x04',zipfile.ZipFile),
+                               'bz2' : (b'\x42\x5a\x68',bz2.BZ2File),
+                               'gz'  : (b'\x1f\x8b\x08',gzip.open)
+                   }
+        
+        keylength = max([len(x[0]) for x in compress.values()])
+        magic=MagicKeyFile(stream,keylength)
+        
+        self.accessor = None
+        
+        for compressor in compress:
+            k,c = compress[compressor]
+            if magic.key.startswith(k):
+                self.accessor = c(magic)
+        
+        if self.accessor is None:
+            self.accessor = magic
+        
+        if ((hasattr(stream, 'headers') and 
+             hasattr(stream.headers, "keys") and
+             'Content-type' in stream.headers and
+             stream.headers['Content-type'].startswith('text/')) or
+            'b' not in magic.stream_mode):
+            self.accessor = io.TextIOWrapper(self.accessor)
+        
+    
+    def __getattr__(self,name):
+        return getattr(self.accessor, name)
+    
+    def __iter__(self):
+        for x in self.accessor:
+            yield x 

python/obitools3/files/universalopener.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+from .uncompress cimport CompressedFile
+
+cpdef CompressedFile uopen(str name, mode=?)

python/obitools3/files/universalopener.pyx

@@ -0,0 +1,23 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+from urllib.request import urlopen
+
+
+cpdef CompressedFile uopen(str name, mode='r'):
+    cdef CompressedFile c
+    
+    try:
+        f = urlopen(name)
+    except:
+        f = open(name,mode) 
+        
+    c = CompressedFile(f)
+    
+    return c
+    

python/obitools3/format/fasta.pxd

@@ -0,0 +1,10 @@
+from ..utils cimport bytes2str
+from .header cimport HeaderFormat
+from cython.view cimport array as cvarray
+
+cdef class FastaFormat:
+
+    cdef HeaderFormat headerFormater
+    
+    cdef size_t  sequenceBufferLength
+    cdef char*   sequenceBuffer

python/obitools3/format/fasta.pyx

@@ -0,0 +1,32 @@
+cimport cython
+from libc.stdlib cimport malloc, free, realloc
+from libc.string cimport strncpy
+
+cdef class FastaFormat:
+	
+	def __init__(self, list tags=[], bint printNAKeys=False):
+		self.headerFormater = HeaderFormat(True,
+										   tags,
+										   printNAKeys)
+		
+	@cython.boundscheck(False)	
+	def __call__(self, dict data):
+		cdef bytes brawseq      = data['sequence']
+		cdef size_t lseq        = len(brawseq)
+		cdef size_t k=0
+		cdef list   lines       = []
+		
+		for k in range(0,lseq,60):
+			lines.append(brawseq[k:(k+60)])
+			
+		brawseq = b'\n'.join(lines)
+		
+		return "%s\n%s" % (self.headerFormater(data),bytes2str(brawseq))
+					
+		
+				
+			
+		
+		
+		
+		

python/obitools3/format/header.pxd

@@ -0,0 +1,7 @@
+cdef class HeaderFormat:
+
+    cdef str    start
+    cdef set    tags
+    cdef bint   printNaKeys
+    cdef size_t headerBufferLength
+    

python/obitools3/format/header.pyx

@@ -0,0 +1,60 @@
+
+cdef class HeaderFormat:
+    
+    def __init__(self, bint fastaHeader=True, list tags=[], bint printNAKeys=False):
+        '''
+        
+            @param fastaHeader: 
+            @type  fastaHeader: `bool`
+            
+            @param tags:
+            @type  tags: `list` of `bytes`
+            
+            @param printNAKeys: 
+            @type  printNAKeys: `bool` 
+        '''
+        
+        self.tags           = set(tags)
+        self.printNaKeys    = printNAKeys
+        
+        if fastaHeader:
+            self.start=">"
+        else:
+            self.start="@"
+        
+        self.headerBufferLength = 1000
+        #self.headerBuffer = []
+        
+    def __call__(self, dict data):
+        cdef str header
+        cdef dict tags = data['tags']
+        cdef set  ktags
+        cdef list lines = [""]
+        cdef str tagline
+        
+        if self.tags is not None and self.tags:
+            ktags = self.tags
+        else:
+            ktags = set(tags.keys())
+            
+        for k in ktags:
+            if k in tags:
+                value = tags[k]
+                if value is not None or self.printNaKeys:
+                    lines.append("%s=%s;" % (k,tags[k]))
+                    
+        if len(lines) > 1:
+            tagline=" ".join(lines)
+        else:
+            tagline=""
+            
+        if data['definition'] is not None:
+            header = "%s%s%s %s" % (self.start,data['id'],
+                                    tagline,
+                                    data['definition'])
+        else:
+            header = "%s%s%s" % (self.start,data['id'],
+                                 tagline)
+            
+        return header
+        

python/obitools3/obidms/_obidms.cfiles

@@ -8,8 +8,10 @@
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -21,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -35,17 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_bool.h
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_char.h
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_float.h
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_int.h
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_seq.h
-../../../src/obidmscolumn_str.c
-../../../src/obidmscolumn_str.h

python/obitools3/obidms/_obidms.pxd

@@ -6,11 +6,12 @@ from .capi.obiview      cimport Obiview_p
 from .capi.obitypes     cimport obiversion_t, OBIType_t, index_t
 from ._obitaxo          cimport OBI_Taxonomy
 
+
 cdef class OBIDMS_column:
 
     cdef  OBIDMS_column_p*   pointer
     cdef  OBIDMS             dms
-    cdef  Obiview_p          view
+    cdef  OBIView            view
     cdef  str                data_type
     cdef  str                dms_name
     cdef  str                column_name
@@ -45,7 +46,6 @@ cdef class OBIView:
     cdef str       name
     cdef str       comments
     cdef dict      columns
-    cdef dict      columns_pp    # TODO this dict might be unnecessary
     cdef OBIDMS    dms
     
     cpdef delete_column(self, str column_name)
@@ -69,7 +69,8 @@ cdef class OBIView_NUC_SEQS(OBIView):
 
     cdef OBIDMS_column ids
     cdef OBIDMS_column sequences
-    cdef OBIDMS_column descriptions
+    cdef OBIDMS_column definitions
+    cdef OBIDMS_column qualities
     
     cpdef delete_column(self, str column_name)
     
@@ -90,5 +91,5 @@ cdef class OBIDMS:
     cpdef OBIView open_view(self, str view_name)
     cpdef OBIView new_view(self, str view_name, object view_to_clone=*, list line_selection=*, str view_type=*, str comments=*)
     cpdef dict read_view_infos(self, str view_name)
-    cpdef dict read_views(self)
+#    cpdef dict read_views(self)    TODO
 

python/obitools3/obidms/_obidms.pyx

@@ -6,9 +6,10 @@ from .capi.obidms cimport obi_dms, \
                           obi_close_dms
                           
 from .capi.obidmscolumn cimport obi_close_column, \
-                                obi_column_format_date, \
                                 OBIDMS_column_p, \
                                 OBIDMS_column_header_p
+
+from .capi.obiutils cimport obi_format_date
                                 
 from .capi.obitypes cimport const_char_p, \
                             OBIType_t, \
@@ -16,6 +17,7 @@ from .capi.obitypes cimport const_char_p, \
                             OBI_FLOAT, \
                             OBI_BOOL, \
                             OBI_CHAR, \
+                            OBI_QUAL, \
                             OBI_STR, \
                             OBI_SEQ, \
                             name_data_type, \
@@ -42,6 +44,9 @@ from ._obidmscolumn_bool cimport OBIDMS_column_bool, \
 from ._obidmscolumn_char cimport OBIDMS_column_char, \
                                  OBIDMS_column_multi_elts_char
 
+from ._obidmscolumn_qual cimport OBIDMS_column_qual, \
+                                 OBIDMS_column_multi_elts_qual
+
 from ._obidmscolumn_str cimport OBIDMS_column_str, \
                                 OBIDMS_column_multi_elts_str
 
@@ -49,19 +54,19 @@ from ._obidmscolumn_seq cimport OBIDMS_column_seq, \
                                 OBIDMS_column_multi_elts_seq
 
 from .capi.obiview cimport Obiview_p, \
-                           Obiviews_infos_all_p, \
                            Obiview_infos_p, \
                            Column_reference_p, \
                            obi_new_view_nuc_seqs, \
                            obi_new_view, \
                            obi_new_view_cloned_from_name, \
                            obi_new_view_nuc_seqs_cloned_from_name, \
+                           obi_view_map_file, \
+                           obi_view_unmap_file, \
                            obi_open_view,  \
-                           obi_read_views, \
-                           obi_unmap_read_views, \
                            obi_view_delete_column, \
                            obi_view_add_column, \
                            obi_view_get_column, \
+                           obi_view_get_column, \
                            obi_view_get_pointer_on_column_in_view, \
                            obi_select_line, \
                            obi_select_lines, \
@@ -69,7 +74,8 @@ from .capi.obiview cimport Obiview_p, \
                            VIEW_TYPE_NUC_SEQS, \
                            NUC_SEQUENCE_COLUMN, \
                            ID_COLUMN, \
-                           DESCRIPTION_COLUMN
+                           DEFINITION_COLUMN, \
+                           QUALITY_COLUMN
                            
 from libc.stdlib cimport malloc
 from cpython.pycapsule cimport PyCapsule_New, PyCapsule_GetPointer
@@ -83,13 +89,13 @@ cdef class OBIDMS_column :
         cdef OBIDMS_column_p  column_p
         cdef OBIDMS_column_p* column_pp
         
-        column_pp = <OBIDMS_column_p*> PyCapsule_GetPointer(((view.columns_pp)[column_name]), NULL) # or use C function
+        column_pp = obi_view_get_pointer_on_column_in_view(view.pointer, str2bytes(column_name))
         column_p = column_pp[0]     # TODO ugly cython dereferencing but can't find better
         
         # Fill structure
         self.pointer = column_pp
         self.dms = view.dms
-        self.view = view.pointer    # TODO pointer or instance?
+        self.view = view
         self.data_type = bytes2str(name_data_type((column_p.header).returned_data_type))
         self.column_name = bytes2str((column_p.header).name)
         self.nb_elements_per_line = (column_p.header).nb_elements_per_line
@@ -119,7 +125,7 @@ cdef class OBIDMS_column :
             yield self.get_line(line_nb)
     
     cpdef update_pointer(self):
-        self.pointer = <OBIDMS_column_p*> obi_view_get_pointer_on_column_in_view(self.view, str2bytes(self.column_name))
+        self.pointer = <OBIDMS_column_p*> obi_view_get_pointer_on_column_in_view(self.view.pointer, str2bytes(self.column_name))
 
     cpdef list get_elements_names(self):
         return self.elements_names
@@ -131,17 +137,20 @@ cdef class OBIDMS_column :
         return (self.pointer)[0].header.lines_used
         
     cpdef str get_creation_date(self):
-        return bytes2str(obi_column_format_date((self.pointer)[0].header.creation_date))
+        return bytes2str(obi_format_date((self.pointer)[0].header.creation_date))
     
     cpdef str get_comments(self):
         return bytes2str((self.pointer)[0].header.comments)
     
-    def __repr__(self) :
+    def __str__(self) :
         cdef str to_print
         to_print = ''
         for line in self :
             to_print = to_print + str(line) + "\n"
         return to_print
+
+    def __repr__(self) :
+        return (self.column_name + ", version " + str((self.pointer)[0].header.version) + ", data type: " + self.data_type)
         
     cpdef close(self):
         if obi_close_column((self.pointer)[0]) < 0 :
@@ -182,6 +191,11 @@ cdef class OBIDMS_column :
                 subclass = OBIDMS_column_char
             else :
                 subclass = OBIDMS_column_multi_elts_char
+        elif col_type == OBI_QUAL :
+            if col_one_element_per_line :
+                subclass = OBIDMS_column_qual
+            else :
+                subclass = OBIDMS_column_multi_elts_qual
         elif col_type == OBI_STR :
             if col_one_element_per_line :
                 subclass = OBIDMS_column_str
@@ -235,28 +249,28 @@ cdef class OBIDMS_column_line :
 cdef class OBIView :
     
     def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
-        
-        cdef Obiview_p view = NULL
-        cdef int i
-        cdef list col_list
-        cdef str col_name
-        cdef OBIDMS_column column
-        cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
+                
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
         cdef OBIDMS_column_header_p header
-        cdef index_t* line_selection_p
-        cdef object col_capsule
+        cdef index_t*               line_selection_p
         
         self.dms = dms
         
+        # Create the C array for the line selection if needed
         if line_selection is not None :
             line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
             for i in range(len(line_selection)) :
-                line_selection_p[i] = line_selection[i]     # TODO type problem?
+                line_selection_p[i] = line_selection[i]
             line_selection_p[len(line_selection)] = -1
         else :
             line_selection_p = NULL
        
+        # Create the view if needed
         if new :
             if view_to_clone is not None :
                 if type(view_to_clone) == str :
@@ -265,102 +279,73 @@ cdef class OBIView :
                     view = obi_new_view(dms.pointer, str2bytes(view_name), (<OBIView> view_to_clone).pointer, line_selection_p, str2bytes(comments))
             elif view_to_clone is None :
                 view = obi_new_view(dms.pointer, str2bytes(view_name), NULL, line_selection_p, str2bytes(comments))
+        # Else, open the existing view
         elif not new :
             if view_name is not None :
                 view = obi_open_view(dms.pointer, str2bytes(view_name))
             elif view_name is None :
-                view = obi_open_view(dms.pointer, NULL)
+                view = obi_open_view(dms.pointer, NULL) # TODO discuss
         
         if view == NULL :
-            raise Exception("Error creating/opening view")
+            raise Exception("Error creating/opening a view")
 
         self.pointer = view
-        self.name = bytes2str(view.name)
+        self.name = bytes2str(view.infos.name)
         
-        # go through columns to build list and open python object (TODO make separate function?)
+        # Go through columns to build list of corresponding python instances
         self.columns = {}
-        self.columns_pp = {}
-                
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO discuss
-            (self.columns_pp)[col_name] = col_capsule
-            
-            subclass = OBIDMS_column.get_subclass_type(column_p)
+        for i in range(view.infos.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
             self.columns[col_name] = subclass(self, col_name)
-            
-            i+=1
 
 
     def __repr__(self) :
-        
         cdef str s
-        cdef OBIDMS_column column
-        cdef OBIDMS_column_p column_p
-        
-        s = self.name
-        s = s + ", " + self.comments + ", " + str(self.pointer.line_count) + " lines" 
-        for column_name in self.columns :   # TODO make function in OBIDMS_column class
-            column = self.columns[column_name]
-            column_p = (column.pointer)[0]
-            s = s + "\n" + column_name + ", version " + str(column_p.header.version) + ", data type: " + column.data_type
+        s = str(self.name) + ", " + str(self.comments) + ", " + str(self.pointer.infos.line_count) + " lines\n" 
+        for column_name in self.columns :
+            s = s + self.columns[column_name].__repr__() + '\n'
         return s
         
     
     cpdef delete_column(self, str column_name) :
         
         cdef int i
-        cdef Obiview_p view
+        cdef Obiview_p view_p
         cdef OBIDMS_column column
         cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
         cdef OBIDMS_column_header_p header
         cdef str column_n
  
         view = self.pointer
  
-        if obi_view_delete_column(view, str2bytes(column_name)) < 0 :
+        if obi_view_delete_column(view_p, str2bytes(column_name)) < 0 :
             raise Exception("Problem deleting a column from a view")
         
         # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
         (self.columns).pop(column_name)
-        (self.columns_pp).pop(column_name)
         
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)
-            (self.columns_pp)[col_name] = col_capsule
-            i+=1
-
         for column_n in self.columns :
             (self.columns[column_n]).update_pointer()
-
+    
     
     cpdef add_column(self, 
                      str column_name, 
                      obiversion_t version_number=-1, 
                      str type='',
                      index_t nb_lines=0, 
-                     index_t nb_elements_per_line=1,    # TODO 1?
+                     index_t nb_elements_per_line=1,
                      list elements_names=None,
                      str indexer_name="",
                      str comments="",
-                     bint create=True   # TODO
+                     bint create=True
                      ) :
         
         cdef bytes column_name_b
         cdef bytes elements_names_b
         cdef object subclass
-        cdef OBIDMS_column_p* column_pp
         cdef OBIDMS_column_p column_p
         
         column_name_b = str2bytes(column_name)
@@ -379,6 +364,8 @@ cdef class OBIView :
                 data_type = OBI_BOOL
             elif type == 'OBI_CHAR' :
                 data_type = OBI_CHAR
+            elif type == 'OBI_QUAL' :
+                data_type = OBI_QUAL
             elif type == 'OBI_STR' :
                 data_type = OBI_STR
             elif type == 'OBI_SEQ' :
@@ -386,22 +373,16 @@ cdef class OBIView :
             else :
                 raise Exception("Invalid provided data type")
         
-        if (obi_view_add_column(self.pointer, column_name_b, version_number,     # should return pointer on column?
+        if (obi_view_add_column(self.pointer, column_name_b, version_number,     # TODO should return pointer on column?
                                 data_type, nb_lines, nb_elements_per_line, 
                                 elements_names_b, str2bytes(indexer_name), 
                                 str2bytes(comments), create) < 0) :
             raise Exception("Problem adding a column in a view")
         
-        # Store the column pointer
-        column_pp = obi_view_get_pointer_on_column_in_view(self.pointer, column_name_b)
-        if column_pp == NULL :
-            raise Exception("Problem getting a column in a view")
-        
-        col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO
-        (self.columns_pp)[column_name] = col_capsule
+        # Get the column pointer
+        column_p = obi_view_get_column(self.pointer, column_name_b)
         
         # Open and store the subclass
-        column_p = column_pp[0]     # TODO ugly cython dereferencing
         subclass = OBIDMS_column.get_subclass_type(column_p)
         (self.columns)[column_name] = subclass(self, column_name)
                 
@@ -415,12 +396,12 @@ cdef class OBIView :
         # iter on each line of all columns
         
         # Declarations
-        cdef index_t lines_used
-        cdef index_t line_nb
-        cdef OBIView_line line  # TODO for NUC SEQS View
+        cdef index_t      lines_used
+        cdef index_t      line_nb
+        cdef OBIView_line line          # TODO Check that this works for NUC SEQ views
         
-        # Yield each line TODO line class
-        lines_used = (self.pointer).line_count
+        # Yield each line
+        lines_used = self.pointer.infos.line_count
         
         for line_nb in range(lines_used) :
             line = self[line_nb]
@@ -430,7 +411,7 @@ cdef class OBIView :
     def __getitem__(self, object item) :
         if type(item) == str :
             return (self.columns)[item]
-        elif type(item) == int :     # TODO int?
+        elif type(item) == int :
             return OBIView_line(self, item)
 
 
@@ -443,7 +424,7 @@ cdef class OBIView :
         cdef index_t* line_selection_p
         line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
         for i in range(len(line_selection)) :
-            line_selection_p[i] = line_selection[i]     # TODO type problem?
+            line_selection_p[i] = line_selection[i]
         line_selection_p[len(line_selection)] = -1
         if obi_select_lines(self.pointer, line_selection_p) < 0 :
             raise Exception("Problem selecting a list of lines")
@@ -467,22 +448,21 @@ cdef class OBIView_NUC_SEQS(OBIView):
     
     def __init__(self, OBIDMS dms, str view_name, bint new=False, object view_to_clone=None, list line_selection=None, str comments=""): 
         
-        cdef Obiview_p view = NULL
-        cdef int i
-        cdef list col_list
-        cdef str col_name
-        cdef OBIDMS_column column
-        cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
+        cdef Obiview_p              view = NULL
+        cdef int                    i
+        cdef list                   col_list
+        cdef str                    col_name
+        cdef OBIDMS_column          column
+        cdef OBIDMS_column_p        column_p
         cdef OBIDMS_column_header_p header
-        cdef index_t* line_selection_p
+        cdef index_t*               line_selection_p
                 
         self.dms = dms
         
         if line_selection is not None :
             line_selection_p = <index_t*> malloc((len(line_selection) + 1) * sizeof(index_t))
             for i in range(len(line_selection)) :
-                line_selection_p[i] = line_selection[i]     # TODO type problem?
+                line_selection_p[i] = line_selection[i]
             line_selection_p[len(line_selection)] = -1
         else :
             line_selection_p = NULL
@@ -505,76 +485,29 @@ cdef class OBIView_NUC_SEQS(OBIView):
             raise Exception("Error creating/opening view")
 
         self.pointer = view
-        self.name = bytes2str(view.name)
-        self.comments = bytes2str(view.comments)
+        self.name = bytes2str(view.infos.name)
+        self.comments = bytes2str(view.infos.comments)
         
-        # go through columns to build list and open python object (TODO make separate function?)
+        # Go through columns to build list of corresponding python instances
         self.columns = {}
-        self.columns_pp = {}
-                
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)   # TODO discuss
-            (self.columns_pp)[col_name] = col_capsule
-            
-            subclass = OBIDMS_column.get_subclass_type(column_p)
+        for i in range(view.infos.column_count) :
+            column_p               = <OBIDMS_column_p> (view.columns)[i]
+            header                 = (column_p).header
+            col_name               = bytes2str(header.name)         
+            subclass               = OBIDMS_column.get_subclass_type(column_p)
             self.columns[col_name] = subclass(self, col_name)
-            
-            i+=1
         
         self.ids = self.columns[bytes2str(ID_COLUMN)]
         self.sequences = self.columns[bytes2str(NUC_SEQUENCE_COLUMN)]
-        self.descriptions = self.columns[bytes2str(DESCRIPTION_COLUMN)]
-
-
-    cpdef delete_column(self, str column_name) :
-        
-        cdef int i
-        cdef Obiview_p view
-        cdef OBIDMS_column column
-        cdef OBIDMS_column_p column_p
-        cdef OBIDMS_column_p* column_pp
-        cdef OBIDMS_column_header_p header
-        cdef str column_n
- 
-        if ((column_name == bytes2str(ID_COLUMN)) or (column_name == bytes2str(NUC_SEQUENCE_COLUMN)) or (column_name == bytes2str(DESCRIPTION_COLUMN))) :
-            raise Exception("Can't delete an obligatory column from a NUC_SEQS view")
-        
-        view = self.pointer
- 
-        if obi_view_delete_column(view, str2bytes(column_name)) < 0 :
-            raise Exception("Problem deleting a column from a view")
-        
-        # Update the dictionaries of column pointers and column objects, and update pointers in column objects (make function?):
-        (self.columns).pop(column_name)
-        (self.columns_pp).pop(column_name)
-        
-        i = 0
-        while i < view.column_count :
-            column_pp = <OBIDMS_column_p*> ((view.columns)+i)
-            column_p = <OBIDMS_column_p> (view.columns)[i]
-            header = (column_p).header
-            col_name = bytes2str(header.name)
-            col_capsule = PyCapsule_New(column_pp, NULL, NULL)
-            (self.columns_pp)[col_name] = col_capsule
-            i+=1
-
-        for column_n in self.columns :
-            (self.columns[column_n]).update_pointer()
-
+        self.definitions = self.columns[bytes2str(DEFINITION_COLUMN)]
+        self.qualities = self.columns[bytes2str(QUALITY_COLUMN)]
             
     def __getitem__(self, object item) :
         if type(item) == str :
             return (self.columns)[item]
-        elif type(item) == int :     # TODO int?
+        elif type(item) == int :
             return OBI_Nuc_Seq_Stored(self, item)
     
-    
     def __setitem__(self, index_t line_idx, OBI_Nuc_Seq sequence_obj) :
         for key in sequence_obj :
             self[line_idx][key] = sequence_obj[key]
@@ -593,6 +526,7 @@ cdef class OBIView_line :
     
     def __setitem__(self, str column_name, object value):
         # TODO detect multiple elements (dict type)? put somewhere else? but more risky (in get)
+        # TODO OBI_QUAL ?
         cdef type value_type
         cdef str value_obitype
         if column_name not in self.view :
@@ -606,7 +540,7 @@ cdef class OBIView_line :
             elif value_type == bool :
                 value_obitype = 'OBI_BOOL'            
             elif value_type == str :
-                if only_ATGC(str2bytes(value)) :    # TODO
+                if only_ATGC(str2bytes(value)) :    # TODO detect IUPAC?
                     value_obitype = 'OBI_SEQ'
                 elif len(value) == 1 :
                     value_obitype = 'OBI_CHAR'
@@ -668,7 +602,7 @@ cdef class OBIDMS :
             view_class = OBIView_NUC_SEQS
         else :
             view_class = OBIView
-        
+                
         return view_class(self, view_name)
     
 
@@ -686,54 +620,84 @@ cdef class OBIDMS :
 
 
     cpdef dict read_view_infos(self, str view_name) :
-        all_views = self.read_views()
-        return all_views[view_name]
+        
+        cdef Obiview_infos_p      view_infos_p
+        cdef dict                 view_infos_d
+        cdef Column_reference_p   column_refs
+        cdef int                  i, j
+        cdef str                  column_name
+
+        view_infos_p = obi_view_map_file(self.pointer, str2bytes(view_name))
+        view_infos_d = {}
+        view_infos_d["name"] = bytes2str(view_infos_p.name)
+        view_infos_d["comments"] = bytes2str(view_infos_p.comments)
+        view_infos_d["view_type"] = bytes2str(view_infos_p.view_type)
+        view_infos_d["column_count"] = <int> view_infos_p.column_count
+        view_infos_d["line_count"] = <int> view_infos_p.line_count
+        view_infos_d["created_from"] = bytes2str(view_infos_p.created_from)
+        view_infos_d["creation_date"] = bytes2str(obi_format_date(view_infos_p.creation_date))
+        if (view_infos_p.all_lines) :
+            view_infos_d["line_selection"] = None
+        else :
+            view_infos_d["line_selection"] = {}
+            view_infos_d["line_selection"]["column_name"] = bytes2str((view_infos_p.line_selection).column_name)
+            view_infos_d["line_selection"]["version"] = <int> (view_infos_p.line_selection).version
+        view_infos_d["column_references"] = {}
+        column_refs = view_infos_p.column_references
+        for j in range(view_infos_d["column_count"]) :
+            column_name = bytes2str((column_refs[j]).column_name)
+            view_infos_d["column_references"][column_name] = {}
+            view_infos_d["column_references"][column_name]["version"] = column_refs[j].version
+        
+        obi_view_unmap_file(self.pointer, view_infos_p)
+
+        return view_infos_d
 
 
-    cpdef dict read_views(self) :    # TODO function that prints the dic nicely and function that prints 1 view. Add column type in col ref
-        
-        cdef Obiviews_infos_all_p all_views_p
-        cdef Obiview_infos_p view_p
-        cdef Column_reference_p column_refs
-        cdef int nb_views
-        cdef int i, j
-        cdef str view_name
-        cdef str column_name
-        cdef dict views
-        cdef bytes name_b
-        
-        views = {}
-        all_views_p = obi_read_views(self.pointer)
-        if all_views_p == NULL :
-            raise Exception("No views to read")
-        nb_views = <int> (all_views_p.header).view_count
-        for i in range(nb_views) :
-            view_p = (<Obiview_infos_p> (all_views_p.view_infos)) + i
-            view_name = bytes2str(view_p.name)
-            views[view_name] = {}
-            views[view_name]["comments"] = bytes2str(view_p.comments)
-            views[view_name]["view_type"] = bytes2str(view_p.view_type)
-            views[view_name]["column_count"] = <int> view_p.column_count
-            views[view_name]["line_count"] = <int> view_p.line_count
-            views[view_name]["view_number"] = <int> view_p.view_number
-            views[view_name]["created_from"] = bytes2str(view_p.created_from)
-            views[view_name]["creation_date"] = bytes2str(obi_column_format_date(view_p.creation_date))     # TODO move this function in utils or somethings
-            if (view_p.all_lines) :
-                views[view_name]["line_selection"] = None
-            else :
-                views[view_name]["line_selection"] = {}
-                views[view_name]["line_selection"]["column_name"] = bytes2str((view_p.line_selection).column_name)
-                views[view_name]["line_selection"]["version"] = <int> (view_p.line_selection).version
-            views[view_name]["column_references"] = {}
-            column_refs = view_p.column_references
-            for j in range(views[view_name]["column_count"]) :
-                column_name = bytes2str((column_refs[j]).column_name)
-                views[view_name]["column_references"][column_name] = {}
-                views[view_name]["column_references"][column_name]["version"] = column_refs[j].version
-        
-        obi_unmap_read_views(all_views_p);
-        
-        return views
+#     cpdef dict read_views(self) :    # TODO function that prints the dic nicely and function that prints 1 view nicely. Add column type in col ref
+#         
+#         cdef Obiviews_infos_all_p all_views_p
+#         cdef Obiview_infos_p      view_p
+#         cdef Column_reference_p   column_refs
+#         cdef int                  nb_views
+#         cdef int                  i, j
+#         cdef str                  view_name
+#         cdef str                  column_name
+#         cdef dict                 views
+#         cdef bytes                name_b
+#         
+#         views = {}
+#         all_views_p = obi_read_view_infos(self.pointer)
+#         if all_views_p == NULL :
+#             raise Exception("No views to read")
+#         nb_views = <int> (all_views_p.header).view_count
+#         for i in range(nb_views) :
+#             view_p = (<Obiview_infos_p> (all_views_p.view_infos)) + i
+#             view_name = bytes2str(view_p.name)
+#             views[view_name] = {}
+#             views[view_name]["comments"] = bytes2str(view_p.comments)
+#             views[view_name]["view_type"] = bytes2str(view_p.view_type)
+#             views[view_name]["column_count"] = <int> view_p.column_count
+#             views[view_name]["line_count"] = <int> view_p.line_count
+#             views[view_name]["view_number"] = <int> view_p.view_number
+#             views[view_name]["created_from"] = bytes2str(view_p.created_from)
+#             views[view_name]["creation_date"] = bytes2str(obi_format_date(view_p.creation_date))
+#             if (view_p.all_lines) :
+#                 views[view_name]["line_selection"] = None
+#             else :
+#                 views[view_name]["line_selection"] = {}
+#                 views[view_name]["line_selection"]["column_name"] = bytes2str((view_p.line_selection).column_name)
+#                 views[view_name]["line_selection"]["version"] = <int> (view_p.line_selection).version
+#             views[view_name]["column_references"] = {}
+#             column_refs = view_p.column_references
+#             for j in range(views[view_name]["column_count"]) :
+#                 column_name = bytes2str((column_refs[j]).column_name)
+#                 views[view_name]["column_references"][column_name] = {}
+#                 views[view_name]["column_references"][column_name]["version"] = column_refs[j].version
+#         
+#         obi_close_view_infos(all_views_p);
+#         
+#         return views
     
     
     

python/obitools3/obidms/_obidmscolumn_bool.cfiles

@@ -1,5 +1,3 @@
-../../../src/obidmscolumn_bool.c
-../../../src/obidmscolumn_bool.h
 ../../../src/bloom.h
 ../../../src/bloom.c
 ../../../src/char_str_indexer.h
@@ -10,8 +8,10 @@
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -23,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -37,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_bool.pyx

@@ -17,7 +17,7 @@ cdef class OBIDMS_column_bool(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obibool_t value
         cdef object result
-        value = obi_column_get_obibool_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        value = obi_column_get_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIBool_NA :
@@ -29,7 +29,7 @@ cdef class OBIDMS_column_bool(OBIDMS_column):
     cpdef set_line(self, index_t line_nb, object value):
         if value is None :
             value = OBIBool_NA
-        if obi_column_set_obibool_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, <obibool_t> value) < 0:
+        if obi_column_set_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obibool_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
 
@@ -38,7 +38,7 @@ cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obibool_t value
         cdef object result
-        value = obi_column_get_obibool_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        value = obi_column_get_obibool_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIBool_NA :
@@ -56,7 +56,7 @@ cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obibool_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obibool_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
             if value == OBIBool_NA :
@@ -73,5 +73,5 @@ cdef class OBIDMS_column_multi_elts_bool(OBIDMS_column_multi_elts):
     cpdef set_item(self, index_t line_nb, str element_name, object value):
         if value is None :
             value = OBIBool_NA
-        if obi_column_set_obibool_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), <obibool_t> value) < 0:
+        if obi_column_set_obibool_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obibool_t> value) < 0:
             raise Exception("Problem setting a value in a column")

python/obitools3/obidms/_obidmscolumn_char.cfiles

@@ -1,5 +1,3 @@
-../../../src/obidmscolumn_char.c
-../../../src/obidmscolumn_char.h
 ../../../src/bloom.h
 ../../../src/bloom.c
 ../../../src/char_str_indexer.h
@@ -10,8 +8,10 @@
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -23,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -37,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_char.pyx

@@ -15,7 +15,7 @@ cdef class OBIDMS_column_char(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obichar_t value
         cdef object result
-        value = obi_column_get_obichar_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        value = obi_column_get_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIChar_NA :
@@ -27,7 +27,7 @@ cdef class OBIDMS_column_char(OBIDMS_column):
     cpdef set_line(self, index_t line_nb, object value):
         if value is None :
             value = OBIChar_NA
-        if obi_column_set_obichar_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, str2bytes(value)[0]) < 0:
+        if obi_column_set_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)[0]) < 0:
             raise Exception("Problem setting a value in a column")
 
 
@@ -36,7 +36,7 @@ cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obichar_t value
         cdef object result
-        value = obi_column_get_obichar_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        value = obi_column_get_obichar_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIChar_NA :
@@ -54,7 +54,7 @@ cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obichar_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obichar_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
             if value == OBIChar_NA :
@@ -71,6 +71,6 @@ cdef class OBIDMS_column_multi_elts_char(OBIDMS_column_multi_elts):
     cpdef set_item(self, index_t line_nb, str element_name, object value):
         if value is None :
             value = OBIChar_NA
-        if obi_column_set_obichar_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)[0]) < 0:
+        if obi_column_set_obichar_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)[0]) < 0:
             raise Exception("Problem setting a value in a column")
 

python/obitools3/obidms/_obidmscolumn_float.cfiles

@@ -1,17 +1,17 @@
-../../../src/obidmscolumn_float.c
-../../../src/obidmscolumn_float.h
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
 ../../../src/bloom.h
 ../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
 ../../../src/crc64.h
 ../../../src/crc64.c
 ../../../src/dna_seq_indexer.h
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -23,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -37,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_float.pyx

@@ -15,7 +15,7 @@ cdef class OBIDMS_column_float(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obifloat_t value
         cdef object result
-        value = obi_column_get_obifloat_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        value = obi_column_get_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIFloat_NA :
@@ -27,7 +27,7 @@ cdef class OBIDMS_column_float(OBIDMS_column):
     cpdef set_line(self, index_t line_nb, object value):
         if value is None :
             value = OBIFloat_NA
-        if obi_column_set_obifloat_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, <obifloat_t> value) < 0:
+        if obi_column_set_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obifloat_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
 
@@ -36,7 +36,7 @@ cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obifloat_t value
         cdef object result
-        value = obi_column_get_obifloat_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        value = obi_column_get_obifloat_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIFloat_NA :
@@ -54,7 +54,7 @@ cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obifloat_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obifloat_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
             if value == OBIFloat_NA :
@@ -71,6 +71,6 @@ cdef class OBIDMS_column_multi_elts_float(OBIDMS_column_multi_elts):
     cpdef set_item(self, index_t line_nb, str element_name, object value):
         if value is None :
             value = OBIFloat_NA
-        if obi_column_set_obifloat_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), <obifloat_t> value) < 0:
+        if obi_column_set_obifloat_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obifloat_t> value) < 0:
             raise Exception("Problem setting a value in a column")
  

python/obitools3/obidms/_obidmscolumn_int.cfiles

@@ -1,17 +1,17 @@
-../../../src/obidmscolumn_int.c
-../../../src/obidmscolumn_int.h
-../../../src/char_str_indexer.h
-../../../src/char_str_indexer.c
 ../../../src/bloom.h
 ../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
 ../../../src/crc64.h
 ../../../src/crc64.c
 ../../../src/dna_seq_indexer.h
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -23,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -37,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_int.pyx

@@ -17,7 +17,7 @@ cdef class OBIDMS_column_int(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef obiint_t value
         cdef object result
-        value = obi_column_get_obiint_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        value = obi_column_get_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
         if value == OBIInt_NA :
@@ -29,7 +29,7 @@ cdef class OBIDMS_column_int(OBIDMS_column):
     cpdef set_line(self, index_t line_nb, object value):
         if value is None :
             value = OBIInt_NA
-        if obi_column_set_obiint_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, <obiint_t> value) < 0:
+        if obi_column_set_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, <obiint_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 
 
@@ -38,7 +38,7 @@ cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef obiint_t value
         cdef object result
-        value = obi_column_get_obiint_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        value = obi_column_get_obiint_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
         if value == OBIInt_NA :
@@ -56,7 +56,7 @@ cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obiint_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obiint_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
             if value == OBIInt_NA :
@@ -73,6 +73,6 @@ cdef class OBIDMS_column_multi_elts_int(OBIDMS_column_multi_elts):
     cpdef set_item(self, index_t line_nb, str element_name, object value):
         if value is None :
             value = OBIInt_NA
-        if obi_column_set_obiint_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), <obiint_t> value) < 0:
+        if obi_column_set_obiint_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), <obiint_t> value) < 0:
             raise Exception("Problem setting a value in a column")
 

python/obitools3/obidms/_obidmscolumn_qual.cfiles

@@ -0,0 +1,59 @@
+../../../src/bloom.h
+../../../src/bloom.c
+../../../src/char_str_indexer.h
+../../../src/char_str_indexer.c
+../../../src/crc64.h
+../../../src/crc64.c
+../../../src/dna_seq_indexer.h
+../../../src/dna_seq_indexer.c
+../../../src/encode.h
+../../../src/encode.c
+../../../src/murmurhash2.h
+../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
+../../../src/obiavl.h
+../../../src/obiavl.c
+../../../src/obiblob_indexer.h
+../../../src/obiblob_indexer.c
+../../../src/obiblob.h
+../../../src/obiblob.c
+../../../src/obidebug.h
+../../../src/obidms_taxonomy.h
+../../../src/obidms_taxonomy.c
+../../../src/obidms.h
+../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
+../../../src/obidmscolumn_idx.h
+../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
+../../../src/obidmscolumn.h
+../../../src/obidmscolumn.c
+../../../src/obidmscolumndir.h
+../../../src/obidmscolumndir.c
+../../../src/obierrno.h
+../../../src/obierrno.c
+../../../src/obilittlebigman.h
+../../../src/obilittlebigman.c
+../../../src/obitypes.h
+../../../src/obitypes.c
+../../../src/obiview.h
+../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
+../../../src/utils.h
+../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_qual.pxd

@@ -0,0 +1,20 @@
+#cython: language_level=3
+
+from .capi.obitypes cimport index_t
+from ._obidms       cimport OBIDMS_column , OBIDMS_column_multi_elts
+
+
+cdef class OBIDMS_column_qual(OBIDMS_column):
+    cpdef object get_line(self, index_t line_nb)
+    cpdef object get_str_line(self, index_t line_nb)
+    cpdef set_line(self, index_t line_nb, object value)
+    cpdef set_str_line(self, index_t line_nb, object value)
+
+
+cdef class OBIDMS_column_multi_elts_qual(OBIDMS_column_multi_elts):
+    cpdef object get_item(self, index_t line_nb, str element_name)
+    cpdef object get_str_item(self, index_t line_nb, str element_name)
+    cpdef object get_line(self, index_t line_nb)
+    cpdef object get_str_line(self, index_t line_nb)
+    cpdef set_item(self, index_t line_nb, str element_name, object value)
+    cpdef set_str_item(self, index_t line_nb, str element_name, object value)

python/obitools3/obidms/_obidmscolumn_qual.pyx

@@ -0,0 +1,184 @@
+#cython: language_level=3
+ 
+from .capi.obiview      cimport obi_column_get_obiqual_char_with_elt_name_in_view, \
+                                obi_column_get_obiqual_char_with_elt_idx_in_view, \
+                                obi_column_set_obiqual_char_with_elt_name_in_view, \
+                                obi_column_set_obiqual_char_with_elt_idx_in_view, \
+                                obi_column_get_obiqual_int_with_elt_name_in_view, \
+                                obi_column_get_obiqual_int_with_elt_idx_in_view, \
+                                obi_column_set_obiqual_int_with_elt_name_in_view, \
+                                obi_column_set_obiqual_int_with_elt_idx_in_view
+                                
+from .capi.obierrno     cimport obi_errno
+from .capi.obitypes     cimport OBIQual_char_NA, OBIQual_int_NA, const_char_p
+
+from ._obidms           cimport OBIView
+
+from obitools3.utils cimport str2bytes, bytes2str
+
+from libc.stdlib cimport free
+from libc.string cimport strcmp
+from libc.stdint cimport uint8_t
+from libc.stdlib cimport malloc
+
+
+cdef class OBIDMS_column_qual(OBIDMS_column):
+
+    cpdef object get_line(self, index_t line_nb):
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          result
+        cdef int             i
+        value = obi_column_get_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, &value_length)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIQual_int_NA :
+            result = None
+        else :
+            result = []
+            for i in range(value_length) :
+                result.append(<int>value[i])            
+        return result
+    
+    cpdef object get_str_line(self, index_t line_nb):
+        cdef char*  value
+        cdef object result
+        cdef int    i
+        value = obi_column_get_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
+        if obi_errno > 0 :
+            raise IndexError(line_nb)
+        if value == OBIQual_char_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef set_line(self, index_t line_nb, object value):
+        cdef uint8_t* value_b
+        cdef int      value_length
+        if value is None :
+            if obi_column_set_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIQual_int_NA, 0) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            value_length = len(value)
+            value_b = <uint8_t*> malloc(value_length * sizeof(uint8_t))
+            for i in range(value_length) :
+                value_b[i] = <uint8_t>value[i]
+            if obi_column_set_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, value_b, value_length) < 0:
+                raise Exception("Problem setting a value in a column")
+            free(value_b)
+
+    cpdef set_str_line(self, index_t line_nb, object value):
+        if value is None :
+            if obi_column_set_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIQual_char_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
+
+
+cdef class OBIDMS_column_multi_elts_qual(OBIDMS_column_multi_elts):
+ 
+    cpdef object get_item(self, index_t line_nb, str element_name):
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          result
+        cdef int             i
+        value = obi_column_get_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), &value_length)
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBIQual_int_NA :
+            result = None
+        else :
+            result = []
+            for i in range(value_length) :
+                result.append(<int>value[i])            
+        return result
+ 
+    cpdef object get_str_item(self, index_t line_nb, str element_name):
+        cdef char*  value
+        cdef object result
+        value = obi_column_get_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
+        if obi_errno > 0 :
+            raise IndexError(line_nb, element_name)
+        if value == OBIQual_char_NA :
+            result = None
+        else :
+            result = bytes2str(value)
+            free(value)
+        return result
+
+    cpdef object get_line(self, index_t line_nb) :
+        cdef const uint8_t*  value
+        cdef int             value_length
+        cdef object          value_in_result
+        cdef dict            result
+        cdef index_t         i
+        cdef int             j
+        cdef bint            all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiqual_int_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i, &value_length)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIQual_int_NA :
+                value_in_result = None
+            else :
+                value_in_result = []
+                for j in range(value_length) :
+                    value_in_result.append(<int>value[j])            
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+ 
+    cpdef object get_str_line(self, index_t line_nb) :
+        cdef char*   value
+        cdef object  value_in_result
+        cdef dict    result
+        cdef index_t i
+        cdef bint    all_NA
+        result = {}
+        all_NA = True
+        for i in range(self.nb_elements_per_line) :
+            value = obi_column_get_obiqual_char_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
+            if obi_errno > 0 :
+                raise IndexError(line_nb)
+            if value == OBIQual_char_NA :
+                value_in_result = None
+            else :
+                value_in_result = bytes2str(value)
+                free(value)
+            result[self.elements_names[i]] = value_in_result
+            if all_NA and (value_in_result is not None) :
+                all_NA = False
+        if all_NA :
+            result = None
+        return result
+
+    cpdef set_item(self, index_t line_nb, str element_name, object value):
+        cdef uint8_t* value_b
+        cdef int      value_length
+        if value is None :
+            if obi_column_set_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), OBIQual_int_NA, 0) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            value_length = len(value)
+            value_b = <uint8_t*> malloc(value_length * sizeof(uint8_t))
+            for i in range(value_length) :
+                value_b[i] = <uint8_t>value[i]
+            if obi_column_set_obiqual_int_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b, value_length) < 0:
+                raise Exception("Problem setting a value in a column")
+            free(value_b)
+
+    cpdef set_str_item(self, index_t line_nb, str element_name, object value):
+        if value is None :
+            if obi_column_set_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), OBIQual_char_NA) < 0:
+                raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiqual_char_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")

python/obitools3/obidms/_obidmscolumn_seq.cfiles

@@ -1,5 +1,3 @@
-../../../src/obidmscolumn_seq.c
-../../../src/obidmscolumn_seq.h
 ../../../src/bloom.h
 ../../../src/bloom.c
 ../../../src/char_str_indexer.h
@@ -10,8 +8,10 @@
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -23,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -37,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_seq.pxd

@@ -1,12 +1,24 @@
 #cython: language_level=3
 
 from .capi.obitypes cimport index_t
-from ._obidms       cimport OBIDMS_column, OBIDMS_column_multi_elts
+from ._obidms       cimport OBIView, OBIDMS_column, OBIDMS_column_multi_elts
 
 
 cdef class OBIDMS_column_seq(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb)
     cpdef set_line(self, index_t line_nb, object value)
+    
+    # TO DISCUSS : 
+    # I'am not sure that this method has to be declared here
+    # Alignment must be declared outside of the sequence object
+    cpdef align(self, 
+                OBIView score_view,
+                OBIDMS_column score_column, 
+                double threshold = *, 
+                bint normalize = *, 
+                int reference = *,
+                bint similarity_mode = *)
+
 
 cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name)

python/obitools3/obidms/_obidmscolumn_seq.pyx

@@ -4,13 +4,15 @@ from .capi.obiview      cimport obi_column_get_obiseq_with_elt_name_in_view, \
                                 obi_column_get_obiseq_with_elt_idx_in_view, \
                                 obi_column_set_obiseq_with_elt_name_in_view, \
                                 obi_column_set_obiseq_with_elt_idx_in_view
+from .capi.obialign     cimport obi_align_one_column
 from .capi.obierrno     cimport obi_errno
 from .capi.obitypes     cimport OBISeq_NA, const_char_p
 
+from ._obidms           cimport OBIView
+
 from obitools3.utils cimport str2bytes, bytes2str
 
 from libc.stdlib cimport free
-from libc.string cimport strcmp
 
 
 cdef class OBIDMS_column_seq(OBIDMS_column):
@@ -18,41 +20,67 @@ cdef class OBIDMS_column_seq(OBIDMS_column):
     cpdef object get_line(self, index_t line_nb):
         cdef char*  value
         cdef object result
-        value = obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        value = obi_column_get_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
-        if strcmp(value, OBISeq_NA) == 0 :
+        if value == OBISeq_NA :
             result = None
         else :
-            result = bytes2str(value)
-            free(value)
+            try:
+                result = <bytes> value
+            finally:
+                free(value)
         return result
 
     cpdef set_line(self, index_t line_nb, object value):
         cdef bytes value_b
+
         if value is None :
             value_b = OBISeq_NA
-        else :
+        elif isinstance(value, bytes) :
+            value_b = value
+        elif isinstance(value, str) :
             value_b = str2bytes(value)
-        if obi_column_set_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, value_b) < 0:
-            raise Exception("Problem setting a value in a column")
+        else:
+            raise TypeError('Sequence value must be of type Bytes, Str or None')
 
+        if obi_column_set_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, value_b) < 0:
+            raise Exception("Problem setting a value in a column")
+        else :
+            if obi_column_set_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
+    
+    # TODO choose alignment type (lcs or other) with supplementary argument
+    cpdef align(self, 
+                OBIView score_view,
+                OBIDMS_column score_column, 
+                double threshold = 0.0, 
+                bint normalize = True, 
+                int reference = 0,      # TODO
+                bint similarity_mode = True):
+        if (obi_align_one_column(self.view.pointer, (self.pointer)[0], score_view.pointer, (score_column.pointer)[0], threshold, normalize, reference, similarity_mode) < 0) :
+            raise Exception("An error occurred while aligning sequences")
+
+        
 
 cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
 
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef char*  value
         cdef object result
-        value = obi_column_get_obiseq_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        value = obi_column_get_obiseq_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
-        if strcmp(value, OBISeq_NA) == 0 :
+        if value == OBISeq_NA :
             result = None
         else :
-            result = bytes2str(value)
-            free(value)
+            try:
+                result = <bytes> value
+            finally:
+                free(value)
         return result
 
+    
     cpdef object get_line(self, index_t line_nb) :
         cdef char*   value
         cdef object  value_in_result
@@ -62,14 +90,16 @@ cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obiseq_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obiseq_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
-            if strcmp(value, OBISeq_NA) == 0 :
+            if value == OBISeq_NA :
                 value_in_result = None
             else :
-                value_in_result = bytes2str(value)
-                free(value)    
+                try:
+                    value_in_result = <bytes> value
+                finally:
+                    free(value)    
             result[self.elements_names[i]] = value_in_result
             if all_NA and (value_in_result is not None) :
                 all_NA = False
@@ -79,10 +109,19 @@ cdef class OBIDMS_column_multi_elts_seq(OBIDMS_column_multi_elts):
 
     cpdef set_item(self, index_t line_nb, str element_name, object value):
         cdef bytes value_b
+        
         if value is None :
             value_b = OBISeq_NA
-        else :
+        elif isinstance(value, bytes) :
+            value_b = value
+        elif isinstance(value, str) :
             value_b = str2bytes(value)
-        if obi_column_set_obiseq_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
+        else:
+            raise TypeError('Sequence value must be of type Bytes, Str or None')
+        
+        if obi_column_set_obiseq_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
             raise Exception("Problem setting a value in a column")
     
+#    cpdef align(self, ):        # TODO
+#        raise Exception("Columns with multiple sequences per line can't be aligned")    # TODO discuss
+

python/obitools3/obidms/_obidmscolumn_str.cfiles

@@ -1,5 +1,3 @@
-../../../src/obidmscolumn_str.c
-../../../src/obidmscolumn_str.h
 ../../../src/bloom.h
 ../../../src/bloom.c
 ../../../src/char_str_indexer.h
@@ -10,8 +8,10 @@
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -23,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -37,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/_obidmscolumn_str.pyx

@@ -9,18 +9,16 @@ from .capi.obitypes     cimport OBIStr_NA, const_char_p
 
 from obitools3.utils cimport str2bytes, bytes2str
 
-from libc.string cimport strcmp
-
 
 cdef class OBIDMS_column_str(OBIDMS_column):
 
     cpdef object get_line(self, index_t line_nb):
         cdef const_char_p value
         cdef object       result
-        value = obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0)
+        value = obi_column_get_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0)
         if obi_errno > 0 :
             raise IndexError(line_nb)
-        if strcmp(value, OBIStr_NA) == 0 :
+        if value == OBIStr_NA :
             result = None
         else :
             result = bytes2str(value)
@@ -28,13 +26,12 @@ cdef class OBIDMS_column_str(OBIDMS_column):
         return result
 
     cpdef set_line(self, index_t line_nb, object value):
-        cdef bytes value_b
         if value is None :
-            value_b = OBIStr_NA
+            if obi_column_set_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, OBIStr_NA) < 0:
+                raise Exception("Problem setting a value in a column")
         else :
-            value_b = str2bytes(value)
-        if obi_column_set_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, 0, value_b) < 0:
-            raise Exception("Problem setting a value in a column")
+            if obi_column_set_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, 0, str2bytes(value)) < 0:
+                raise Exception("Problem setting a value in a column")
 
 
 cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
@@ -42,10 +39,10 @@ cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
     cpdef object get_item(self, index_t line_nb, str element_name):
         cdef const_char_p value
         cdef object       result
-        value = obi_column_get_obistr_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name))
+        value = obi_column_get_obistr_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name))
         if obi_errno > 0 :
             raise IndexError(line_nb, element_name)
-        if strcmp(value, OBIStr_NA) == 0 :
+        if value == OBIStr_NA :
             result = None
         else :
             result = bytes2str(value)
@@ -61,10 +58,10 @@ cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
         result = {}
         all_NA = True
         for i in range(self.nb_elements_per_line) :
-            value = obi_column_get_obistr_with_elt_idx_in_view(self.view, (self.pointer)[0], line_nb, i)
+            value = obi_column_get_obistr_with_elt_idx_in_view(self.view.pointer, (self.pointer)[0], line_nb, i)
             if obi_errno > 0 :
                 raise IndexError(line_nb)
-            if strcmp(value, OBIStr_NA) == 0 :
+            if value == OBIStr_NA :
                 value_in_result = None
             else :
                 value_in_result = bytes2str(value)
@@ -82,6 +79,6 @@ cdef class OBIDMS_column_multi_elts_str(OBIDMS_column_multi_elts):
             value_b = OBIStr_NA
         else :
             value_b = str2bytes(value)
-        if obi_column_set_obistr_with_elt_name_in_view(self.view, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
+        if obi_column_set_obistr_with_elt_name_in_view(self.view.pointer, (self.pointer)[0], line_nb, str2bytes(element_name), value_b) < 0:
             raise Exception("Problem setting a value in a column")
     

python/obitools3/obidms/_obiseq.cfiles

@@ -8,8 +8,10 @@
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -21,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -35,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/_obiseq.pxd

@@ -4,27 +4,35 @@ from ._obidms cimport OBIView_line
 
 
 cdef class OBI_Seq(dict) :
-    cdef str id
-    cdef str description
-    cdef str sequence
+    cdef object id
+    cdef object definition
+    cdef object sequence
 
-    cpdef set_id(self, str id)
-    cpdef get_id(self)
-    cpdef set_description(self, str description)
-    cpdef get_description(self)
-    cpdef get_sequence(self)
+    cpdef set_id(self, object id)
+    cpdef object get_id(self)
+    cpdef set_definition(self, object definition)
+    cpdef object get_definition(self)
+    cpdef object get_sequence(self)
 
 
 cdef class OBI_Nuc_Seq(OBI_Seq) :
-    #cpdef str reverse_complement(self)    
-    cpdef set_sequence(self, str sequence)
+    cdef object quality
+    
+    #cpdef object reverse_complement(self)    
+    cpdef set_sequence(self, object sequence)
+    cpdef set_quality(self, object quality)
+    cpdef object get_quality(self)
 
 
 cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
-    cpdef set_id(self, str id)
-    cpdef get_id(self)
-    cpdef set_description(self, str description)
-    cpdef get_description(self)
-    cpdef set_sequence(self, str sequence)
-    cpdef get_sequence(self)
-   # cpdef str reverse_complement(self)
+    cpdef set_id(self, object id)
+    cpdef object get_id(self)
+    cpdef set_definition(self, object definition)
+    cpdef object get_definition(self)
+    cpdef set_sequence(self, object sequence)
+    cpdef object get_sequence(self)
+    cpdef set_quality(self, object quality)
+    cpdef object get_quality(self)
+    cpdef object get_str_quality(self)
+
+   # cpdef object reverse_complement(self)

python/obitools3/obidms/_obiseq.pyx

@@ -4,30 +4,31 @@ from obitools3.utils cimport bytes2str, str2bytes
 
 from .capi.obiview cimport NUC_SEQUENCE_COLUMN, \
                            ID_COLUMN, \
-                           DESCRIPTION_COLUMN
+                           DEFINITION_COLUMN, \
+                           QUALITY_COLUMN
 
 cdef class OBI_Seq(dict) :
 
-    def __init__(self, str id, str seq, str description=None) :
+    def __init__(self, object id, object seq, object definition=None) :
         self.set_id(id)
         self.set_sequence(seq)
-        if description is not None :
-            self.set_description(description)
+        if definition is not None :
+            self.set_definition(definition)
         
-    cpdef set_id(self, str id) :
+    cpdef set_id(self, object id) :
         self.id = id
         self[bytes2str(ID_COLUMN)] = id
         
     cpdef get_id(self) :
         return self.id
             
-    cpdef set_description(self, str description) :
-        self.description = description
-        self[bytes2str(DESCRIPTION_COLUMN)] = description
-        
-    cpdef get_description(self) :
-        return self.description     # TODO no
+    cpdef set_definition(self, object definition) :
+        self.definition = definition
+        self[bytes2str(DEFINITION_COLUMN)] = definition
         
+    cpdef get_definition(self) :
+        return self.definition
+
     cpdef get_sequence(self) :
         return self.sequence
 
@@ -37,34 +38,55 @@ cdef class OBI_Seq(dict) :
 
 cdef class OBI_Nuc_Seq(OBI_Seq) :
     
-    cpdef set_sequence(self, str sequence) :
+    cpdef set_sequence(self, object sequence) :
         self.sequence = sequence
         self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
 
+    cpdef set_quality(self, object quality) :
+        self.quality = quality
+        self[bytes2str(QUALITY_COLUMN)] = quality
+
+    cpdef get_quality(self) :
+        return self.quality
+
 #    cpdef str reverse_complement(self) :    TODO in C ?
 #        pass
 
 
 cdef class OBI_Nuc_Seq_Stored(OBIView_line) :
 
-    cpdef set_id(self, str id) :
+    # TODO store the str version of column name macros
+
+    cpdef set_id(self, object id) :
         self[bytes2str(ID_COLUMN)] = id
         
-    cpdef get_id(self) :
+    cpdef object get_id(self) :
         return self[bytes2str(ID_COLUMN)]
             
-    cpdef set_description(self, str description) :
-        self[bytes2str(DESCRIPTION_COLUMN)] = description
+    cpdef set_definition(self, object definition) :
+        self[bytes2str(DEFINITION_COLUMN)] = definition
 
-    cpdef get_description(self) :
-        return self[bytes2str(DESCRIPTION_COLUMN)]
+    cpdef object get_definition(self) :
+        return self[bytes2str(DEFINITION_COLUMN)]
 
-    cpdef set_sequence(self, str sequence) :
+    cpdef set_sequence(self, object sequence) :
         self[bytes2str(NUC_SEQUENCE_COLUMN)] = sequence
 
-    cpdef get_sequence(self) :
+    cpdef object get_sequence(self) :
         return self[bytes2str(NUC_SEQUENCE_COLUMN)]
     
+    cpdef set_quality(self, object quality) :
+        if (type(quality) == list) or (quality is None) :
+            self[bytes2str(QUALITY_COLUMN)] = quality
+        else :  # Quality is in str form
+            (((self.view).columns)[bytes2str(QUALITY_COLUMN)]).set_str_line(self.index, quality)
+
+    cpdef object get_quality(self) :
+        return self[bytes2str(QUALITY_COLUMN)]
+
+    cpdef object get_str_quality(self) :
+        return ((self.view).columns)[bytes2str(QUALITY_COLUMN)].get_str_line(self.index)
+
 #    def __str__(self) :
 #        return self[bytes2str(NUC_SEQUENCE_COLUMN)]    # or not
     

python/obitools3/obidms/_obitaxo.cfiles

@@ -8,8 +8,10 @@
 ../../../src/dna_seq_indexer.c
 ../../../src/encode.h
 ../../../src/encode.c
-../../../src/MurmurHash2.h
+../../../src/murmurhash2.h
 ../../../src/murmurhash2.c
+../../../src/obi_align.h
+../../../src/obi_align.c
 ../../../src/obiavl.h
 ../../../src/obiavl.c
 ../../../src/obiblob_indexer.h
@@ -21,8 +23,22 @@
 ../../../src/obidms_taxonomy.c
 ../../../src/obidms.h
 ../../../src/obidms.c
+../../../src/obidmscolumn_bool.c
+../../../src/obidmscolumn_bool.h
+../../../src/obidmscolumn_char.c
+../../../src/obidmscolumn_char.h
+../../../src/obidmscolumn_float.c
+../../../src/obidmscolumn_float.h
 ../../../src/obidmscolumn_idx.h
 ../../../src/obidmscolumn_idx.c
+../../../src/obidmscolumn_int.c
+../../../src/obidmscolumn_int.h
+../../../src/obidmscolumn_qual.h
+../../../src/obidmscolumn_qual.c
+../../../src/obidmscolumn_seq.c
+../../../src/obidmscolumn_seq.h
+../../../src/obidmscolumn_str.c
+../../../src/obidmscolumn_str.h
 ../../../src/obidmscolumn.h
 ../../../src/obidmscolumn.c
 ../../../src/obidmscolumndir.h
@@ -35,5 +51,9 @@
 ../../../src/obitypes.c
 ../../../src/obiview.h
 ../../../src/obiview.c
+../../../src/sse_banded_LCS_alignment.h
+../../../src/sse_banded_LCS_alignment.c
+../../../src/uint8_indexer.h
+../../../src/uint8_indexer.c
 ../../../src/utils.h
 ../../../src/utils.c

python/obitools3/obidms/capi/obialign.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+from ..capi.obiview      cimport Obiview_p
+from ..capi.obidmscolumn cimport OBIDMS_column_p
+
+
+cdef extern from "obi_align.h" nogil:
+
+    int obi_align_one_column(Obiview_p seq_view, OBIDMS_column_p seq_column, Obiview_p score_view, OBIDMS_column_p score_column, double threshold, bint normalize, int reference, bint similarity_mode)
+

python/obitools3/obidms/capi/obidmscolumn.pxd

@@ -11,6 +11,8 @@ from ..capi.obitypes cimport const_char_p, \
                              index_t, \
                              time_t
 
+from libc.stdint cimport uint8_t
+
 
 cdef extern from "obidmscolumn.h" nogil:
     
@@ -70,8 +72,6 @@ cdef extern from "obidmscolumn.h" nogil:
                                                            obiversion_t version_number)
     
     int obi_close_header(OBIDMS_column_header_p header)
-        
-    char* obi_column_format_date(time_t date)
 
     int obi_select(OBIDMS_column_p line_selection_column, index_t line_to_grep)
 
@@ -196,3 +196,46 @@ cdef extern from "obidmscolumn_seq.h" nogil:
                                              index_t line_nb, 
                                              index_t element_idx)
 
+
+cdef extern from "obidmscolumn_qual.h" nogil:
+
+    int obi_column_set_obiqual_char_with_elt_name(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  const_char_p element_name, 
+                                                  const_char_p value)
+
+    int obi_column_set_obiqual_char_with_elt_idx(OBIDMS_column_p column, 
+                                                 index_t line_nb, 
+                                                 index_t element_idx, 
+                                                 const_char_p value)
+    
+    int obi_column_set_obiqual_int_with_elt_name(OBIDMS_column_p column, 
+                                                  index_t line_nb, 
+                                                  const_char_p element_name, 
+                                                  const uint8_t* value,
+                                                  int value_length)
+    
+    int obi_column_set_obiqual_int_with_elt_idx(OBIDMS_column_p column, 
+                                                index_t line_nb, 
+                                                index_t element_idx, 
+                                                const uint8_t* value,
+                                                int value_length)
+    
+    char* obi_column_get_obiqual_char_with_elt_name(OBIDMS_column_p column, 
+                                                    index_t line_nb, 
+                                                    const_char_p element_name)
+    
+    char* obi_column_get_obiqual_char_with_elt_idx(OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_name(OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            const_char_p element_name,
+                                                            int* value_length)
+    
+    const uint8_t* obi_column_get_obiqual_int_with_elt_idx(OBIDMS_column_p column, 
+                                                   index_t line_nb, 
+                                                   index_t element_idx,
+                                                   int* value_length)
+

python/obitools3/obidms/capi/obitypes.pxd

@@ -1,22 +1,19 @@
 #cython: language_level=3
 
 
-from libc.stdint cimport int32_t, int64_t
+from libc.stdint cimport int32_t, int64_t, uint8_t
+
 from posix.types cimport time_t
 
 
 cdef extern from *:
     ctypedef char* const_char_p "const char*"
 
+
 cdef extern from "encode.h" nogil:
     bint only_ATGC(const_char_p seq)
 
 
-cdef extern from "obidmscolumn.h" nogil:
-
-    ctypedef int32_t obiversion_t
-
-
 cdef extern from "obitypes.h" nogil:
 
     enum OBIType:
@@ -25,6 +22,7 @@ cdef extern from "obitypes.h" nogil:
         OBI_FLOAT,
         OBI_BOOL,
         OBI_CHAR,
+        OBI_QUAL,
         OBI_STR,
         OBI_SEQ,
         OBI_IDX
@@ -37,10 +35,12 @@ cdef extern from "obitypes.h" nogil:
     
     ctypedef OBIBool obibool_t
     ctypedef int32_t obiint_t
-    ctypedef double obifloat_t
-    ctypedef char obichar_t  
+    ctypedef double  obifloat_t
+    ctypedef char    obichar_t  
     ctypedef int64_t index_t
 
+    ctypedef int32_t obiversion_t
+
     extern obiint_t      OBIInt_NA
     extern index_t       OBIIdx_NA
     extern obifloat_t    OBIFloat_NA
@@ -48,5 +48,8 @@ cdef extern from "obitypes.h" nogil:
     extern obibool_t     OBIBool_NA
     extern const_char_p  OBISeq_NA
     extern const_char_p  OBIStr_NA
+    extern const_char_p  OBIQual_char_NA
+    extern uint8_t*      OBIQual_int_NA
 
     const_char_p name_data_type(int data_type)
+

python/obitools3/obidms/capi/obiutils.pxd

@@ -0,0 +1,12 @@
+#cython: language_level=3
+
+
+from posix.types cimport time_t
+
+from ..capi.obitypes cimport const_char_p
+
+
+cdef extern from "utils.h" nogil:
+
+    const_char_p obi_format_date(time_t date)
+

python/obitools3/obidms/capi/obiview.pxd

@@ -12,13 +12,17 @@ from .obitypes           cimport const_char_p, \
 from ..capi.obidms       cimport OBIDMS_p
 from ..capi.obidmscolumn cimport OBIDMS_column_p
 
+from libc.stdint cimport uint8_t
+
 
 cdef extern from "obiview.h" nogil:
 
     extern const_char_p VIEW_TYPE_NUC_SEQS
     extern const_char_p NUC_SEQUENCE_COLUMN
     extern const_char_p ID_COLUMN
-    extern const_char_p DESCRIPTION_COLUMN
+    extern const_char_p DEFINITION_COLUMN
+    extern const_char_p QUALITY_COLUMN
+
 
     struct Column_reference_t :
         const_char_p  column_name
@@ -27,50 +31,30 @@ cdef extern from "obiview.h" nogil:
     ctypedef Column_reference_t* Column_reference_p
 
 
-    struct Obiview_t :
-        OBIDMS_p            dms
-        const_char_p        name
-        OBIDMS_column_p     line_selection
-        OBIDMS_column_p     new_line_selection
-        OBIDMS_column_p     columns
-        bint                read_only
-        Column_reference_t  line_selection_reference
-        index_t             line_count
-        int                 column_count
-        const_char_p        comments
-
-    ctypedef Obiview_t* Obiview_p
-
-
     struct Obiview_infos_t :
-        int                 view_number
-        int                 column_count
-        index_t             line_count
+        time_t              creation_date
         const_char_p        name
         const_char_p        created_from
-        time_t              creation_date
+        const_char_p        view_type
         bint                all_lines
         Column_reference_t  line_selection
+        index_t             line_count
+        int                 column_count
         Column_reference_p  column_references
-        const_char_p        view_type
         const_char_p        comments
 
     ctypedef Obiview_infos_t* Obiview_infos_p
 
 
-    struct Obiviews_header_t :
-        size_t header_size
-        size_t views_size
-        int    view_count
-    
-    ctypedef Obiviews_header_t* Obiviews_header_p
+    struct Obiview_t :
+        Obiview_infos_p     infos
+        OBIDMS_p            dms
+        bint                read_only
+        OBIDMS_column_p     line_selection
+        OBIDMS_column_p     new_line_selection
+        OBIDMS_column_p     columns
 
-
-    struct Obiviews_infos_all_t :
-        Obiviews_header_p    header
-        Obiview_infos_p      view_infos
-
-    ctypedef Obiviews_infos_all_t* Obiviews_infos_all_p
+    ctypedef Obiview_t* Obiview_p
 
 
     Obiview_p obi_new_view_nuc_seqs(OBIDMS_p dms, const_char_p view_name, Obiview_p view_to_clone, index_t* line_selection, const_char_p comments)
@@ -81,12 +65,12 @@ cdef extern from "obiview.h" nogil:
 
     Obiview_p obi_new_view_nuc_seqs_cloned_from_name(OBIDMS_p dms, const_char_p view_name, const_char_p view_to_clone_name, index_t* line_selection, const_char_p comments)
               
+    Obiview_infos_p obi_view_map_file(OBIDMS_p dms, const char* view_name)
+
+    int obi_view_unmap_file(OBIDMS_p dms, Obiview_infos_p view_infos)
+
     Obiview_p obi_open_view(OBIDMS_p dms, const_char_p view_name)
     
-    Obiviews_infos_all_p obi_read_views(OBIDMS_p dms)
-
-    int obi_unmap_read_views(Obiviews_infos_all_p views)
-
     int obi_view_add_column(Obiview_p view,
                             const_char_p column_name,
                             obiversion_t version_number,
@@ -103,8 +87,6 @@ cdef extern from "obiview.h" nogil:
     int obi_select_line(Obiview_p view, index_t line_nb)
 
     int obi_select_lines(Obiview_p view, index_t* line_nbs)
-
-    OBIDMS_column_p obi_view_clone_column(Obiview_p view, const_char_p column_name)
         
     OBIDMS_column_p obi_view_get_column(Obiview_p view, const_char_p column_name)
 
@@ -115,7 +97,7 @@ cdef extern from "obiview.h" nogil:
     int obi_close_view(Obiview_p view)
     
     int obi_save_and_close_view(Obiview_p view)
-    
+
     int obi_column_set_obiint_with_elt_name_in_view(Obiview_p view,
                                                     OBIDMS_column_p column, 
                                                     index_t line_nb, 
@@ -204,6 +186,54 @@ cdef extern from "obiview.h" nogil:
                                                             index_t line_nb, 
                                                             index_t element_idx)
 
+    int obi_column_set_obiqual_char_with_elt_idx_in_view(Obiview_p view, 
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         index_t element_idx, 
+                                                         const char* value)
+
+    int obi_column_set_obiqual_int_with_elt_idx_in_view(Obiview_p view, 
+                                                        OBIDMS_column_p column, 
+                                                        index_t line_nb, 
+                                                        index_t element_idx, 
+                                                        const uint8_t* value, 
+                                                        int value_length)
+
+    char* obi_column_get_obiqual_char_with_elt_idx_in_view(Obiview_p view, 
+                                                           OBIDMS_column_p column, 
+                                                           index_t line_nb, 
+                                                           index_t element_idx)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_idx_in_view(Obiview_p view, 
+                                                                   OBIDMS_column_p column, 
+                                                                   index_t line_nb, 
+                                                                   index_t element_idx, 
+                                                                   int* value_length)
+
+    int obi_column_set_obiqual_char_with_elt_name_in_view(Obiview_p view, 
+                                                          OBIDMS_column_p column, 
+                                                          index_t line_nb, 
+                                                          const char* element_name, 
+                                                          const char* value)
+
+    int obi_column_set_obiqual_int_with_elt_name_in_view(Obiview_p view, 
+                                                         OBIDMS_column_p column, 
+                                                         index_t line_nb, 
+                                                         const char* element_name, 
+                                                         const uint8_t* value, 
+                                                         int value_length)
+
+    char* obi_column_get_obiqual_char_with_elt_name_in_view(Obiview_p view, 
+                                                            OBIDMS_column_p column, 
+                                                            index_t line_nb, 
+                                                            const char* element_name)
+
+    const uint8_t* obi_column_get_obiqual_int_with_elt_name_in_view(Obiview_p view, 
+                                                                    OBIDMS_column_p column, 
+                                                                    index_t line_nb, 
+                                                                    const char* element_name, 
+                                                                    int* value_length)
+
     int obi_column_set_obistr_with_elt_name_in_view(Obiview_p view,
                                                     OBIDMS_column_p column, 
                                                     index_t line_nb, 

python/obitools3/obigrep.py

@@ -0,0 +1,51 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obigrep.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+#     parser.add_argument('-k', '--key', dest='key', type=str,
+#                        help='Name of the key that should be considered')
+# 
+#     parser.add_argument('-c', '--comp', dest='comparison', type=int,
+#                        help='Comparison to be made: -1:< ; 0:== ; 1:>')
+# 
+#     parser.add_argument('-v', '--value', dest='value', type=object,
+#                        help='Value to be compared')
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    i = 0
+    for l in v :
+        if l['score'] > 350 : 
+            line_selec.append(i)
+        i+=1
+
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obigrep "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/obihead.py

@@ -0,0 +1,43 @@
+import sys
+import argparse
+
+from obitools3.obidms._obidms import OBIDMS
+
+if __name__ == '__main__': 
+    
+    parser = argparse.ArgumentParser(description='Pseudo obihead.')
+    
+    parser.add_argument('-V', '--view', dest='view', type=str,
+                       help='Name of the view that should be considered')    
+
+    parser.add_argument('-N', '--new_view', dest='new_view', type=str,
+                       help='Name of the new view that should be created')    
+    
+    parser.add_argument('-n', '--nb', dest='nb_lines', type=int,
+                       help='Number of lines that should be taken')    
+    
+
+    args = parser.parse_args()
+    
+    d = OBIDMS('tdms')
+    
+    #condition = 1
+    line_selec = []
+    
+    v = d.open_view(args.view)
+    
+    for i in range(0, args.nb_lines) :
+        line_selec.append(i)
+        
+    new_v = d.new_view(args.new_view, view_to_clone=v, line_selection=line_selec, view_type="NUC_SEQS_VIEW", comments="obihead "+str(args.nb_lines)+", "+args.view+" to "+args.new_view) #args.key+" "+str(args.comparison)+" "+str(args.value)+" "+)
+
+    print("\n")
+    print(new_v.__repr__())
+
+    v.save_and_close()
+    new_v.save_and_close()
+    
+    d.close()
+
+    print("\nDone.")
+    

python/obitools3/parsers/fasta.pxd

@@ -0,0 +1,9 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fasta.pyx

@@ -0,0 +1,55 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+
+def fastaIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    cdef list       s
+    cdef bytes      sequence
+    cdef bytes      quality
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    line = next(i)
+    
+    while True:
+        ident,tags,definition = parseHeader(line)
+        s = []
+        line = next(i)
+        
+        try:
+            while line[0]!='>':
+                s.append(str2bytes(line)[0:-1])
+                line = next(i)
+        except StopIteration:
+            pass
+        
+        sequence  = b"".join(s)
+        quality   = None
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+    
+        
+        

python/obitools3/parsers/fastq.pxd

@@ -0,0 +1,10 @@
+#cython: language_level=3
+
+from ..utils cimport str2bytes
+
+from .header cimport parseHeader
+from ..files.universalopener cimport uopen
+from ..files.linebuffer cimport LineBuffer
+
+
+    

python/obitools3/parsers/fastq.pyx

@@ -0,0 +1,40 @@
+#cython: language_level=3
+
+'''
+Created on 30 mars 2016
+
+@author: coissac
+'''
+
+def fastqIterator(lineiterator, int buffersize=100000000):
+    cdef LineBuffer lb
+    cdef str        ident
+    cdef str        definition
+    cdef dict       tags
+    cdef bytes      sequence
+    cdef bytes      quality
+    
+    if isinstance(lineiterator,(str,bytes)):
+        lineiterator=uopen(lineiterator)
+    
+    if isinstance(lineiterator, LineBuffer):
+        lb=lineiterator
+    else:
+        lb=LineBuffer(lineiterator,buffersize)
+        
+    i = iter(lb)
+    for line in i:
+        ident,tags,definition = parseHeader(line)
+        sequence  = str2bytes(next(i)[0:-1])
+        next(i)
+        quality   = str2bytes(next(i)[0:-1])
+
+        yield { "id"         : ident,
+                "definition" : definition,
+                "sequence"   : sequence,
+                "quality"    : quality,
+                "tags"       : tags,
+                "annotation" : {}
+              }
+        
+        

python/obitools3/parsers/header.pxd

@@ -0,0 +1,5 @@
+#cython: language_level=3
+
+cdef object __etag__(str x)
+
+cpdef tuple parseHeader(str header)

python/obitools3/parsers/header.pyx

@@ -0,0 +1,85 @@
+#cython: language_level=3
+
+'''
+Created on 25 mars 2016
+
+@author: coissac
+'''
+
+import re
+
+__ret__         = re.compile('''(([^ ]+)=('[^']*'|"[^"]*"|[^;]+); *)+?''')
+__re_int__      = re.compile("^[+-]?[0-9]+$")
+__re_float__    = re.compile("^[+-]?[0-9]+(\.[0-9]*)?([eE][+-]?[0-9]+)?$")
+__re_str__      = re.compile("""^"[^"]*"|'[^']*'$""")
+__re_dict__     = re.compile("""^\{\ *
+                                   (
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                   )?
+                                   (\ *,\ *
+                                       ("[^"]*"|'[^']*')
+                                        \ *:\ *
+                                       ([^,}]+|
+                                        "[^"]*"|
+                                        '[^']*'
+                                       )
+                                    )*\ *\}$""", re.VERBOSE)
+
+__re_val__ = re.compile("""(("[^"]*"|'[^']*') *: *([^,}]+|"[^"]*"|'[^']*') *[,}] *)""")
+
+cdef object __etag__(str x):
+    cdef list elements
+    cdef tuple i
+    
+    if __re_int__.match(x):
+        v=int(x)
+    elif __re_float__.match(x):
+        v=float(x)
+    elif __re_str__.match(x):
+        v=x[1:-1]
+    elif x=='None':
+        v=None
+    elif x=='False':
+        v=False
+    elif x=='True':
+        v=True
+    elif __re_dict__.match(x):
+        elements=__re_val__.findall(x)
+        v=dict([(i[1][1:-1],__etag__(i[2])) for i in elements])
+    else:
+        v=x
+    return v
+
+cpdef tuple parseHeader(str header):
+    cdef list m 
+    cdef dict tags 
+    cdef str definition
+    cdef str ident
+    cdef str second
+    
+    m=header[1:-1].split(maxsplit=1)
+   
+    ident=m[0]
+    
+    if len(m)==1:
+        tags={}
+        definition='' 
+    else:
+        second=m[1]
+        m = __ret__.findall(second)
+        
+        if m:
+            tags = dict([(a[1],__etag__(a[2])) for a in m])
+            definition = second.split(m[-1][0],1)[1].strip()
+        else:
+            tags = {}
+            definition = second.strip() 
+        
+    return ident,tags,definition
+    
+

python/obitools3/utils.pxd

@@ -1,5 +1,6 @@
 #cython: language_level=3
 
+
 cdef bytes str2bytes(str string)
 
 cdef str bytes2str(bytes string)

python/obitools3/utils.pyx

@@ -1,7 +1,26 @@
 #cython: language_level=3
 
+
 cdef bytes str2bytes(str string):
+    """
+    Short cut to convert ascii encoded python string (str) to bytes 
+    which can be easily converted to C-strings. 
+    
+        @param string: the python string to be converted.
+        @type string: str
+        @return a transcoded string
+        @rtype: bytes 
+    """
     return string.encode('ascii')
 
 cdef str bytes2str(bytes string):
-    return string.decode('ascii')
+    """
+    Short cut to convert bytes (C-strings) to ascii encoded python string (str).
+    
+        @param string: the binary (C-string) string to be converted.
+        @type string: bytes
+        @return an ascii transcoded string
+        @rtype: str 
+    """
+    return string.decode('ascii')
+

src/_sse.h

@@ -0,0 +1,961 @@
+#ifndef _SSE_H_
+#define _SSE_H_
+
+#include <string.h>
+
+#include <inttypes.h>
+#ifdef __SSE2__
+#include <xmmintrin.h>
+#else
+typedef long long __m128i __attribute__ ((__vector_size__ (16), __may_alias__));
+#endif /* __SSE2__ */
+
+#ifndef MAX
+#define MAX(x,y) (((x)>(y)) ? (x):(y))
+#define MIN(x,y) (((x)<(y)) ? (x):(y))
+#endif
+
+#define ALIGN __attribute__((aligned(16)))
+typedef __m128i vUInt8;
+typedef __m128i vInt8;
+
+typedef __m128i vUInt16;
+typedef __m128i vInt16;
+
+typedef __m128i vUInt64;
+
+typedef union
+{
+     __m128i i;
+     int64_t  s64[ 2];
+     int16_t  s16[ 8];
+     int8_t   s8 [16];
+     uint8_t  u8 [16];
+     uint16_t u16[8 ];
+	 uint32_t u32[4 ];
+     uint64_t u64[2 ];
+} um128;
+
+typedef union
+    {
+	    vUInt8 m;
+	    uint8_t c[16];
+    } uchar_v;
+
+typedef union
+	{
+		vUInt16 m;
+		uint16_t c[8];
+	} ushort_v;
+
+typedef union
+	{
+		vUInt64 m;
+		uint64_t c[2];
+	} uint64_v;
+
+
+#ifdef __SSE2__
+
+static inline int8_t _s2_extract_epi8(__m128i r, const int p)
+{
+#define ACTIONP(r,x) return _mm_extract_epi16(r,x) & 0xFF
+#define ACTIONI(r,x) return _mm_extract_epi16(r,x) >> 8
+	switch (p) {
+	case  0:  ACTIONP(r,0);
+	case  1:  ACTIONI(r,0);
+	case  2:  ACTIONP(r,1);
+	case  3:  ACTIONI(r,1);
+	case  4:  ACTIONP(r,2);
+	case  5:  ACTIONI(r,2);
+	case  6:  ACTIONP(r,3);
+	case  7:  ACTIONI(r,3);
+	case  8:  ACTIONP(r,4);
+	case  9:  ACTIONI(r,4);
+	case 10:  ACTIONP(r,5);
+	case 11:  ACTIONI(r,5);
+	case 12:  ACTIONP(r,6);
+	case 13:  ACTIONI(r,6);
+	case 14:  ACTIONP(r,7);
+	case 15:  ACTIONI(r,7);
+	}
+#undef ACTIONP
+#undef ACTIONI
+
+	return 0;
+}
+
+static inline __m128i _s2_max_epi8(__m128i a, __m128i b)
+{
+	__m128i mask  = _mm_cmpgt_epi8( a, b );
+	a = _mm_and_si128   (a,mask );
+	b = _mm_andnot_si128(mask,b);
+	return _mm_or_si128(a,b);
+}
+
+static inline __m128i _s2_min_epi8(__m128i a, __m128i b)
+{
+	__m128i mask  = _mm_cmplt_epi8( a, b );
+	a = _mm_and_si128   (a,mask );
+	b = _mm_andnot_si128(mask,b);
+	return _mm_or_si128(a,b);
+}
+
+static inline __m128i _s2_insert_epi8(__m128i r, int b, const int p)
+{
+#define ACTIONP(r,x) return _mm_insert_epi16(r,(_mm_extract_epi16(r,x) & 0xFF00) | (b & 0x00FF),x)
+#define ACTIONI(r,x) return _mm_insert_epi16(r,(_mm_extract_epi16(r,x) & 0x00FF) | ((b << 8)& 0xFF00),x)
+	switch (p) {
+	case  0:  ACTIONP(r,0);
+	case  1:  ACTIONI(r,0);
+	case  2:  ACTIONP(r,1);
+	case  3:  ACTIONI(r,1);
+	case  4:  ACTIONP(r,2);
+	case  5:  ACTIONI(r,2);
+	case  6:  ACTIONP(r,3);
+	case  7:  ACTIONI(r,3);
+	case  8:  ACTIONP(r,4);
+	case  9:  ACTIONI(r,4);
+	case 10:  ACTIONP(r,5);
+	case 11:  ACTIONI(r,5);
+	case 12:  ACTIONP(r,6);
+	case 13:  ACTIONI(r,6);
+	case 14:  ACTIONP(r,7);
+	case 15:  ACTIONI(r,7);
+	}
+#undef ACTIONP
+#undef ACTIONI
+
+	return _mm_setzero_si128();
+}
+
+// Fill a SSE Register with 16 time the same 8bits integer value
+#define _MM_SET1_EPI8(x)        _mm_set1_epi8(x)
+#define _MM_INSERT_EPI8(r,x,i)	_s2_insert_epi8((r),(x),(i))
+#define _MM_CMPEQ_EPI8(x,y)     _mm_cmpeq_epi8((x),(y))
+#define _MM_CMPGT_EPI8(x,y)     _mm_cmpgt_epi8((x),(y))
+#define _MM_CMPLT_EPI8(x,y)     _mm_cmplt_epi8((x),(y))
+#define _MM_MAX_EPI8(x,y)       _s2_max_epi8((x),(y))
+#define _MM_MIN_EPI8(x,y)       _s2_min_epi8((x),(y))
+#define _MM_ADD_EPI8(x,y)       _mm_add_epi8((x),(y))
+#define _MM_SUB_EPI8(x,y)       _mm_sub_epi8((x),(y))
+#define _MM_EXTRACT_EPI8(r,p)   _s2_extract_epi8((r),(p))
+
+#define _MM_MIN_EPU8(x,y)       _mm_min_epu8((x),(y))
+
+// Fill a SSE Register with 8 time the same 16bits integer value
+#define _MM_SET1_EPI16(x)       _mm_set1_epi16(x)
+
+#define _MM_INSERT_EPI16(r,x,i)	_mm_insert_epi16((r),(x),(i))
+#define _MM_CMPEQ_EPI16(x,y)    _mm_cmpeq_epi16((x),(y))
+#define _MM_CMPGT_EPI16(x,y)    _mm_cmpgt_epi16((x),(y))
+#define _MM_CMPGT_EPU16(x,y)    _mm_cmpgt_epu16((x),(y))    // n'existe pas ??
+#define _MM_CMPLT_EPI16(x,y)    _mm_cmplt_epi16((x),(y))
+#define _MM_MAX_EPI16(x,y)      _mm_max_epi16((x),(y))
+#define _MM_MIN_EPI16(x,y)      _mm_min_epi16((x),(y))
+#define _MM_ADD_EPI16(x,y)      _mm_add_epi16((x),(y))
+#define _MM_SUB_EPI16(x,y)      _mm_sub_epi16((x),(y))
+#define _MM_EXTRACT_EPI16(r,p)  _mm_extract_epi16((r),(p))
+#define _MM_UNPACKLO_EPI8(a,b)  _mm_unpacklo_epi8((a),(b))
+#define _MM_UNPACKHI_EPI8(a,b)  _mm_unpackhi_epi8((a),(b))
+#define _MM_ADDS_EPU16(x,y)     _mm_adds_epu16((x),(y))
+
+// Multiplication
+#define _MM_MULLO_EPI16(x,y)    _mm_mullo_epi16((x), (y))
+
+#define _MM_SRLI_EPI64(r,x)     _mm_srli_epi64((r),(x))
+#define _MM_SLLI_EPI64(r,x)     _mm_slli_epi64((r),(x))
+
+// Set a SSE Register to 0
+#define _MM_SETZERO_SI128() _mm_setzero_si128()
+
+#define _MM_AND_SI128(x,y)     _mm_and_si128((x),(y))
+#define _MM_ANDNOT_SI128(x,y)  _mm_andnot_si128((x),(y))
+#define _MM_OR_SI128(x,y)      _mm_or_si128((x),(y))
+#define _MM_XOR_SI128(x,y)     _mm_xor_si128((x),(y))
+#define _MM_SLLI_SI128(r,s)	   _mm_slli_si128((r),(s))
+#define _MM_SRLI_SI128(r,s)    _mm_srli_si128((r),(s))
+
+// Load a SSE register from an unaligned address
+#define _MM_LOADU_SI128(x) _mm_loadu_si128(x)
+
+// Load a SSE register from an aligned address (/!\ not defined when SSE not available)
+#define _MM_LOAD_SI128(x)  _mm_load_si128(x)
+
+// #define	_MM_UNPACKLO_EPI8(x,y) _mm_unpacklo_epi8((x),(y))
+
+#else /* __SSE2__ Not defined */
+
+static inline __m128i _em_set1_epi8(int x)
+{
+	um128 a;
+
+	x&=0xFF;
+	a.s8[0]=x;
+	a.s8[1]=x;
+	a.u16[1]=a.u16[0];
+	a.u32[1]=a.u32[0];
+	a.u64[1]=a.u64[0];
+
+	return a.i;
+}
+
+static inline __m128i _em_insert_epi8(__m128i r, int x, const int i)
+{
+	um128 a;
+	a.i=r;
+	a.s8[i]=x & 0xFF;
+	return a.i;
+}
+
+static inline __m128i _em_cmpeq_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=(x.s8[z]==y.s8[z]) ? 0xFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmpgt_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=(x.s8[z]>y.s8[z]) ? 0xFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmplt_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=(x.s8[z]<y.s8[z]) ? 0xFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_max_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=MAX(x.s8[z],y.s8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_min_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=MIN(x.s8[z],y.s8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_add_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=x.s8[z]+y.s8[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_sub_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s8[z]=x.s8[z]+y.s8[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+
+static inline int _em_extract_epi8(__m128i r, const int i)
+{
+	um128 a;
+
+	a.i=r;
+
+	return a.s8[i] & 0xFF;
+}
+
+static inline __m128i _em_min_epu8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.u8[z]=MIN(x.u8[z],y.u8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_set1_epi16(int x)
+{
+	um128 a;
+
+	x&=0xFFFF;
+	a.s16[0]=x;
+	a.s16[1]=x;
+	a.u32[1]=a.u32[0];
+	a.u64[1]=a.u64[0];
+
+	return a.i;
+}
+
+static inline __m128i _em_insert_epi16(__m128i r, int x, const int i)
+{
+	um128 a;
+	a.i=r;
+	a.s16[i]=x & 0xFFFF;
+	return a.i;
+}
+
+static inline __m128i _em_cmpeq_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(x.s16[z]==y.s16[z]) ? 0xFFFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmpgt_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(x.s16[z]>y.s16[z]) ? 0xFFFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_cmplt_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(x.s16[z]<y.s16[z]) ? 0xFFFF:0
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_max_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.s16[z]=MAX(x.s16[z],y.s16[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_min_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.s16[z]=MIN(x.s16[z],y.s16[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_add_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=x.s16[z]+y.s16[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_sub_epi16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=x.s16[z]+y.s16[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline int _em_extract_epi16(__m128i r, const int i)
+{
+	um128 a;
+	a.i=r;
+	return a.s16[i] & 0xFFFF;
+}
+
+static inline __m128i _em_unpacklo_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(((int16_t)(y.s8[z])) << 8) | (int16_t)(x.s8[z])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_unpackhi_epi8(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.s16[z]=(((int16_t)(y.s8[z+8])) << 8) | (int16_t)(x.s8[z+8])
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_adds_epu16(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.u16[z]=x.u16[z]+y.u16[z]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_srli_epi64(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+	x.s64[0]>>=b;
+	x.s64[1]>>=b;
+
+	return x.i;
+}
+
+static inline __m128i _em_slli_epi64(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+	x.s64[0]<<=b;
+	x.s64[1]<<=b;
+
+	return x.i;
+}
+
+static inline __m128i _em_setzero_si128()
+{
+	um128 x;
+
+	x.s64[0]=x.s64[1]=0;
+
+	return x.i;
+}
+
+static inline __m128i _em_and_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.u64[z]=x.u64[z] & y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_andnot_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+
+#define R(z) 	r.u64[z]=(~x.u64[z]) & y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_or_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.u64[z]=x.u64[z] | y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_xor_si128(__m128i a, __m128i b)
+{
+	um128 x;
+	um128 y;
+	um128 r;
+
+	x.i=a;
+	y.i=b;
+
+#define R(z) 	r.u64[z]=x.u64[z] ^ y.u64[z]
+	R(0);
+	R(1);
+#undef R
+
+	return r.i;
+}
+
+static inline __m128i _em_slli_si128(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+#define R(z) 	x.u8[z]=(z>=b) ? x.u8[z-b]:0
+	R(15);
+	R(14);
+	R(13);
+	R(12);
+	R(11);
+	R(10);
+	R(9);
+	R(8);
+	R(7);
+	R(6);
+	R(5);
+	R(4);
+	R(3);
+	R(2);
+	R(1);
+	R(0);
+#undef R
+
+	return x.i;
+}
+
+static inline __m128i _em_srli_si128(__m128i a, int b)
+{
+	um128 x;
+
+	x.i=a;
+
+#define R(z) 	x.u8[z]=((b+z) > 15) ? 0:x.u8[z+b]
+	R(0);
+	R(1);
+	R(2);
+	R(3);
+	R(4);
+	R(5);
+	R(6);
+	R(7);
+	R(8);
+	R(9);
+	R(10);
+	R(11);
+	R(12);
+	R(13);
+	R(14);
+	R(15);
+#undef R
+
+	return x.i;
+}
+
+inline static __m128i _em_loadu_si128(__m128i const *P)
+{
+	um128 tmp;
+	um128 *pp=(um128*)P;
+
+	tmp.u8[0]=(*pp).u8[0];
+    tmp.u8[1]=(*pp).u8[1];
+	tmp.u8[2]=(*pp).u8[2];
+    tmp.u8[3]=(*pp).u8[3];
+	tmp.u8[4]=(*pp).u8[4];
+    tmp.u8[5]=(*pp).u8[5];
+	tmp.u8[6]=(*pp).u8[6];
+    tmp.u8[7]=(*pp).u8[7];
+	tmp.u8[8]=(*pp).u8[8];
+    tmp.u8[9]=(*pp).u8[9];
+	tmp.u8[10]=(*pp).u8[10];
+    tmp.u8[11]=(*pp).u8[11];
+	tmp.u8[12]=(*pp).u8[12];
+    tmp.u8[13]=(*pp).u8[13];
+	tmp.u8[14]=(*pp).u8[14];
+    tmp.u8[15]=(*pp).u8[15];
+	return tmp.i;
+}
+
+
+#define _MM_SET1_EPI8(x)        _em_set1_epi8(x)
+#define _MM_INSERT_EPI8(r,x,i)	_em_insert_epi8((r),(x),(i))
+#define _MM_CMPEQ_EPI8(x,y)     _em_cmpeq_epi8((x),(y))
+#define _MM_CMPGT_EPI8(x,y)     _em_cmpgt_epi8((x),(y))
+#define _MM_CMPLT_EPI8(x,y)     _em_cmplt_epi8((x),(y))
+#define _MM_MAX_EPI8(x,y)       _em_max_epi8((x),(y))
+#define _MM_MIN_EPI8(x,y)       _em_min_epi8((x),(y))
+#define _MM_ADD_EPI8(x,y)       _em_add_epi8((x),(y))
+#define _MM_SUB_EPI8(x,y)       _em_sub_epi8((x),(y))
+#define _MM_EXTRACT_EPI8(r,p)   _em_extract_epi8((r),(p))
+
+#define _MM_MIN_EPU8(x,y)       _em_min_epu8((x),(y))
+
+#define _MM_SET1_EPI16(x)       _em_set1_epi16(x)
+#define _MM_INSERT_EPI16(r,x,i)	_em_insert_epi16((r),(x),(i))
+#define _MM_CMPEQ_EPI16(x,y)    _em_cmpeq_epi16((x),(y))
+#define _MM_CMPGT_EPI16(x,y)    _em_cmpgt_epi16((x),(y))
+#define _MM_CMPLT_EPI16(x,y)    _em_cmplt_epi16((x),(y))
+#define _MM_MAX_EPI16(x,y)      _em_max_epi16((x),(y))
+#define _MM_MIN_EPI16(x,y)      _em_min_epi16((x),(y))
+#define _MM_ADD_EPI16(x,y)      _em_add_epi16((x),(y))
+#define _MM_SUB_EPI16(x,y)      _em_sub_epi16((x),(y))
+#define _MM_EXTRACT_EPI16(r,p)  _em_extract_epi16((r),(p))
+#define _MM_UNPACKLO_EPI8(a,b)  _em_unpacklo_epi8((a),(b))
+#define _MM_UNPACKHI_EPI8(a,b)  _em_unpackhi_epi8((a),(b))
+#define _MM_ADDS_EPU16(x,y)     _em_adds_epu16((x),(y))
+
+#define _MM_SRLI_EPI64(r,x)     _em_srli_epi64((r),(x))
+#define _MM_SLLI_EPI64(r,x)     _em_slli_epi64((r),(x))
+
+#define _MM_SETZERO_SI128()     _em_setzero_si128()
+
+#define _MM_AND_SI128(x,y)      _em_and_si128((x),(y))
+#define _MM_ANDNOT_SI128(x,y)   _em_andnot_si128((x),(y))
+#define _MM_OR_SI128(x,y)       _em_or_si128((x),(y))
+#define _MM_XOR_SI128(x,y)      _em_xor_si128((x),(y))
+#define _MM_SLLI_SI128(r,s)	    _em_slli_si128((r),(s))
+#define _MM_SRLI_SI128(r,s)	    _em_srli_si128((r),(s))
+
+#define _MM_LOADU_SI128(x)      _em_loadu_si128(x)
+#define _MM_LOAD_SI128(x)       _em_loadu_si128(x)
+
+
+#endif /* __SSE2__ */
+
+#define _MM_NOT_SI128(x)     _MM_XOR_SI128((x),(_MM_SET1_EPI8(0xFFFF)))
+
+#endif

src/bloom.c

@@ -26,9 +26,9 @@
 #define MAKESTRING(n) STRING(n)
 #define STRING(n) #n
 
-#ifdef __linux__
-unsigned detect_bucket_size(unsigned fallback_size);
-#endif
+//#ifdef __linux__										// TODO commented because triggers error on luke21
+//unsigned detect_bucket_size(unsigned fallback_size);
+//#endif
 
 
 static int test_bit_set_bit(unsigned char * buf, unsigned int x, int set_bit)
@@ -89,11 +89,11 @@ static void setup_buckets(struct bloom * bloom, unsigned int cache_size)
   // either compute it or use built-in default
 
   if (cache_size == 0) {
-#ifdef __linux__
-    cache_size = detect_bucket_size(BLOOM_BUCKET_SIZE_FALLBACK);
-#else
+//#ifdef __linux__													// TODO commented because triggers error on luke21
+//    cache_size = detect_bucket_size(BLOOM_BUCKET_SIZE_FALLBACK);
+//#else
     cache_size = BLOOM_BUCKET_SIZE_FALLBACK;
-#endif
+//#endif
   }
 
   bloom->buckets = (bloom->bytes / cache_size);
@@ -122,7 +122,6 @@ static void setup_buckets(struct bloom * bloom, unsigned int cache_size)
 }
 
 
-// TODO
 int bloom_filter_size(int entries, double error)
 {
 	int bytes;
@@ -185,7 +184,8 @@ int bloom_init_size(struct bloom * bloom, int entries, double error,
 
   setup_buckets(bloom, cache_size);
 
-  // TODO comment
+  // celine.mercier@metabarcoding.org :
+  // Replaced the calloc with a memset, as the memory for the bloom filter is mapped in our data structure
   memset(bloom->bf, 0, bloom->bytes);
   //bloom->bf = (unsigned char *)calloc(bloom->bytes, sizeof(unsigned char));
   //if (bloom->bf == NULL) {

src/bloom.h

@@ -9,7 +9,12 @@
 #define _BLOOM_H
 
 
-// TODO
+/** ***************************************************************************
+ * Added by celine.mercier@metabarcoding.org
+ *
+ * Bloom filter error rate wanted.
+ *
+ */
 #define BLOOM_FILTER_ERROR_RATE (0.001)
 
 
@@ -72,7 +77,13 @@ struct bloom
 typedef struct bloom bloom_t;
 
 
-// TODO
+/** ***************************************************************************
+ * Added by celine.mercier@metabarcoding.org
+ *
+ * This function computes the size needed by the bloom filter
+ * depending on the number of entries and the error rate.
+ *
+ */
 int bloom_filter_size(int entries, double error);
 
 

src/char_str_indexer.c

@@ -14,6 +14,7 @@
 #include <stdio.h>
 #include <math.h>
 
+#include "char_str_indexer.h"
 #include "obiblob.h"
 #include "obiblob_indexer.h"
 #include "obidebug.h"
@@ -25,24 +26,16 @@
 
 Obi_blob_p obi_str_to_blob(const char* value)
 {
-	Obi_blob_p 		 value_b;
-	int32_t 		 length;
+	int32_t length;
 
 	// Compute the number of bytes on which the value will be encoded
 	length = strlen(value) + 1;		// +1 to store \0 at the end (makes retrieving faster)
 
-	value_b = obi_blob((byte_t*)value, ELEMENT_SIZE_STR, length, length);
-	if (value_b == NULL)
-	{
-		obidebug(1, "\nError encoding a character string in a blob");
-		return NULL;
-	}
-
-	return value_b;
+	return obi_blob((byte_t*)value, ELEMENT_SIZE_STR, length, length);
 }
 
 
-char* obi_blob_to_str(Obi_blob_p value_b)
+const char* obi_blob_to_str(Obi_blob_p value_b)
 {
 	return value_b->value;
 }
@@ -67,7 +60,7 @@ index_t obi_index_char_str(Obi_indexer_p indexer, const char* value)
 }
 
 
-char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx)
+const char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx)
 {
 	Obi_blob_p  value_b;
 

src/char_str_indexer.h

@@ -35,15 +35,17 @@
  * @since October 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
-Obi_blob_p obi_str_to_blob(char* value);
+Obi_blob_p obi_str_to_blob(const char* value);
 
 
 /**
  * @brief Converts a blob to a character string.
  *
+ * @warning The character string returned is mapped.
+ *
  * @param value_b The blob to convert.
  *
- * @returns A pointer to the character string contained in the blob.
+ * @returns A pointer on the character string contained in the blob.
  *
  * @since October 2015
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
@@ -51,10 +53,34 @@ Obi_blob_p obi_str_to_blob(char* value);
 const char* obi_blob_to_str(Obi_blob_p value_b);
 
 
-// TODO doc
+/**
+ * @brief Stores a character string in an indexer and returns the index.
+ *
+ * @param indexer The indexer structure.
+ * @param value The character string to index.
+ *
+ * @returns The index referring to the stored character string in the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 index_t obi_index_char_str(Obi_indexer_p indexer, const char* value);
 
-char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx);
+
+/**
+ * @brief Retrieves a character string from an indexer.
+ *
+ * @warning The character string returned is mapped.
+ *
+ * @param indexer The indexer structure.
+ * @param idx The index referring to the character string to retrieve in the indexer.
+ *
+ * @returns A pointer on the character string.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+const char* obi_retrieve_char_str(Obi_indexer_p indexer, index_t idx);
 
 
 #endif /* CHAR_STR_INDEXER_H_ */

src/crc64.h

@@ -6,4 +6,9 @@
 
 #include <stdint.h>
 
+
+/**
+ * @brief Generates and returns a 64-bit Cyclic Redundancy Check from a
+ *        character string s of length l.
+ */
 uint64_t crc64(const char* s, uint64_t l);

src/dna_seq_indexer.c

@@ -14,6 +14,7 @@
 #include <stdio.h>
 #include <math.h>
 
+#include "dna_seq_indexer.h"
 #include "obiblob.h"
 #include "obiblob_indexer.h"
 #include "obidebug.h"

src/dna_seq_indexer.h

@@ -53,9 +53,33 @@ Obi_blob_p obi_seq_to_blob(const char* seq);
 char* obi_blob_to_seq(Obi_blob_p value_b);
 
 
-// TODO doc
+/**
+ * @brief Stores a DNA sequence in an indexer and returns the index.
+ *
+ * @param indexer The indexer structure.
+ * @param value The DNA sequence to index.
+ *
+ * @returns The index referring to the stored DNA sequence in the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 index_t obi_index_dna_seq(Obi_indexer_p indexer, const char* value);
 
+
+/**
+ * @brief Retrieves a DNA sequence from an indexer.
+ *
+ * @warning The DNA sequence returned must be freed by the caller.
+ *
+ * @param indexer The indexer structure.
+ * @param idx The index referring to the DNA sequence to retrieve in the indexer.
+ *
+ * @returns A pointer on the DNA sequence.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 char* obi_retrieve_dna_seq(Obi_indexer_p indexer, index_t idx);
 
 

src/encode.c

@@ -63,17 +63,14 @@ byte_t* encode_seq_on_2_bits(const char* seq, int32_t length)
 
 	length_b = ceil((double) length / (double) 4.0);
 
-	seq_b = (byte_t*) malloc(length_b * sizeof(byte_t));
+	seq_b = (byte_t*) calloc(length_b, sizeof(byte_t));
 	if (seq_b == NULL)
 	{
-		obi_set_errno(OBI_MALLOC_ERROR);		// TODO
+		obi_set_errno(OBI_MALLOC_ERROR);
 		obidebug(1, "\nError allocating memory for an encoded DNA sequence");
 		return NULL;
 	}
 
-	// Initialize all the bits to 0
-	memset(seq_b, 0, length_b);
-
 	for (i=0; i<length; i++)
 	{
 		// Shift of 2 to make place for new nucleotide
@@ -99,7 +96,7 @@ byte_t* encode_seq_on_2_bits(const char* seq, int32_t length)
 			seq_b[i/4] |= NUC_T_2b;
 			break;
 		default:
-			obi_set_errno(OBI_ENCODE_ERROR);		// TODO
+			obi_set_errno(OBI_ENCODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when encoding (not [atgcATGC])");
 			return NULL;
 		}
@@ -125,7 +122,7 @@ char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq)
 	seq = (char*) malloc((length_seq+1) * sizeof(char));
 	if (seq == NULL)
 	{
-		obi_set_errno(OBI_MALLOC_ERROR);		// TODO
+		obi_set_errno(OBI_MALLOC_ERROR);
 		obidebug(1, "\nError allocating memory for a decoded DNA sequence");
 		return NULL;
 	}
@@ -151,7 +148,7 @@ char* decode_seq_on_2_bits(byte_t* seq_b, int32_t length_seq)
 			seq[i] = 't';
 			break;
 		default:
-			obi_set_errno(OBI_DECODE_ERROR);		// TODO
+			obi_set_errno(OBI_DECODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when decoding");
 			return NULL;
 		}
@@ -172,17 +169,14 @@ byte_t* encode_seq_on_4_bits(const char* seq, int32_t length)
 
 	length_b = ceil((double) length / (double) 2.0);
 
-	seq_b = (byte_t*) malloc(length_b * sizeof(byte_t));
+	seq_b = (byte_t*) calloc(length_b, sizeof(byte_t));
 	if (seq_b == NULL)
 	{
-		obi_set_errno(OBI_MALLOC_ERROR);		// TODO
+		obi_set_errno(OBI_MALLOC_ERROR);
 		obidebug(1, "\nError allocating memory for an encoded DNA sequence");
 		return NULL;
 	}
 
-	// Initialize all the bits to 0
-	memset(seq_b, 0, length_b);
-
 	for (i=0; i<length; i++)
 	{
 		// Shift of 4 to make place for new nucleotide
@@ -252,7 +246,7 @@ byte_t* encode_seq_on_4_bits(const char* seq, int32_t length)
 			seq_b[i/2] |= NUC_N_4b;
 			break;
 		default:
-			obi_set_errno(OBI_ENCODE_ERROR);	// TODO
+			obi_set_errno(OBI_ENCODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when encoding (not IUPAC)");
 			return NULL;
 		}
@@ -278,7 +272,7 @@ char* decode_seq_on_4_bits(byte_t* seq_b, int32_t length_seq)
 	seq = (char*) malloc((length_seq+1) * sizeof(char));
 	if (seq == NULL)
 	{
-		obi_set_errno(OBI_MALLOC_ERROR);		// TODO
+		obi_set_errno(OBI_MALLOC_ERROR);
 		obidebug(1, "\nError allocating memory for a decoded DNA sequence");
 		return NULL;
 	}
@@ -337,7 +331,7 @@ char* decode_seq_on_4_bits(byte_t* seq_b, int32_t length_seq)
 			seq[i] = 'n';
 			break;
 		default:
-			obi_set_errno(OBI_DECODE_ERROR);	// TODO
+			obi_set_errno(OBI_DECODE_ERROR);
 			obidebug(1, "\nInvalid nucleotide base when decoding");
 			return NULL;
 		}

src/encode.h

@@ -22,10 +22,10 @@
 #include "obitypes.h"
 
 
-#define NUC_MASK_2B 0x3   									/**< Binary: 11 to use when decoding 2 bits sequences
- 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
-#define NUC_MASK_4B 0xF  									/**< Binary: 1111 to use when decoding 4 bits sequences
- 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+#define NUC_MASK_2B 0x3   	/**< Binary: 11 to use when decoding 2 bits sequences
+ 	 	 	 	 	 	 	 */
+#define NUC_MASK_4B 0xF  	/**< Binary: 1111 to use when decoding 4 bits sequences
+ 	 	 	 	 	 	 	 */
 
 
 /**
@@ -107,7 +107,7 @@ byte_t* encode_seq_on_2_bits(const char* seq, int32_t length);
  *    10 -> t
  *    11 -> g
  *
- * @param seq The sequence to decode.
+ * @param seq_b The sequence to decode.
  * @param length_seq The initial length of the sequence before it was encoded.
  *
  * @returns The decoded sequence ended with '\0'.
@@ -169,7 +169,7 @@ byte_t* encode_seq_on_4_bits(const char* seq, int32_t length);
  *      V or v : 1110
  *      N or n : 1111
  *
- * @param seq The sequence to decode.
+ * @param seq_b The sequence to decode.
  * @param length_seq The initial length of the sequence before it was encoded.
  *
  * @returns The decoded sequence ended with '\0'.

src/murmurhash2.c

@@ -10,7 +10,7 @@
 
 // 1. It will not work incrementally.
 // 2. It will not produce the same results on little-endian and big-endian
-//    machines.
+//    machines.	->TODO
 
 unsigned int murmurhash2(const void * key, int len, const unsigned int seed)
 {

src/murmurhash2.h

@@ -2,6 +2,10 @@
 #ifndef _BLOOM_MURMURHASH2
 #define _BLOOM_MURMURHASH2
 
+
+/**
+ * @brief Generates and returns a hash code from a byte array.
+ */
 unsigned int murmurhash2(const void * key, int len, const unsigned int seed);
 
 #endif

src/obi_align.c

@@ -0,0 +1,109 @@
+/****************************************************************************
+ * Sequence alignment functions				                                *
+ ****************************************************************************/
+
+/**
+ * @file obi_align.c
+ * @author Celine Mercier
+ * @date May 4th 2016
+ * @brief Functions handling sequence alignments.
+ */
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <stdbool.h>
+
+#include "obidebug.h"
+#include "obierrno.h"
+#include "obitypes.h"
+#include "obiview.h"
+#include "sse_banded_LCS_alignment.h"
+
+
+#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
+
+
+// TODO
+// use openMP pragmas : garder scores en memoire et ecrire a la fin? (normalement c bon avec index)
+// tout ecrire en stdint?
+// check NUC_SEQS and score type (int or float if normalize)
+// what's with multiple sequence/line columns?
+// make function that put blobs in int16
+
+
+int obi_align_one_column(Obiview_p seq_view, OBIDMS_column_p seq_column, Obiview_p score_view, OBIDMS_column_p score_column, double threshold, bool normalize, int reference, bool similarity_mode)
+{
+	index_t i, j, k;
+	index_t seq_count;
+	char* seq1;
+	char* seq2;
+	double score;
+
+	k = 0;
+
+	if ((seq_column->header)->returned_data_type != OBI_SEQ)
+	{
+		obi_set_errno(OBI_ALIGN_ERROR);
+		obidebug(1, "\nTrying to align a column of a different type than OBI_SEQ");
+		return -1;
+	}
+
+	if ((normalize && ((score_column->header)->returned_data_type != OBI_FLOAT)) ||
+			(!normalize && ((score_column->header)->returned_data_type != OBI_INT)))
+	{
+		obi_set_errno(OBI_ALIGN_ERROR);
+		obidebug(1, "\nTrying to store alignment scores in a column of an inappropriate type");
+		return -1;
+	}
+
+	seq_count = (seq_column->header)->lines_used;
+
+	for (i=0; i < (seq_count - 1); i++)
+	{
+		for (j=i+1; j < seq_count; j++)
+		{
+			//fprintf(stderr, "\ni=%lld, j=%lld, k=%lld", i, j, k);
+
+			seq1 = obi_column_get_obiseq_with_elt_idx_in_view(seq_view, seq_column, i, 0);
+			seq2 = obi_column_get_obiseq_with_elt_idx_in_view(seq_view, seq_column, j, 0);
+
+			if ((seq1 == NULL) || (seq2 == NULL))
+			{
+				obidebug(1, "\nError retrieving sequences to align");
+				return -1;
+			}
+
+			// TODO filter
+			score = generic_sse_banded_lcs_align(seq1, seq2, threshold, normalize, reference, similarity_mode);
+
+			if (normalize)
+			{
+				if (obi_column_set_obifloat_with_elt_idx_in_view(score_view, score_column, k, 0, (obifloat_t) score) < 0)
+				{
+					obidebug(1, "\nError writing alignment score in a column");
+					return -1;
+				}
+			}
+			else
+			{
+				if (obi_column_set_obiint_with_elt_idx_in_view(score_view, score_column, k, 0, (obiint_t) score) < 0)
+				{
+					obidebug(1, "\nError writing alignment score in a column");
+					return -1;
+				}
+			}
+
+			k++;
+		}
+	}
+
+	return 0;
+}
+
+
+int obi_align_two_columns(OBIDMS_column_p seq_column_1, OBIDMS_column_p seq_column_2, OBIDMS_column_p score_column, double threshold, bool normalize, int reference, bool similarity_mode)
+{
+	// TODO
+	return 0;
+}

src/obi_align.h

@@ -0,0 +1,37 @@
+/****************************************************************************
+ * Sequence alignment functions header file	                                *
+ ****************************************************************************/
+
+/**
+ * @file obi_align.h
+ * @author Celine Mercier
+ * @date May 11th 2016
+ * @brief Header file for the functions handling the alignment of DNA sequences.
+ */
+
+
+#ifndef OBI_ALIGN_H_
+#define OBI_ALIGN_H_
+
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <stdbool.h>
+
+#include "obidms.h"
+#include "obidmscolumn.h"
+#include "obitypes.h"
+
+
+
+/**
+ * @brief
+ *
+ * TODO
+ *
+ */
+int obi_align_one_column(Obiview_p seq_view, OBIDMS_column_p seq_column, Obiview_p score_view, OBIDMS_column_p score_column, double threshold, bool normalize, int reference, bool similarity_mode);
+
+
+#endif /* OBI_ALIGN_H_ */
+

src/obiavl.c

@@ -6,7 +6,7 @@
  * @file obiavl.c
  * @author Celine Mercier
  * @date December 3rd 2015
- * @brief Functions handling AVL trees for storing and retrieving bit arrays.
+ * @brief Functions handling AVL trees for storing and retrieving blobs.
  */
 
 
@@ -108,7 +108,44 @@ static char* build_avl_data_file_name(const char* avl_name);
 
 
 /**
- * @brief Internal function returning the size of an AVL tree header on this platform.
+ * @brief Internal function building the full path of an AVL tree file.
+ *
+ * @warning The returned pointer has to be freed by the caller.
+ *
+ * @param dms A pointer to the OBIDMS to which the AVL tree belongs.
+ * @param avl_name The name of the AVL tree.
+ * @param avl_idx The index of the AVL if it's part of an AVL group, or -1 if not.
+ *
+ * @returns A pointer to the full path of the file where the AVL tree is stored.
+ * @retval NULL if an error occurred.
+ *
+ * @since May 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+static char* get_full_path_of_avl_file_name(OBIDMS_p dms, const char* avl_name, int avl_idx);
+
+
+/**
+ * @brief Internal function building the file name for an AVL data file.
+ *
+ * @warning The returned pointer has to be freed by the caller.
+ *
+ * @param dms A pointer to the OBIDMS to which the AVL tree belongs.
+ * @param avl_name The name of the AVL tree.
+ * @param avl_idx The index of the AVL if it's part of an AVL group, or -1 if not.
+ *
+ * @returns A pointer to the full path of the file where the data referred to by the AVL tree is stored.
+ * @retval NULL if an error occurred.
+ *
+ * @since May 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+static char* get_full_path_of_avl_data_file_name(OBIDMS_p dms, const char* avl_name, int avl_idx);
+
+
+/**
+ * @brief Internal function returning the size of an AVL tree header on this platform,
+ *        including the size of the bloom filter associated with the AVL tree.
  *
  * @returns The size of an AVL tree header in bytes.
  *
@@ -252,9 +289,12 @@ int remap_an_avl(OBIDMS_avl_p avl);
 
 
 /**
- * @brief Internal function (re)mapping the tree and data parts of an AVL tree structure.
+ * @brief Internal function creating and adding a new AVL in an AVL group.
  *
- * @param avl A pointer to the AVL tree group structure.
+ * @warning The previous AVL in the list of the group is unmapped,
+ * 			if it's not the 1st AVL being added.
+ *
+ * @param avl A pointer on the AVL tree group structure.
  *
  * @retval 0 if the operation was successfully completed.
  * @retval -1 if an error occurred.
@@ -282,38 +322,6 @@ int add_new_avl_in_group(OBIDMS_avl_group_p avl_group);
 int maybe_in_avl(OBIDMS_avl_p avl, Obi_blob_p value);
 
 
-/**
- * @brief Internal function comparing two blobs.
- *
- * The encoding is compared first, then the length of the
- * values, then the values themselves.
- *
- * @param value_1 A pointer to the first blob structure.
- * @param value_2 A pointer to the second blob structure.
- *
- * @returns A value < 0 if value_1 < value_2,
- * 			a value > 0 if value_1 > value_2,
- * 			and 0 if value_1 == value_2.
- *
- * @since October 2015
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int blob_compare(Obi_blob_p value_1, Obi_blob_p value_2);
-
-
-/**
- * @brief Internal function calculating the size in bytes of a blob.
- *
- * @param value A pointer to the blob structure.
- *
- * @returns The size of the blob in bytes.
- *
- * @since October 2015
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-int blob_sizeof(Obi_blob_p value);
-
-
 /**
  * @brief Internal function storing a value (blob) in the data array referred to by an AVL tree.
  *
@@ -578,6 +586,102 @@ static char* build_avl_data_file_name(const char* avl_name)
 }
 
 
+static char* get_full_path_of_avl_file_name(OBIDMS_p dms, const char* avl_name, int avl_idx)
+{
+	char* complete_avl_name;
+	char* full_path;
+	char* avl_file_name;
+
+	if (avl_idx >= 0)
+	{
+		complete_avl_name = build_avl_name_with_idx(avl_name, avl_idx);
+		if (complete_avl_name == NULL)
+			return NULL;
+	}
+	else
+	{
+		complete_avl_name = (char*) malloc((strlen(avl_name)+1)*sizeof(char));
+		if (complete_avl_name == NULL)
+		{
+    		obi_set_errno(OBI_MALLOC_ERROR);
+    		obidebug(1, "\nError allocating memory for an AVL name");
+			return NULL;
+		}
+		strcpy(complete_avl_name, avl_name);
+	}
+
+	avl_file_name = build_avl_file_name(complete_avl_name);
+	if (avl_file_name == NULL)
+	{
+		free(complete_avl_name);
+		return NULL;
+	}
+
+	full_path = get_full_path_of_avl_dir(dms, avl_name);
+	if (full_path == NULL)
+	{
+		free(complete_avl_name);
+		free(avl_file_name);
+		return NULL;
+	}
+
+	strcat(full_path, "/");
+	strcat(full_path, avl_file_name);
+
+	free(complete_avl_name);
+
+	return full_path;
+}
+
+
+static char* get_full_path_of_avl_data_file_name(OBIDMS_p dms, const char* avl_name, int avl_idx)
+{
+	char* complete_avl_name;
+	char* full_path;
+	char* avl_data_file_name;
+
+	if (avl_idx >= 0)
+	{
+		complete_avl_name = build_avl_name_with_idx(avl_name, avl_idx);
+		if (complete_avl_name == NULL)
+			return NULL;
+	}
+	else
+	{
+		complete_avl_name = (char*) malloc((strlen(avl_name)+1)*sizeof(char));
+		if (complete_avl_name == NULL)
+		{
+    		obi_set_errno(OBI_MALLOC_ERROR);
+    		obidebug(1, "\nError allocating memory for an AVL name");
+			return NULL;
+		}
+		strcpy(complete_avl_name, avl_name);
+	}
+
+	avl_data_file_name = build_avl_data_file_name(complete_avl_name);
+	if (avl_data_file_name == NULL)
+	{
+		free(complete_avl_name);
+		return NULL;
+	}
+
+	full_path = get_full_path_of_avl_dir(dms, avl_name);
+	if (full_path == NULL)
+	{
+		free(complete_avl_name);
+		free(avl_data_file_name);
+		return NULL;
+	}
+
+	strcat(full_path, "/");
+	strcat(full_path, avl_data_file_name);
+
+	free(complete_avl_name);
+
+	return full_path;
+}
+
+
 size_t get_avl_header_size()
 {
 	size_t header_size;
@@ -586,7 +690,7 @@ size_t get_avl_header_size()
 
 	header_size = sizeof(OBIDMS_avl_header_t) + bloom_filter_size(MAX_NODE_COUNT_PER_AVL, BLOOM_FILTER_ERROR_RATE);
 
-	multiple = 	ceil((double) header_size / (double) getpagesize());
+	multiple = ceil((double) header_size / (double) getpagesize());
 
 	rounded_header_size = multiple * getpagesize();
 
@@ -677,7 +781,6 @@ int truncate_avl_to_size_used(OBIDMS_avl_p avl)	// TODO is it necessary to unmap
 					 file_descriptor,
 					 (avl->header)->header_size
 					);
-
 	if (avl->tree == MAP_FAILED)
 	{
 		obi_set_errno(OBI_AVL_ERROR);
@@ -954,23 +1057,24 @@ int remap_an_avl(OBIDMS_avl_p avl)
 int add_new_avl_in_group(OBIDMS_avl_group_p avl_group)
 {
 	// Check that maximum number of AVLs in a group was not reached
-	if (avl_group->current_avl_idx == (MAX_NB_OF_AVLS_IN_GROUP-1))
+	if (avl_group->last_avl_idx == (MAX_NB_OF_AVLS_IN_GROUP - 1))
 	{
 		obi_set_errno(OBI_AVL_ERROR);
 		obidebug(1, "\nError: Trying to add new AVL in AVL group but maximum number of AVLs in a group reached");
 		return -1;
 	}
 
-	// Unmap the previous AVL
-	if (unmap_an_avl((avl_group->sub_avls)[avl_group->current_avl_idx]) < 0)
-		return -1;
+	// Unmap the previous AVL if it's not the 1st
+	if (avl_group->last_avl_idx > 0)
+		if (unmap_an_avl((avl_group->sub_avls)[avl_group->last_avl_idx]) < 0)
+			return -1;
 
 	// Increment current AVL index
-	(avl_group->current_avl_idx)++;
+	(avl_group->last_avl_idx)++;
 
 	// Create the new AVL
-	(avl_group->sub_avls)[avl_group->current_avl_idx] = obi_create_avl(avl_group->dms, avl_group->name, avl_group->current_avl_idx);
-	if ((avl_group->sub_avls)[avl_group->current_avl_idx] == NULL)
+	(avl_group->sub_avls)[avl_group->last_avl_idx] = obi_create_avl(avl_group->dms, avl_group->name, avl_group->last_avl_idx);
+	if ((avl_group->sub_avls)[avl_group->last_avl_idx] == NULL)
 	{
 		obidebug(1, "\nError creating a new AVL tree in a group");
 		return -1;
@@ -980,43 +1084,39 @@ int add_new_avl_in_group(OBIDMS_avl_group_p avl_group)
 }
 
 
+// TODO doc
+int add_existing_avl_in_group(OBIDMS_avl_group_p avl_group_dest, OBIDMS_avl_group_p avl_group_source, int avl_idx)
+{
+	if (link(get_full_path_of_avl_file_name(avl_group_source->dms, avl_group_source->name, avl_idx), get_full_path_of_avl_file_name(avl_group_dest->dms, avl_group_dest->name, avl_idx)) < 0)
+	{
+		obi_set_errno(OBI_AVL_ERROR);
+		obidebug(1, "\nError creating a hard link to an existing AVL tree file");
+		return -1;
+	}
+	if (link(get_full_path_of_avl_data_file_name(avl_group_source->dms, avl_group_source->name, avl_idx), get_full_path_of_avl_data_file_name(avl_group_dest->dms, avl_group_dest->name, avl_idx)) < 0)
+	{
+		obi_set_errno(OBI_AVL_ERROR);
+		obidebug(1, "\nError creating a hard link to an existing AVL data file");
+		return -1;
+	}
+
+	// Increment current AVL index
+	(avl_group_dest->last_avl_idx)++;
+
+	// Open AVL for that group 		TODO ideally not needed, but needed for now
+	avl_group_dest->sub_avls[avl_group_dest->last_avl_idx] = obi_open_avl(avl_group_source->dms, avl_group_source->name, avl_idx);
+	if ((avl_group_dest->sub_avls)[avl_group_dest->last_avl_idx] == NULL)
+	{
+		obidebug(1, "\nError opening an AVL to add in an AVL group");
+		return -1;
+	}
+
+	return 0;
+}
+
 int maybe_in_avl(OBIDMS_avl_p avl, Obi_blob_p value)
 {
-	return (bloom_check(&((avl->header)->bloom_filter), value, blob_sizeof(value)));
-}
-
-
-int blob_compare(Obi_blob_p value_1, Obi_blob_p value_2)
-{
-	int 	comp;
-	int32_t b;
-
-	if (value_1->element_size != value_2->element_size)
-		return (value_1->element_size - value_2->element_size);
-
-	if (value_1->length_encoded_value != value_2->length_encoded_value)
-		return (value_1->length_encoded_value - value_2->length_encoded_value);
-
-	if (value_1->element_size != ELEMENT_SIZE_STR) // because if so, length_decoded_value == length_encoded_value
-	{
-		if (value_1->length_decoded_value != value_2->length_decoded_value)
-			return (value_1->length_decoded_value - value_2->length_decoded_value);
-	}
-
-	b = 0;
-	comp = 0;
-	while (!comp && (b < value_1->length_encoded_value))
-	{
-		comp = *((value_1->value)+b) - *((value_2->value)+b);
-		b++;
-	}
-	return comp;
-}
-
-
-int blob_sizeof(Obi_blob_p value)
-{
-	return (sizeof(Obi_blob_t) + (value->length_encoded_value));
+	return (bloom_check(&((avl->header)->bloom_filter), value, obi_blob_sizeof(value)));
 }
 
 
@@ -1028,7 +1128,7 @@ index_t avl_add_value_in_data_array(OBIDMS_avl_p avl, Obi_blob_p value)
 	value_idx = ((avl->data)->header)->data_size_used;
 
 	// Grow the data if needed
-	value_size = blob_sizeof(value);
+	value_size = obi_blob_sizeof(value);
 	while (((avl->data)->header)->data_size_max < (value_idx + value_size))
 	{
 		if (grow_avl_data(avl->data) < 0)
@@ -1067,7 +1167,7 @@ AVL_node_p avl_create_node(OBIDMS_avl_p avl, index_t node_idx)
 // Update the balance factors of the nodes from the node that will need balancing
 void avl_update_balance_factors(OBIDMS_avl_p avl)
 {
-	uint8_t n;
+	uint8_t    n;
 	AVL_node_p node;
 
 	// Update balance factors from the node where balancing might be needed
@@ -1221,8 +1321,8 @@ index_t avl_balance_node(OBIDMS_avl_p avl, AVL_node_p node, index_t node_idx)
 // Balance a given tree
 void avl_balance(OBIDMS_avl_p avl)
 {
-	index_t new_root;
-	index_t node_index;
+	index_t    new_root;
+	index_t    node_index;
 	AVL_node_p node_to_balance;
 	AVL_node_p parent_of_node_to_balance;
 
@@ -1594,8 +1694,7 @@ OBIDMS_avl_p obi_create_avl(OBIDMS_p dms, const char* avl_name, int avl_idx)
 	// Bloom filter
 	bloom_init(&((avl->header)->bloom_filter), MAX_NODE_COUNT_PER_AVL);
 
-	if (avl_idx >= 0)
-		free(complete_avl_name);
+	free(complete_avl_name);
 
 	return avl;
 }
@@ -1842,8 +1941,7 @@ OBIDMS_avl_p obi_open_avl(OBIDMS_p dms, const char* avl_name, int avl_idx)
 	avl->dir_fd    = avl_dir_file_descriptor;
 	avl->avl_fd    = avl_file_descriptor;
 
-	if (avl_idx >= 0)
-		free(complete_avl_name);
+	free(complete_avl_name);
 
 	return avl;
 }
@@ -1871,6 +1969,7 @@ OBIDMS_avl_group_p obi_avl_group(OBIDMS_p dms, const char* avl_name)
 OBIDMS_avl_group_p obi_create_avl_group(OBIDMS_p dms, const char* avl_name)
 {
 	OBIDMS_avl_group_p avl_group;
+	char* 			   avl_dir_name;
 
 	avl_group = (OBIDMS_avl_group_p) malloc(sizeof(OBIDMS_avl_group_t));
 	if (avl_group == NULL)
@@ -1880,18 +1979,22 @@ OBIDMS_avl_group_p obi_create_avl_group(OBIDMS_p dms, const char* avl_name)
 		return NULL;
 	}
 
-	// Create 1st avl
-	(avl_group->sub_avls)[0] = obi_create_avl(dms, avl_name, 0);
-	if ((avl_group->sub_avls)[0] == NULL)
-	{
-		obidebug(1, "\nError creating the first AVL of an AVL group");
-		return NULL;
-	}
-
-	avl_group->current_avl_idx = 0;
+	avl_group->last_avl_idx = -1;
+	avl_group->dms = dms;
 	strcpy(avl_group->name, avl_name);
 
-	avl_group->dms = dms;
+	// Create the directory for that AVL group
+	avl_dir_name = get_full_path_of_avl_dir(dms, avl_name);
+	if (avl_dir_name == NULL)
+		return NULL;
+
+	if (mkdirat(dms->indexer_dir_fd, avl_dir_name, 00777) < 0)
+	{
+		obi_set_errno(OBI_AVL_ERROR);
+		obidebug(1, "\nError creating an AVL directory");
+		free(avl_dir_name);
+		return NULL;
+	}
 
 	// Add in the list of open indexers
 	obi_dms_list_indexer(dms, avl_group);
@@ -1948,7 +2051,7 @@ OBIDMS_avl_group_p obi_open_avl_group(OBIDMS_p dms, const char* avl_name)
 		if ((avl_group->sub_avls)[i] == NULL)
 			return NULL;
 	}
-	avl_group->current_avl_idx = avl_count-1;			// TODO latest. discuss
+	avl_group->last_avl_idx = avl_count-1;			// TODO latest. discuss
 	strcpy(avl_group->name, avl_name);
 
 	avl_group->dms = dms;
@@ -1966,15 +2069,68 @@ OBIDMS_avl_group_p obi_open_avl_group(OBIDMS_p dms, const char* avl_name)
 }
 
 
+void obi_clone_avl(OBIDMS_avl_p avl, OBIDMS_avl_p new_avl)
+{
+	// Clone AVL tree
+	memcpy(new_avl->tree, avl->tree, (avl->header)->avl_size);
+	(new_avl->header)->avl_size     = (avl->header)->avl_size;
+	(new_avl->header)->nb_items     = (avl->header)->nb_items;
+	(new_avl->header)->root_idx     = (avl->header)->root_idx;
+	(new_avl->header)->bloom_filter = (avl->header)->bloom_filter;
+
+	// Clone AVL data
+	memcpy((new_avl->data)->data, (avl->data)->data, ((avl->data)->header)->data_size_used);
+	((new_avl->data)->header)->data_size_used = ((avl->data)->header)->data_size_used;
+	((new_avl->data)->header)->data_size_max  = ((avl->data)->header)->data_size_max;
+	((new_avl->data)->header)->nb_items       = ((avl->data)->header)->nb_items;
+}
+
+
+OBIDMS_avl_group_p obi_clone_avl_group(OBIDMS_avl_group_p avl_group, const char* new_avl_name)
+{
+	OBIDMS_avl_group_p new_avl_group;
+	int i;
+
+	// Create the new AVL group
+	new_avl_group = obi_create_avl_group(avl_group->dms, new_avl_name);
+
+	// Create hard links to all the full AVLs that won't be modified: all but the last one
+	for (i=0; i<(avl_group->last_avl_idx); i++)
+	{
+		if (add_existing_avl_in_group(new_avl_group, avl_group, i) < 0)
+			return NULL;
+	}
+
+	// Create the last AVL to copy data in it
+	if (add_new_avl_in_group(new_avl_group) < 0)
+	{
+		obi_close_avl_group(new_avl_group);
+		return NULL;
+	}
+
+	// Copy the data from the last AVL to a new one that can be modified
+	obi_clone_avl((avl_group->sub_avls)[avl_group->last_avl_idx], (new_avl_group->sub_avls)[new_avl_group->last_avl_idx]);
+
+	// Close old AVL group
+	if (obi_close_avl_group(avl_group) < 0)
+	{
+		obi_close_avl_group(new_avl_group);
+		return NULL;
+	}
+
+	return new_avl_group;
+}
+
+
 int obi_close_avl(OBIDMS_avl_p avl)
 {
-	int                 	ret_val = 0;
+	int ret_val = 0;
 
 	ret_val = close_avl_data(avl->data);
 
 	ret_val = truncate_avl_to_size_used(avl);
 
-	if (munmap(avl->tree, (((avl->header)->nb_items_max) * sizeof(AVL_node_t))) < 0)
+	if (munmap(avl->tree, (avl->header)->avl_size) < 0)
 	{
 		obi_set_errno(OBI_AVL_ERROR);
 		obidebug(1, "\nError munmapping the tree of an AVL tree file");
@@ -2011,9 +2167,17 @@ int obi_close_avl_group(OBIDMS_avl_group_p avl_group)
 		ret_val = obi_dms_unlist_indexer(avl_group->dms, avl_group);
 
 		// Close each AVL of the group
-		for (i=0; i < (avl_group->current_avl_idx); i++)
+		for (i=0; i <= (avl_group->last_avl_idx); i++)
+		{
+			// Remap all but the last AVL (already mapped) before closing to truncate and close properly
+			if (i < (avl_group->last_avl_idx))
+			{
+				if (remap_an_avl((avl_group->sub_avls)[i]) < 0)
+					ret_val = -1;
+			}
 			if (obi_close_avl((avl_group->sub_avls)[i]) < 0)
 				ret_val = -1;
+		}
 
 		free(avl_group);
 	}
@@ -2043,7 +2207,7 @@ index_t obi_avl_add(OBIDMS_avl_p avl, Obi_blob_p value)
 
 	n = 0;
 	depth = 0;
-	crc = crc64((byte_t*)value, blob_sizeof(value));
+	crc = crc64((byte_t*)value, obi_blob_sizeof(value));
 
 	// Check if first node
 	if (!((avl->header)->nb_items))
@@ -2090,7 +2254,7 @@ index_t obi_avl_add(OBIDMS_avl_p avl, Obi_blob_p value)
 		if (comp == 0)
 		{ // check if really same value
 			to_compare = obi_avl_get(avl, current_node->value);
-			comp = blob_compare(to_compare, value);
+			comp = obi_blob_compare(to_compare, value);
 		}
 
 		if (comp > 0)
@@ -2172,7 +2336,7 @@ index_t obi_avl_find(OBIDMS_avl_p avl, Obi_blob_p value)
 	AVL_node_p       current_node;
 	uint64_t         crc;
 
-	crc = crc64((byte_t*)value, blob_sizeof(value));
+	crc = crc64((byte_t*)value, obi_blob_sizeof(value));
 
 	next = (avl->header)->root_idx;
 	while (next != -1)
@@ -2185,7 +2349,7 @@ index_t obi_avl_find(OBIDMS_avl_p avl, Obi_blob_p value)
 		if (comp == 0)
 		{ // Check if really same value
 			to_compare = obi_avl_get(avl, current_node->value);
-			comp = blob_compare(to_compare, value);
+			comp = obi_blob_compare(to_compare, value);
 		}
 
 		if (comp > 0)
@@ -2222,18 +2386,29 @@ index_t obi_avl_group_add(OBIDMS_avl_group_p avl_group, Obi_blob_p value)
 	index_t index_with_avl;
 	int     i;
 
-	if (maybe_in_avl((avl_group->sub_avls)[avl_group->current_avl_idx], value))
+	// Create 1st AVL if group is empty
+	if (avl_group->last_avl_idx == -1)
 	{
-		index_in_avl = (int32_t) obi_avl_find((avl_group->sub_avls)[avl_group->current_avl_idx], value);
+		if (add_new_avl_in_group(avl_group) < 0)
+		{
+			obidebug(1, "\nError creating the first AVL of an AVL group");
+			return -1;
+		}
+	}
+	
+	if (maybe_in_avl((avl_group->sub_avls)[avl_group->last_avl_idx], value))
+	{
+		index_in_avl = (int32_t) obi_avl_find((avl_group->sub_avls)[avl_group->last_avl_idx], value);
 		if (index_in_avl >= 0)
 		{
-			index_with_avl = avl_group->current_avl_idx;
+			index_with_avl = avl_group->last_avl_idx;
 			index_with_avl = index_with_avl << 32;
 			index_with_avl = index_with_avl + index_in_avl;
 			return index_with_avl;
 		}
 	}
-	for (i=0; i < (avl_group->current_avl_idx); i++)
+
+	for (i=0; i < (avl_group->last_avl_idx); i++)
 	{
 		if (maybe_in_avl((avl_group->sub_avls)[i], value))
 		{
@@ -2257,24 +2432,23 @@ index_t obi_avl_group_add(OBIDMS_avl_group_p avl_group, Obi_blob_p value)
 	// Check if the AVL group is writable
 	if (!(avl_group->writable))
 	{
-		obi_set_errno(OBI_AVL_ERROR);
-		obidebug(1, "\nTrying to add a value in an AVL group that is read-only.");
+		obi_set_errno(OBI_READ_ONLY_INDEXER_ERROR);
 		return -1;
 	}
 
 	// Check if need to make new AVL
-	if (((((avl_group->sub_avls)[avl_group->current_avl_idx])->header)->nb_items == MAX_NODE_COUNT_PER_AVL) || (((((avl_group->sub_avls)[avl_group->current_avl_idx])->data)->header)->data_size_used >= MAX_DATA_SIZE_PER_AVL))
+	if (((((avl_group->sub_avls)[avl_group->last_avl_idx])->header)->nb_items == MAX_NODE_COUNT_PER_AVL) || (((((avl_group->sub_avls)[avl_group->last_avl_idx])->data)->header)->data_size_used >= MAX_DATA_SIZE_PER_AVL))
 	{
 		if (add_new_avl_in_group(avl_group) < 0)
 			return -1;
 	}
 
 	// Add in the current AVL
-	bloom_add(&((((avl_group->sub_avls)[avl_group->current_avl_idx])->header)->bloom_filter), value, blob_sizeof(value));
-	index_in_avl = (int32_t) obi_avl_add((avl_group->sub_avls)[avl_group->current_avl_idx], value);
+	bloom_add(&((((avl_group->sub_avls)[avl_group->last_avl_idx])->header)->bloom_filter), value, obi_blob_sizeof(value));
+	index_in_avl = (int32_t) obi_avl_add((avl_group->sub_avls)[avl_group->last_avl_idx], value);
 
 	// Build the index containing the AVL index
-	index_with_avl = avl_group->current_avl_idx;
+	index_with_avl = avl_group->last_avl_idx;
 	index_with_avl = index_with_avl << 32;
 	index_with_avl = index_with_avl + index_in_avl;
 
@@ -2282,3 +2456,8 @@ index_t obi_avl_group_add(OBIDMS_avl_group_p avl_group, Obi_blob_p value)
 }
 
 
+const char* obi_avl_group_get_name(OBIDMS_avl_group_p avl_group)
+{
+	return avl_group->name;
+}
+

src/obiavl.h

@@ -6,7 +6,7 @@
  * @file obiavl.h
  * @author Celine Mercier
  * @date December 3rd 2015
- * @brief Header file for handling AVL trees for storing and retrieving blobs (i.e. coding for character strings).
+ * @brief Header file for handling AVL trees for storing and retrieving blobs.
  */
 
 
@@ -73,7 +73,7 @@ typedef struct AVL_node {
  * @brief OBIDMS AVL tree data header structure.
  */
 typedef struct OBIDMS_avl_data_header {
-	int			header_size;						/**< Size of the header in bytes.
+	size_t		header_size;						/**< Size of the header in bytes.
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	index_t		data_size_used;						/**< Size of the data used in bytes.
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
@@ -105,7 +105,7 @@ typedef struct OBIDMS_avl_data {
  * @brief OBIDMS AVL tree header structure.
  */
 typedef struct OBIDMS_avl_header {
-	int			header_size;						/**< Size of the header in bytes.
+	size_t		header_size;						/**< Size of the header in bytes.
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	size_t		avl_size;							/**< Size of the AVL tree in bytes.
 	 	 	 	 	 	 	 	 	 	 	 	 	 */
@@ -120,14 +120,13 @@ typedef struct OBIDMS_avl_header {
 	time_t		creation_date;			    	    /**< Date of creation of the file.
 												     */
 	bloom_t 	bloom_filter;			    	    /**< Bloom filter associated with the AVL tree, enabling to know if a value
-													 *   might already be stored in the data associated with the tree.
+													 *   might already be stored in the data referred to by the tree.
      	 	 	 	 	 	 	 	 	 	 	 	 */
 } OBIDMS_avl_header_t, *OBIDMS_avl_header_p;
 
 
 /**
  * @brief OBIDMS AVL tree structure.
- * 		  TODO doc
  */
 typedef struct OBIDMS_avl {
 	OBIDMS_p                	dms;			 			/**< A pointer to the OBIDMS structure to which the AVL tree belongs.
@@ -161,7 +160,7 @@ typedef struct OBIDMS_avl {
 typedef struct OBIDMS_avl_group {
 	OBIDMS_avl_p 	sub_avls[MAX_NB_OF_AVLS_IN_GROUP];		/**< Array containing the pointers to the AVL trees of the group.
 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
-	int 			current_avl_idx;						/**< Index in the sub_avls array of the AVL tree currently being filled.
+	int 			last_avl_idx;							/**< Index in the sub_avls array of the AVL tree currently being filled.
 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	char			name[AVL_MAX_NAME+1];				 	/**< Base name of the AVL group. The AVL trees in it have names of the form basename_idx.
 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
@@ -195,7 +194,7 @@ int obi_avl_exists(OBIDMS_p dms, const char* avl_name);
  * @brief Creates an AVL tree. Fails if it already exists.
  *
  * Note: An AVL tree is made of two files (referred to by two structures).
- * 		 One file contains the indices referring to the data, and the other
+ * 		 One file contains the tree referring to the data, and the other
  * 		 file contains the data itself. The AVL tree as a whole is referred
  * 		 to via the OBIDMS_avl structure. An AVL tree is stored in a directory
  * 		 with the same name, or with the base name of the AVL group if it is
@@ -203,7 +202,8 @@ int obi_avl_exists(OBIDMS_p dms, const char* avl_name);
  *
  * @param dms The OBIDMS to which the AVL tree belongs.
  * @param avl_name The name of the AVL tree.
- * @param avl_idx The index of the AVL tree if it is part of an AVL group.
+ * @param avl_idx The index of the AVL tree if it is part of an AVL group,
+ *        or -1 if it is not part of an AVL group.
  *
  * @returns A pointer to the newly created AVL tree structure.
  * @retval NULL if an error occurred.
@@ -215,16 +215,17 @@ OBIDMS_avl_p obi_create_avl(OBIDMS_p dms, const char* avl_name, int avl_idx);
 
 
 /**
- * @brief Opens an AVL tree. Fails if it does not already exist.
+ * @brief Opens an AVL tree in read-only mode. Fails if it does not already exist.
  *
  * Note: An AVL tree is made of two files (referred to by two structures).
- * 		 One file contains the indices referring to the data, and the other
+ * 		 One file contains the tree referring to the data, and the other
  * 		 file contains the data itself. The AVL tree as a whole is referred
  * 		 to via the OBIDMS_avl structure.
  *
  * @param dms The OBIDMS to which the AVL tree belongs.
  * @param avl_name The name of the AVL tree.
- * @param avl_idx The index of the AVL tree if it is part of an AVL group.
+ * @param avl_idx The index of the AVL tree if it is part of an AVL group,
+ *        or -1 if it is not part of an AVL group.
  *
  * @returns A pointer to the AVL tree structure.
  * @retval NULL if an error occurred.
@@ -272,7 +273,7 @@ OBIDMS_avl_group_p obi_create_avl_group(OBIDMS_p dms, const char* avl_name);
 
 
 /**
- * @brief Opens an AVL tree group.
+ * @brief Opens an AVL tree group in read-only mode.
  *
  * Note: An AVL tree group is composed of multiple AVL trees that all have the
  *       same base name, and an index differentiating them.
@@ -289,6 +290,37 @@ OBIDMS_avl_group_p obi_create_avl_group(OBIDMS_p dms, const char* avl_name);
 OBIDMS_avl_group_p obi_open_avl_group(OBIDMS_p dms, const char* avl_name);
 
 
+/**
+ * @brief Clones an AVL.
+ *
+ * The tree and the data are both cloned into the new AVL.
+ *
+ * @param avl A pointer on the AVL to clone.
+ * @param new_avl A pointer on the new AVL to fill.
+ *
+ * @since May 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+void obi_clone_avl(OBIDMS_avl_p avl, OBIDMS_avl_p new_avl);
+
+
+/**
+ * @brief Clones an AVL group.
+ *
+ * @warning The AVL group that has be cloned is closed by the function.
+ *
+ * @param avl_group A pointer on the AVL group to clone.
+ * @param new_avl_name The name of the new AVL group.
+ *
+ * @returns A pointer on the new AVL group structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since May 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+OBIDMS_avl_group_p obi_clone_avl_group(OBIDMS_avl_group_p avl_group, const char* new_avl_name);
+
+
 /**
  * @brief Closes an AVL tree.
  *
@@ -321,6 +353,7 @@ int obi_close_avl_group(OBIDMS_avl_group_p avl_group);
  * @brief Recovers a value (blob) in an AVL tree.
  *
  * @warning The blob recovered must be decoded to get the original value.
+ * @warning The blob recovered is mapped in memory.
  *
  * @param avl A pointer to the AVL tree.
  * @param index The index of the value in the data array.
@@ -334,9 +367,10 @@ Obi_blob_p obi_avl_get(OBIDMS_avl_p avl, index_t index);
 
 
 /**
- * @brief Adds a value (blob) in an AVL tree NOT checking first if it is already in it.		// TODO to discuss
+ * @brief Adds a value (blob) in an AVL tree.
  *
  * @warning The value given must be already be encoded into a blob structure (Obi_blob_t).
+ * @warning If the value is already in the AVL tree, an error will be triggered.		// TODO to discuss
  *
  * @param avl A pointer to the AVL tree.
  * @param value The blob to add in the AVL tree.
@@ -371,9 +405,13 @@ index_t obi_avl_find(OBIDMS_avl_p avl, Obi_blob_p value);
  * @brief Recovers a value (blob) in an AVL tree.
  *
  * @warning The blob recovered must be decoded to get the original value.
+ * @warning The blob recovered is mapped in memory.
  *
  * @param avl_group A pointer to the AVL tree.
- * @param index The index of the value in the data array.
+ * @param index The index of the value in the form of a 64-bit integer
+ *              with the 32 left-most bits coding for the index of the tree of
+ *              the group in which the value is stored, and the 32 right-most bits
+ *              coding for the index at which the value is stored in that AVL tree.
  *
  * @returns A pointer to the blob recovered.
  *
@@ -391,7 +429,10 @@ Obi_blob_p obi_avl_group_get(OBIDMS_avl_group_p avl_group, index_t idx);
  * @param avl_group A pointer to the AVL tree group.
  * @param value The blob to add in the AVL tree group.
  *
- * @returns The index of the value newly added in the AVL tree group.
+ * @returns The index of the value newly added in the AVL tree group, in the form of a
+ *          64-bit integer with the 32 left-most bits coding for the index of the tree
+ *          of the group in which the value is stored, and the 32 right-most bits
+ *          coding for the index at which the value is stored in that AVL tree.
  * @retval -1 if an error occurred.
  *
  * @since April 2016
@@ -400,5 +441,18 @@ Obi_blob_p obi_avl_group_get(OBIDMS_avl_group_p avl_group, index_t idx);
 index_t obi_avl_group_add(OBIDMS_avl_group_p avl_group, Obi_blob_p value);
 
 
+/**
+ * @brief Recovers the name of an AVL group.
+ *
+ * @param avl_group A pointer on the AVL group structure.
+ *
+ * @returns A pointer on the name of the AVL group.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+const char* obi_avl_group_get_name(OBIDMS_avl_group_p avl_group);
+
+
 #endif /* OBIAVL_H_ */
 

src/obiblob.c

@@ -55,3 +55,37 @@ Obi_blob_p obi_blob(byte_t* encoded_value, uint8_t element_size, int32_t length_
 	return blob;
 }
 
+
+int obi_blob_compare(Obi_blob_p value_1, Obi_blob_p value_2)
+{
+	int 	comp;
+	int32_t b;
+
+	if (value_1->element_size != value_2->element_size)
+		return (value_1->element_size - value_2->element_size);
+
+	if (value_1->length_encoded_value != value_2->length_encoded_value)
+		return (value_1->length_encoded_value - value_2->length_encoded_value);
+
+	if (value_1->element_size != ELEMENT_SIZE_STR) // because if so, length_decoded_value == length_encoded_value
+	{
+		if (value_1->length_decoded_value != value_2->length_decoded_value)
+			return (value_1->length_decoded_value - value_2->length_decoded_value);
+	}
+
+	b = 0;
+	comp = 0;
+	while (!comp && (b < value_1->length_encoded_value))
+	{
+		comp = *((value_1->value)+b) - *((value_2->value)+b);
+		b++;
+	}
+	return comp;
+}
+
+
+int obi_blob_sizeof(Obi_blob_p value)
+{
+	return (sizeof(Obi_blob_t) + (value->length_encoded_value));
+}
+

src/obiblob.h

@@ -21,20 +21,21 @@
 #include "obitypes.h"
 
 
-#define ELEMENT_SIZE_STR (8)								/**< The size of an element from a value of type character string.
-                              	  	 	 	 	 	 	 	 */
-#define ELEMENT_SIZE_SEQ_2 (2)								/**< The size of an element from a value of type DNA sequence encoded on 2 bits.
-                              	  	 	 	 	 	 	 	 */
-#define ELEMENT_SIZE_SEQ_4 (4)								/**< The size of an element from a value of type DNA sequence encoded on 4 bits.
-                              	  	 	 	 	 	 	 	 */
+#define ELEMENT_SIZE_STR (8)		/**< The size of an element from a value of type character string.
+                              	  	 */
+#define ELEMENT_SIZE_UINT8 (8)		/**< The size of an element from a value of type uint8_t.
+                              	  	 */
+#define ELEMENT_SIZE_SEQ_2 (2)		/**< The size of an element from a value of type DNA sequence encoded on 2 bits.
+                              	  	 */
+#define ELEMENT_SIZE_SEQ_4 (4)		/**< The size of an element from a value of type DNA sequence encoded on 4 bits.
+                              	  	 */
 
 
 /**
- * @brief Blob structure.
- *  TODO
+ * @brief Blob structure, for handling encoded values.
  */
 typedef struct Obi_blob {
-	uint8_t  element_size;		 	/**< Size in bits of one element from the value.
+	uint8_t  element_size;		 	/**< Size in bits of one element from the encoded value.
 	 	 	 	 	 	 	 	 	 */
 	int32_t  length_encoded_value;	/**< Length in bytes of the encoded value.
 	 	 	 	 	 	 	 	 	 */
@@ -46,9 +47,54 @@ typedef struct Obi_blob {
 
 
 
-// TODO doc
+/**
+ * @brief Function building a blob structure.
+ *
+ * @param encoded_value A pointer to the encoded value that will be stored in the blob.
+ * @param element_size The size in bits of one element from the encoded value.
+ * @param length_encoded_value The length in bytes of the encoded value.
+ * @param length_decoded_value The length in bytes of the decoded value.
+ *
+ * @returns A pointer to the created blob structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 Obi_blob_p obi_blob(byte_t* encoded_value, uint8_t element_size, int32_t length_encoded_value, int32_t length_decoded_value);
 
 
+/**
+ * @brief Function comparing two blobs.
+ *
+ * The encoding is compared first, then the length of the
+ * values, then the values themselves.
+ *
+ * @param value_1 A pointer to the first blob structure.
+ * @param value_2 A pointer to the second blob structure.
+ *
+ * @returns A value < 0 if value_1 < value_2,
+ * 			a value > 0 if value_1 > value_2,
+ * 			and 0 if value_1 == value_2.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_blob_compare(Obi_blob_p value_1, Obi_blob_p value_2);
+
+
+/**
+ * @brief Function calculating the size in bytes of a blob.
+ *
+ * @param value A pointer to the blob structure.
+ *
+ * @returns The size of the blob in bytes.
+ *
+ * @since October 2015
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+int obi_blob_sizeof(Obi_blob_p value);
+
+
 #endif /* OBIBLOB_H_ */
 

src/obiblob_indexer.c

@@ -15,7 +15,8 @@
 
 #include "obiblob_indexer.h"
 #include "obidms.h"
-#include "obiavl.h"
+#include "obierrno.h"
+#include "obidebug.h"
 
 
 #define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
@@ -29,9 +30,32 @@ inline Obi_indexer_p obi_create_indexer(OBIDMS_p dms, const char* name);
 
 inline Obi_indexer_p obi_open_indexer(OBIDMS_p dms, const char* name);
 
+inline Obi_indexer_p obi_clone_indexer(Obi_indexer_p indexer, const char* name);
+
 inline int obi_close_indexer(Obi_indexer_p indexer);
 
 inline index_t obi_indexer_add(Obi_indexer_p indexer, Obi_blob_p value);
 
 inline Obi_blob_p obi_indexer_get(Obi_indexer_p indexer, index_t idx);
 
+inline const char* obi_indexer_get_name(Obi_indexer_p indexer);
+
+
+char* obi_build_indexer_name(const char* column_name, obiversion_t column_version)
+{
+	char* indexer_name;
+
+	indexer_name = (char*) malloc(INDEXER_MAX_NAME * sizeof(char));
+	if (indexer_name == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating the memory for an indexer name");
+		return NULL;
+	}
+
+	strcpy(indexer_name, column_name);
+	sprintf(indexer_name+strlen(column_name), "_%d_indexer", column_version);
+
+	return indexer_name;
+}
+

src/obiblob_indexer.h

@@ -33,48 +33,182 @@ typedef OBIDMS_avl_group_p Obi_indexer_p; 		/**< Typedef to refer to the pointer
  	 	 	 	 	 	 	 	 	 	 	 	 */
 
 
-// TODO doc
+/**
+ * @brief Checks if an indexer already exists or not.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A value indicating whether the indexer exists or not.
+ * @retval 1 if the indexer exists.
+ * @retval 0 if the indexer does not exist.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 inline int obi_indexer_exists(OBIDMS_p dms, const char* name)
 {
 	return obi_avl_exists(dms, name);
 }
 
 
+/**
+ * @brief Opens an indexer and creates it if it does not already exist.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A pointer to the indexer structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 inline Obi_indexer_p obi_indexer(OBIDMS_p dms, const char* name)
 {
 	return obi_avl_group(dms, name);
 }
 
 
+/**
+ * @brief Creates an indexer.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A pointer to the AVL tree group structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 inline Obi_indexer_p obi_create_indexer(OBIDMS_p dms, const char* name)
 {
 	return obi_create_avl_group(dms, name);
 }
 
 
+/**
+ * @brief Opens an indexer.
+ *
+ * @param dms The OBIDMS to which the indexer belongs.
+ * @param name The name of the indexer.
+ *
+ * @returns A pointer to the indexer structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 inline Obi_indexer_p obi_open_indexer(OBIDMS_p dms, const char* name)
 {
 	return obi_open_avl_group(dms, name);
 }
 
 
+/**
+ * @brief Clones an indexer.
+ *
+ * @param indexer The indexer to clone.
+ * @param name The name of the new indexer.
+ *
+ * @returns A pointer on the new indexer structure.
+ * @retval NULL if an error occurred.
+ *
+ * @since May 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline Obi_indexer_p obi_clone_indexer(Obi_indexer_p indexer, const char* name)
+{
+	return obi_clone_avl_group(indexer, name);
+}
+
+
+/**
+ * @brief Closes an indexer.
+ *
+ * @param indexer A pointer to the indexer structure to close and free.
+ *
+ * @retval 0 if the operation was successfully completed.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 inline int obi_close_indexer(Obi_indexer_p indexer)
 {
 	return obi_close_avl_group(indexer);
 }
 
 
+/**
+ * @brief Indexes a blob in an indexer and returns the index referring to the blob.
+ *
+ * @param indexer A pointer to the indexer.
+ * @param value The blob to index.
+ *
+ * @returns The index of the blob newly added in the indexer.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 inline index_t obi_indexer_add(Obi_indexer_p indexer, Obi_blob_p value)
 {
 	return obi_avl_group_add(indexer, value);
 }
 
 
+/**
+ * @brief Recovers a blob from an indexer.
+ *
+ * @param indexer A pointer to the indexer.
+ * @param index The index of the blob in the indexer.
+ *
+ * @returns A pointer to the blob recovered.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 inline Obi_blob_p obi_indexer_get(Obi_indexer_p indexer, index_t idx)
 {
 	return obi_avl_group_get(indexer, idx);
 }
 
 
+/**
+ * @brief Recovers the name of an indexer.
+ *
+ * @param indexer A pointer on the indexer.
+ *
+ * @returns A pointer on the name of the indexer.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+inline const char* obi_indexer_get_name(Obi_indexer_p indexer)
+{
+	return obi_avl_group_get_name(indexer);
+}
+
+
+/**
+ * @brief Builds an indexer name in the form columnname_columnversion_indexer.
+ *
+ * @warning The returned pointer has to be freed by the caller.
+ *
+ * @param column_name The name of the column associated with the indexer.
+ * @param column_version The version of the column associated with the indexer.
+ *
+ * @returns A pointer on the indexer name built.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_build_indexer_name(const char* column_name, obiversion_t column_version);
+
+
 #endif /* OBIBLOB_INDEXER_H_ */
 

src/obidms.c

@@ -18,6 +18,7 @@
 #include <fcntl.h>
 #include <sys/types.h>
 #include <dirent.h>
+#include <unistd.h>
 
 #include "obidms.h"
 #include "obierrno.h"
@@ -202,14 +203,14 @@ int create_dms_infos_file(int dms_file_descriptor, const char* dms_name)
  *
  **********************************************************************/
 
-int obi_dms_exists(const char* dms_name)
+int obi_dms_exists(const char* dms_path)
 {
     struct stat buffer;
 	char*       directory_name;
     int 		check_dir;
 
 	// Build and check the directory name
-    directory_name = build_directory_name(dms_name);
+    directory_name = build_directory_name(dms_path);
 	if (directory_name == NULL)
 		return -1;
 
@@ -224,14 +225,15 @@ int obi_dms_exists(const char* dms_name)
 }
 
 
-OBIDMS_p obi_create_dms(const char* dms_name)
+OBIDMS_p obi_create_dms(const char* dms_path)
 {
-	char* directory_name;
-	DIR*  dms_dir;
-	int   dms_file_descriptor;
+	char*  directory_name;
+	DIR*   dms_dir;
+	int    dms_file_descriptor;
+	size_t i, j;
 
 	// Build and check the directory name
-	directory_name = build_directory_name(dms_name);
+	directory_name = build_directory_name(dms_path);
 	if (directory_name == NULL)
 		return NULL;
 
@@ -250,7 +252,7 @@ OBIDMS_p obi_create_dms(const char* dms_name)
 		return NULL;
 	}
 
-	// Get file descriptor of DMS directory to create the indexer directory
+	// Get file descriptor of DMS directory to create other directories
 	dms_dir = opendir(directory_name);
 	if (dms_dir == NULL)
 	{
@@ -278,22 +280,40 @@ OBIDMS_p obi_create_dms(const char* dms_name)
 		return NULL;
 	}
 
+	// Create the view directory
+	if (mkdirat(dms_file_descriptor, VIEW_DIR_NAME, 00777) < 0)
+	{
+		obi_set_errno(OBIVIEW_ERROR);
+		obidebug(1, "\nProblem creating a view directory");
+		return NULL;
+	}
+
+	// Isolate the dms name
+	j = 0;
+	for (i=0; i<strlen(dms_path); i++)
+	{
+		if (dms_path[i] == '/')
+			j = i+1;
+		i++;
+	}
+
 	// Create the informations file
-	if (create_dms_infos_file(dms_file_descriptor, dms_name) < 0)
+	if (create_dms_infos_file(dms_file_descriptor, dms_path+j) < 0)
 		return NULL;
 
-	return obi_open_dms(dms_name);
+	return obi_open_dms(dms_path);
 }
 
 
-OBIDMS_p obi_open_dms(const char* dms_name)
+OBIDMS_p obi_open_dms(const char* dms_path)
 {
 	OBIDMS_p	dms;
-	char*	 	directory_name;
+	char*	 	relative_directory_path;
 	char* 		infos_file_name;
 	int			infos_file_descriptor;
 	bool		little_endian_dms;
 	bool 		little_endian_platform;
+	size_t		i, j;
 
 	dms = NULL;
 
@@ -307,18 +327,37 @@ OBIDMS_p obi_open_dms(const char* dms_name)
 	}
 
 	// Build and check the directory name
-	directory_name = build_directory_name(dms_name);
-	if (directory_name == NULL)
+	relative_directory_path = build_directory_name(dms_path);
+	if (relative_directory_path == NULL)
 	{
 		free(dms);
 		return NULL;
 	}
 
-	strncpy(dms->directory_name, directory_name, OBIDMS_MAX_NAME);
-	free(directory_name);
+	// Build and store the absolute path to the DMS
+	if (getcwd(dms->directory_path, MAX_PATH_LEN) == NULL)
+	{
+		obi_set_errno(OBIDMS_UNKNOWN_ERROR);
+		obidebug(1, "\nError getting the absolute path to the current working directory");
+		free(relative_directory_path);
+		return NULL;
+	}
+	strcat(dms->directory_path, "/");
+	strcat(dms->directory_path, relative_directory_path);
+	free(relative_directory_path);
+
+	// Isolate and store the dms name
+	j = 0;
+	for (i=0; i<strlen(dms_path); i++)
+	{
+		if (dms_path[i] == '/')
+			j = i+1;
+		i++;
+	}
+	strcpy(dms->dms_name, dms_path+j);
 
 	// Try to open the directory
-	dms->directory = opendir(dms->directory_name);
+	dms->directory = opendir(dms->directory_path);
 	if (dms->directory == NULL)
 	{
 		switch (errno)
@@ -355,7 +394,7 @@ OBIDMS_p obi_open_dms(const char* dms_name)
 	}
 
 	// Open informations file to check endianness
-	infos_file_name = build_infos_file_name(dms_name);
+	infos_file_name = build_infos_file_name(dms->dms_name);
 	infos_file_descriptor = openat(dms->dir_fd, infos_file_name, O_RDONLY, 0777);
 	if (infos_file_descriptor < 0)
 	{
@@ -416,11 +455,35 @@ OBIDMS_p obi_open_dms(const char* dms_name)
 		return NULL;
 	}
 
+	// Open the view directory
+	dms->view_directory = opendir_in_dms(dms, VIEW_DIR_NAME);
+	if (dms->view_directory == NULL)
+	{
+		obi_set_errno(OBIDMS_UNKNOWN_ERROR);
+		obidebug(1, "\nError opening the view directory");
+		closedir(dms->indexer_directory);
+		closedir(dms->directory);
+		free(dms);
+		return NULL;
+	}
+
+	// Store the view directory's file descriptor
+	dms->view_dir_fd = dirfd(dms->view_directory);
+	if (dms->view_dir_fd < 0)
+	{
+		obi_set_errno(OBIDMS_UNKNOWN_ERROR);
+		obidebug(1, "\nError getting the file descriptor of the view directory");
+		closedir(dms->view_directory);
+		closedir(dms->directory);
+		free(dms);
+		return NULL;
+	}
+
 	// Initialize the list of opened columns
 	dms->opened_columns = (Opened_columns_list_p) malloc(sizeof(Opened_columns_list_t));
 	(dms->opened_columns)->nb_opened_columns = 0;
 
-	// Initialize the list of opened indexers	// TODO should be handled somewhere else?
+	// Initialize the list of opened indexers
 	dms->opened_indexers = (Opened_indexers_list_p) malloc(sizeof(Opened_indexers_list_t));
 	(dms->opened_indexers)->nb_opened_indexers = 0;
 
@@ -455,7 +518,7 @@ int obi_close_dms(OBIDMS_p dms)
 		while ((dms->opened_columns)->nb_opened_columns > 0)
 			obi_close_column(*((dms->opened_columns)->columns));
 
-		// Close dms and indexer directories
+		// Close dms, and view and indexer directories
 		if (closedir(dms->directory) < 0)
 		{
 			obi_set_errno(OBIDMS_MEMORY_ERROR);
@@ -463,13 +526,20 @@ int obi_close_dms(OBIDMS_p dms)
 			free(dms);
 			return -1;
 		}
-		if (closedir(dms->indexer_directory) < 0)		// TODO should be handled somewhere else?
+		if (closedir(dms->indexer_directory) < 0)
 		{
 			obi_set_errno(OBI_INDEXER_ERROR);
 			obidebug(1, "\nError closing an indexer directory");
 			free(dms);
 			return -1;
 		}
+		if (closedir(dms->view_directory) < 0)
+		{
+			obi_set_errno(OBIVIEW_ERROR);
+			obidebug(1, "\nError closing a view directory");
+			free(dms);
+			return -1;
+		}
 		free(dms);
 	}
 
@@ -488,10 +558,10 @@ int obi_dms_is_column_name_in_list(OBIDMS_p dms, const char* column_name)
 	{
 		if (!strcmp(((*((columns_list->columns)+i))->header)->name, column_name))
 		{	// Found it
-			return 0;
+			return 1;
 		}
 	}
-	return 1;
+	return 0;
 }
 
 
@@ -512,7 +582,7 @@ OBIDMS_column_p obi_dms_get_column_from_list(OBIDMS_p dms, const char* column_na
 }
 
 
-void obi_dms_list_column(OBIDMS_p dms, OBIDMS_column_p column)
+void obi_dms_list_column(OBIDMS_p dms, OBIDMS_column_p column)	// TODO add check if column already in list?
 {
 	*(((dms->opened_columns)->columns)+((dms->opened_columns)->nb_opened_columns)) = column;
 	((dms->opened_columns)->nb_opened_columns)++;
@@ -551,7 +621,7 @@ Obi_indexer_p obi_dms_get_indexer_from_list(OBIDMS_p dms, const char* indexer_na
 
 	for (i=0; i < (indexers_list->nb_opened_indexers); i++)
 	{
-		if (!strcmp(((indexers_list->indexers)[i])->name, indexer_name))	// TODO it references something in AVL_group struct
+		if (!strcmp(obi_indexer_get_name((indexers_list->indexers)[i]), indexer_name))
 		{	// Found the indexer already opened, return it
 			return (indexers_list->indexers)[i];
 		}
@@ -561,7 +631,7 @@ Obi_indexer_p obi_dms_get_indexer_from_list(OBIDMS_p dms, const char* indexer_na
 }
 
 
-void obi_dms_list_indexer(OBIDMS_p dms, Obi_indexer_p indexer)
+void obi_dms_list_indexer(OBIDMS_p dms, Obi_indexer_p indexer)  	// TODO add check if indexer already in list?
 {
 	*(((dms->opened_indexers)->indexers)+((dms->opened_indexers)->nb_opened_indexers)) = indexer;
 	((dms->opened_indexers)->nb_opened_indexers)++;
@@ -577,7 +647,7 @@ int obi_dms_unlist_indexer(OBIDMS_p dms, Obi_indexer_p indexer)
 
 	for (i=0; i < indexers_list->nb_opened_indexers; i++)
 	{
-		if (!strcmp(((indexers_list->indexers)[i])->name, indexer->name))	// TODO it references something in AVL_group struct
+		if (!strcmp(obi_indexer_get_name((indexers_list->indexers)[i]), indexer->name))
 		{	// Found the indexer. Rearrange list
 			(indexers_list->nb_opened_indexers)--;
 			(indexers_list->indexers)[i] = (indexers_list->indexers)[indexers_list->nb_opened_indexers];
@@ -602,15 +672,7 @@ char* obi_dms_get_dms_path(OBIDMS_p dms)
 		return NULL;
 	}
 
-	if (getcwd(full_path, MAX_PATH_LEN) == NULL)	// TODO not sure at all about this because the DMS must be in the working directory.
-	{												// Maybe better to store when opening, but opening function seems to assume that too.
-		obi_set_errno(OBI_UTILS_ERROR);
-		obidebug(1, "\nError getting the path to a file or directory");
-		return NULL;
-	}
-
-	strcat(full_path, "/");
-	strcat(full_path, dms->directory_name);
+	strcpy(full_path, dms->directory_path);
 
 	return full_path;
 }

src/obidms.h

@@ -23,14 +23,15 @@
 #include <stdbool.h>
 
 #include "obierrno.h"
-//#include "obidmscolumn.h"
-//#include "obiblob_indexer.h"
+#include "obitypes.h"
 
 
 #define OBIDMS_MAX_NAME (2048) 	    			/**< The maximum length of an OBIDMS name.
  	 	 	 	 	 	 	 	 	 	 	 	 */
 #define INDEXER_DIR_NAME "OBIBLOB_INDEXERS"  	/**< The name of the Obiblob indexer directory.
                                 	 	 	 	 */
+#define VIEW_DIR_NAME "VIEWS"					/**< The name of the view directory.
+												 */
 #define TAXONOMY_DIR_NAME "TAXONOMY"    		/**< The name of the taxonomy directory.
                                 	 	 	 	 */
 #define MAX_NB_OPENED_COLUMNS (100) 			/**< The maximum number of columns open at the same time.
@@ -42,26 +43,33 @@
  	 	 	 	 	 	 	 	 	 	 	 	 */
 
 
-typedef int32_t obiversion_t; 	/**< TODO double
-                                 */
+struct OBIDMS_column;							/**< Declarations to avoid circular dependencies. */
+typedef struct OBIDMS_column* OBIDMS_column_p;  /**< Declarations to avoid circular dependencies. */
 
-
-struct OBIDMS_column;		// TODO
-typedef struct OBIDMS_column* OBIDMS_column_p;
-
-typedef struct Opened_columns_list {		// TODO Handle the problem linked to columns with the same name + means only one version
-	int	 		 			nb_opened_columns;
-	OBIDMS_column_p		 	columns[MAX_NB_OPENED_COLUMNS];
+/**
+ * @brief Structure listing the columns opened in a DMS, identified by their name and version number.
+ */
+typedef struct Opened_columns_list {
+	int	 		 			nb_opened_columns;	  				/**< Number of opened columns.
+	 	   	   	   	   	   	   	   	   	   	   	   	   	   	   	 */
+	OBIDMS_column_p		 	columns[MAX_NB_OPENED_COLUMNS];	  	/**< Array of pointers on the opened columns.
+ 	   	   	   	   	   	   	 	 	 	 	 	 	 	 	 	 */
 } Opened_columns_list_t, *Opened_columns_list_p;
 
 
-struct OBIDMS_avl_group;		// TODO
-typedef struct OBIDMS_avl_group* OBIDMS_avl_group_p;
-typedef OBIDMS_avl_group_p Obi_indexer_p;
+// TODO Need to find a way to not refer to AVLs specifically
+struct OBIDMS_avl_group;								/**< Declarations to avoid circular dependencies. */
+typedef struct OBIDMS_avl_group* OBIDMS_avl_group_p;	/**< Declarations to avoid circular dependencies. */
+typedef OBIDMS_avl_group_p Obi_indexer_p;				/**< Declarations to avoid circular dependencies. */
 
+/**
+ * @brief Structure listing the indexers opened in a DMS, identified by their name.
+ */
 typedef struct Opened_indexers_list {
-	int		 		 		nb_opened_indexers;
-	Obi_indexer_p		    indexers[MAX_NB_OPENED_INDEXERS];
+	int		 		 		nb_opened_indexers;					/**< Number of opened indexers.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
+	Obi_indexer_p		    indexers[MAX_NB_OPENED_INDEXERS];	/**< Array of pointers on the opened indexers.
+	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 } Opened_indexers_list_t, *Opened_indexers_list_p;
 
 
@@ -72,21 +80,32 @@ typedef struct Opened_indexers_list {
  * and opening of an OBITools Data Management System (DMS)
  */
 typedef struct OBIDMS {
+	char					dms_name[OBIDMS_MAX_NAME+1];		/** The name of the DMS.
+																 */
 	char					directory_name[OBIDMS_MAX_NAME+1];	/**< The name of the directory
 																 *   containing the DMS.
 																 */
+	char					directory_path[MAX_PATH_LEN];     	/**< The absolute path of the directory
+																 *   containing the DMS.
+																 */
 	DIR*					directory;							/**< A directory entry usable to
 																 *   refer and scan the database directory.
 																 */
 	int     				dir_fd;								/**< The file descriptor of the directory entry
 																 *   usable to refer and scan the database directory.
 																 */
-	DIR*					indexer_directory;						/**< A directory entry usable to
+	DIR*					indexer_directory;					/**< A directory entry usable to
 																 *   refer and scan the indexer directory.
 																 */
-	int						indexer_dir_fd;							/**< The file descriptor of the directory entry
+	int						indexer_dir_fd;						/**< The file descriptor of the directory entry
 																 *   usable to refer and scan the indexer directory.
 																 */
+	DIR*					view_directory;						/**< A directory entry usable to
+																 *   refer and scan the view directory.
+																 */
+	int						view_dir_fd;						/**< The file descriptor of the directory entry
+																 *   usable to refer and scan the view directory.
+																 */
 	bool					little_endian;						/**< Endianness of the database.
 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 */
 	Opened_columns_list_p 	opened_columns;						/**< List of opened columns.
@@ -99,9 +118,9 @@ typedef struct OBIDMS {
 /**
  * @brief Checks if an OBIDMS exists.
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *
- * @returns An integer value indicating the status of the database
+ * @returns An integer value indicating the status of the database.
  * @retval 1 if the database exists.
  * @retval 0 if the database does not exist.
  * @retval -1 if an error occurred.
@@ -109,7 +128,7 @@ typedef struct OBIDMS {
  * @since May 2015
  * @author Eric Coissac (eric.coissac@metabarcoding.org)
  */
-int obi_dms_exists(const char* dms_name);
+int obi_dms_exists(const char* dms_path);
 
 
 /**
@@ -121,7 +140,7 @@ int obi_dms_exists(const char* dms_name);
  *
  * A directory to store Obiblob indexers is also created.
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *                 The actual directory name used to store the DMS will be
  *                 `<dms_name>.obidms`.
  *
@@ -138,7 +157,7 @@ OBIDMS_p obi_create_dms(const char* dms_name);
 /**
  * @brief Opens an existing OBITools Data Management instance (OBIDMS).
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *
  * @returns A pointer to an OBIDMS structure describing the opened DMS.
  * @retval NULL if an error occurred.
@@ -147,7 +166,7 @@ OBIDMS_p obi_create_dms(const char* dms_name);
  * @since May 2015
  * @author Eric Coissac (eric.coissac@metabarcoding.org)
  */
-OBIDMS_p obi_open_dms(const char* dms_name);
+OBIDMS_p obi_open_dms(const char* dms_path);
 
 
 /**
@@ -156,7 +175,7 @@ OBIDMS_p obi_open_dms(const char* dms_name);
  * If the database already exists, this function opens it, otherwise it
  * creates a new database.
  *
- * @param dms_name A pointer to a C string containing the name of the database.
+ * @param dms_path A pointer to a C string containing the path to the database.
  *
  * @returns A pointer to an OBIDMS structure describing the OBIDMS.
  * @retval NULL if an error occurred.
@@ -165,7 +184,7 @@ OBIDMS_p obi_open_dms(const char* dms_name);
  * @since May 2015
  * @author Eric Coissac (eric.coissac@metabarcoding.org)
  */
-OBIDMS_p obi_dms(const char* dms_name);
+OBIDMS_p obi_dms(const char* dms_path);
 
 
 /**
@@ -186,60 +205,152 @@ OBIDMS_p obi_dms(const char* dms_name);
 int obi_close_dms(OBIDMS_p dms);
 
 
-// TODO doc
+/**
+ * @brief Checks if a column with a given name is in the list of opened columns.
+ *
+ * @warning: Checking only the name means that the column can be of any version.
+ *
+ * @param dms The OBIDMS.
+ * @param column_name The column name that should be looked for.
+ *
+ * @returns An integer value indicating whether there is at least one column with that name
+ * 			in the list of opened columns.
+ * @retval 0 if there is at least one column with that name in the list of opened columns.
+ * @retval 1 if there is no column with that name in the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 int obi_dms_is_column_name_in_list(OBIDMS_p dms, const char* column_name);
 
 
+/**
+ * @brief Returns a column identified by its name and its version number from the list of opened columns.
+ *
+ * @param dms The OBIDMS.
+ * @param column_name The column name that should be looked for.
+ * @param version The version number of the column that should be looked for.
+ *
+ * @returns A pointer on the column if it was found in the list of opened columns.
+ * @retval NULL if the column was not found in the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 OBIDMS_column_p obi_dms_get_column_from_list(OBIDMS_p dms, const char* column_name, obiversion_t version);
 
 
+/**
+ * @brief Adds a column identified by its name and its version number in the list of opened columns.
+ *
+ * @warning obi_dms_get_column_from_list() should be used first to check if the column isn't already listed.
+ *
+ * @param dms The OBIDMS.
+ * @param column A pointer on the column that should be added in the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 void obi_dms_list_column(OBIDMS_p dms, OBIDMS_column_p column);
 
 
+/**
+ * @brief Removes a column identified by its name and its version number from the list of opened columns.
+ *
+ * @param dms The OBIDMS.
+ * @param column A pointer on the column that should be removed from the list of opened columns.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 int obi_dms_unlist_column(OBIDMS_p dms, OBIDMS_column_p column);
 
 
+/**
+ * @brief Returns an indexer identified by its name from the list of opened indexers.
+ *
+ * @param dms The OBIDMS.
+ * @param indexer_name The indexer name that should be looked for.
+ *
+ * @returns A pointer on the indexer if it was found in the list of opened indexers.
+ * @retval NULL if the indexer was not found in the list of opened indexers.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 Obi_indexer_p obi_dms_get_indexer_from_list(OBIDMS_p dms, const char* indexer_name);
 
 
+/**
+ * @brief Adds an indexer identified by its name in the list of opened indexers.
+ *
+ * @warning obi_dms_get_indexer_from_list() should be used first to check if the indexer isn't already listed.
+ *
+ * @param dms The OBIDMS.
+ * @param indexer A pointer on the indexer that should be added in the list of opened indexers.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 void obi_dms_list_indexer(OBIDMS_p dms, Obi_indexer_p indexer);
 
 
+/**
+ * @brief Removes an indexer identified by its name from the list of opened indexers.
+ *
+ * @param dms The OBIDMS.
+ * @param column A pointer on the indexer that should be removed from the list of opened indexers.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 int obi_dms_unlist_indexer(OBIDMS_p dms, Obi_indexer_p indexer);
 
 
-char* obi_dms_get_path(OBIDMS_p dms);
-
-
-/**	TODO
- * @brief Internal function getting the full path of a file or a directory from its
- *        path relative to a directory file descriptor.
+/**
+ * @brief Gets the full path to the DMS.
  *
  * @warning The returned pointer has to be freed by the caller.
  *
- * @param directory_file_descriptor The file descriptor for the directory to which
- * 									path_name is relative.
- * @param path_name The path name for the file or directory, relative to directory_file_descriptor.
+ * @param dms The DMS.
  *
  * @returns A pointer to the full path.
  * @retval NULL if an error occurred.
  *
- * @since June 2015
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
+char* obi_dms_get_dms_path(OBIDMS_p dms);
+
+
+/**
+ * @brief Gets the full path of a file or a directory from its
+ *        path relative to the DMS.
+ *
+ * @warning The returned pointer has to be freed by the caller.
+ *
+ * @param dms The DMS to which path_name is relative.
+ * @param path_name The path name for the file or directory, relative to the DMS.
+ *
+ * @returns A pointer to the full path.
+ * @retval NULL if an error occurred.
+ *
+ * @since April 2016
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
 char* obi_dms_get_full_path(OBIDMS_p dms, const char* path_name);
 
 
 /**
- * @brief Replacement function for opendirat() : open a directory relative to a directory file descriptor.
+ * @brief Opens a directory relative to the DMS.
  *
- * @param directory_file_descriptor The file descriptor for the directory in which the directory should be opened.
- * @param path_name The path name for the directory to be opened, relative to directory_file_descriptor.
+ * @param dms The DMS to which path_name is relative.
+ * @param path_name The path name for the directory to be opened, relative to the DMS.
  *
- * @returns The file descriptor of the opened directory.
+ * @returns The directory stream of the opened directory.
  * @retval NULL if an error occurred.
  *
- * @since June 2015
+ * @since April 2016
  * @author Celine Mercier (celine.mercier@metabarcoding.org)
  */
 DIR* opendir_in_dms(OBIDMS_p dms, const char* path_name);

src/obidmscolumn.c

@@ -161,7 +161,14 @@ static char* build_column_file_name(const char* column_name, obiversion_t versio
 
 	// Build the file name
 	version_number_length = (version_number == 0 ? 1 : (int)(log10(version_number)+1));
-	file_name =	(char*) malloc((strlen(column_name) + version_number_length + 6)*sizeof(char));	// TODO check the mallocs...
+	file_name =	(char*) malloc((strlen(column_name) + version_number_length + 6)*sizeof(char));
+	if (file_name == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating the memory for a column file name");
+		return NULL;
+	}
+
 	if (sprintf(file_name,"%s@%d.odc", column_name, version_number) < 0)
 	{
 		obi_set_errno(OBICOL_MEMORY_ERROR);
@@ -179,6 +186,13 @@ static char* build_version_file_name(const char* column_name)
 
 	// Build the file name
 	file_name =	(char*) malloc((strlen(column_name) + 5)*sizeof(char));
+	if (file_name == NULL)
+	{
+		obi_set_errno(OBI_MALLOC_ERROR);
+		obidebug(1, "\nError allocating the memory for a version file name");
+		return NULL;
+	}
+
 	if (sprintf(file_name,"%s.odv", column_name) < 0)
 	{
 		obi_set_errno(OBICOL_MEMORY_ERROR);
@@ -535,7 +549,7 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 	index_t						minimum_line_count;
 	OBIType_t  					returned_data_type;
 	OBIType_t  					stored_data_type;
-	char						final_indexer_name[INDEXER_MAX_NAME];
+	char*			    		final_indexer_name;
 
 	new_column = NULL;
 
@@ -555,16 +569,41 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 		obidebug(1, "\nCan't create column because of invalid data type");
 		return NULL;
 	}
-	if (((data_type == OBI_STR) || (data_type == OBI_SEQ)) && (strcmp(indexer_name, "") == 0))
-	{	// TODO make function in obiblob_indexer.c and discuss the fact that it's here
-		strcpy(final_indexer_name, column_name);
-		strcat(final_indexer_name, "_indexer");
+
+	// Get the column directory structure associated to the column
+	column_directory = obi_column_directory(dms, column_name);
+	if (column_directory == NULL)
+	{
+		obi_set_errno(OBICOLDIR_UNKNOWN_ERROR);
+		obidebug(1, "\nError opening a column directory structure");
+		return NULL;
+	}
+
+	// Get the latest version number
+	version_number = obi_get_new_version_number(column_directory, true);
+	if (version_number < 0)
+	{
+		return NULL;
+	}
+
+	// Build the indexer name if needed
+	if ((data_type == OBI_STR) || (data_type == OBI_SEQ) || (data_type == OBI_QUAL))
+	{
+		if (strcmp(indexer_name, "") == 0)
+		{
+			final_indexer_name = obi_build_indexer_name(column_name, version_number);
+			if (final_indexer_name == NULL)
+				return NULL;
+		}
+		else
+		{
+			final_indexer_name = (char*) malloc((strlen(indexer_name)+1)*sizeof(char));
+			strcpy(final_indexer_name, indexer_name);
+		}
 	}
-	else
-		strcpy(final_indexer_name, indexer_name);
 
 	returned_data_type = data_type;
-	if ((data_type == OBI_STR) || (data_type == OBI_SEQ))
+	if ((data_type == OBI_STR) || (data_type == OBI_SEQ) || (data_type == OBI_QUAL))
 	// stored data is indices referring to data stored elsewhere
 		stored_data_type = OBI_IDX;
 	else
@@ -609,27 +648,11 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 		return NULL;
 	}
 
-	// Get the column directory structure associated to the column
-	column_directory = obi_column_directory(dms, column_name);
-	if (column_directory == NULL)
-	{
-		obi_set_errno(OBICOLDIR_UNKNOWN_ERROR);
-		obidebug(1, "\nError opening a column directory structure");
-		return NULL;
-	}
-
 	// Calculate the size needed
 	header_size = obi_get_platform_header_size();
 	data_size = obi_array_sizeof(stored_data_type, nb_lines, nb_elements_per_line);
 	file_size = header_size + data_size;
 
-	// Get the latest version number
-	version_number = obi_get_new_version_number(column_directory, true);
-	if (version_number < 0)
-	{
-		return NULL;
-	}
-
 	// Get the column file name
 	column_file_name = build_column_file_name(column_name, version_number);
 	if (column_file_name == NULL)
@@ -727,8 +750,8 @@ OBIDMS_column_p obi_create_column(OBIDMS_p    dms,
 	if (comments != NULL)
 		strncpy(header->comments, comments, COMMENTS_MAX_LENGTH);
 
-	// If the data type is OBI_STR or OBI_SEQ, the associated obi_indexer is opened or created
-	if ((returned_data_type == OBI_STR) || (returned_data_type == OBI_SEQ))
+	// If the data type is OBI_STR, OBI_SEQ or OBI_QUAL, the associated obi_indexer is opened or created
+	if ((returned_data_type == OBI_STR) || (returned_data_type == OBI_SEQ) || (returned_data_type == OBI_QUAL))
 	{
 		new_column->indexer = obi_indexer(dms, final_indexer_name);
 		if (new_column->indexer == NULL)
@@ -773,7 +796,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	column_directory = obi_open_column_directory(dms, column_name);
 	if (column_directory == NULL)
 	{
-		//obidebug(1, "\nError opening a column directory structure");	// TODO
+		obidebug(1, "\nError opening a column directory structure");
 		return NULL;
 	}
 
@@ -809,7 +832,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	if (column_file_descriptor < 0)
 	{
 		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		//obidebug(1, "\nError opening column file");
+		obidebug(1, "\nError opening column file");
 		free(column_file_name);
 		return NULL;
 	}
@@ -842,7 +865,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	column->header 			 = mmap(NULL,
 			                  	  	header_size,
 			                  	  	PROT_READ,
-			                  	  	MAP_PRIVATE,
+			                  	  	MAP_SHARED,
 			                  	  	column_file_descriptor,
 			                  	  	0
 			                 	   );
@@ -860,7 +883,7 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 	column->data   			 = mmap(NULL,
 			                 	 	(column->header)->data_size,
 			                 	 	PROT_READ,
-									MAP_PRIVATE,
+									MAP_SHARED,
 			                 	 	column_file_descriptor,
 			                 	 	header_size
 			                	   );
@@ -877,8 +900,8 @@ OBIDMS_column_p obi_open_column(OBIDMS_p dms,
 
 	column->writable = false;
 
-	// If the data type is OBI_STR or OBI_SEQ, the associated indexer is opened
-	if (((column->header)->returned_data_type == OBI_STR) || ((column->header)->returned_data_type == OBI_SEQ))
+	// If the data type is OBI_STR, OBI_SEQ or OBI_QUAL, the associated indexer is opened
+	if (((column->header)->returned_data_type == OBI_STR) || ((column->header)->returned_data_type == OBI_SEQ) || ((column->header)->returned_data_type == OBI_QUAL))
 	{
 		column->indexer = obi_open_indexer(dms, (column->header)->indexer_name);
 		if (column->indexer == NULL)
@@ -917,8 +940,6 @@ OBIDMS_column_p obi_clone_column(OBIDMS_p         dms,
 	size_t 			  line_size;
 	index_t			  i, index;
 
-	obidebug(1, "\nline_selection == NULL = %d\n", (line_selection == NULL));
-
 	column_to_clone = obi_open_column(dms, column_name, version_number);
 	if (column_to_clone == NULL)
 	{
@@ -1005,8 +1026,8 @@ int obi_close_column(OBIDMS_column_p column)
 		// Close column directory if it was the last column opened from that directory
 		close_dir = obi_dms_is_column_name_in_list(column->dms, (column->header)->name);
 
-		// If the data type is OBI_STR or OBI_SEQ, the associated indexer is closed
-		if (((column->header)->returned_data_type == OBI_STR) || ((column->header)->returned_data_type == OBI_SEQ))
+		// If the data type is OBI_STR, OBI_SEQ or OBI_QUAL, the associated indexer is closed
+		if (((column->header)->returned_data_type == OBI_STR) || ((column->header)->returned_data_type == OBI_SEQ) || ((column->header)->returned_data_type == OBI_QUAL))
 			if (obi_close_indexer(column->indexer) < 0)
 				ret_val = -1;
 
@@ -1132,6 +1153,14 @@ int obi_enlarge_column(OBIDMS_column_p column)
 	int 	column_file_descriptor;
 	char* 	column_file_name;
 
+	// Check if the column is read-only
+	if (!(column->writable))
+	{
+		obi_set_errno(OBICOL_UNKNOWN_ERROR);
+		obidebug(1, "\nError trying to enlarge a read-only column");
+    	return -1;
+	}
+
 	// Get the column file name
 	column_file_name = build_column_file_name((column->header)->name, (column->header)->version);
 	if (column_file_name == NULL)
@@ -1186,12 +1215,12 @@ int obi_enlarge_column(OBIDMS_column_p column)
 	}
 
 	column->data  = mmap(NULL,
-					new_data_size,
-					PROT_READ | PROT_WRITE,
-					MAP_SHARED,
-					column_file_descriptor,
-					header_size
-					);
+						 new_data_size,
+						 PROT_READ | PROT_WRITE,
+						 MAP_SHARED,
+						 column_file_descriptor,
+						 header_size
+						);
 	if (column->data == MAP_FAILED)
 	{
 		obi_set_errno(OBICOL_UNKNOWN_ERROR);
@@ -1251,13 +1280,29 @@ void obi_ini_to_NA_values(OBIDMS_column_p column,
 					}
 					break;
 
-	case OBI_STR:   // fallthrough
-	case OBI_SEQ:   // fallthrough
 	case OBI_IDX:   for (i=start;i<end;i++)
 					{
 						*(((index_t*) (column->data)) + i) = OBIIdx_NA;
 					}
 					break;
+
+	case OBI_QUAL:  for (i=start;i<end;i++)		// case not used since OBI_QUAL is only a returned_data_type
+					{
+						*(((index_t*) (column->data)) + i) = OBIIdx_NA;
+					}
+					break;
+
+	case OBI_STR:   for (i=start;i<end;i++)		// case not used since OBI_QUAL is only a returned_data_type
+					{
+						*(((index_t*) (column->data)) + i) = OBIIdx_NA;
+					}
+					break;
+
+	case OBI_SEQ:   for (i=start;i<end;i++)		// case not used since OBI_QUAL is only a returned_data_type
+					{
+						*(((index_t*) (column->data)) + i) = OBIIdx_NA;
+					}
+					break;
 	}
 }
 
@@ -1319,12 +1364,12 @@ OBIDMS_column_header_p obi_column_get_header_from_name(OBIDMS_p dms, const char*
 
 	// Fill the header structure
 	header = mmap(NULL,
-			 header_size,
-			 PROT_READ,
-			 MAP_SHARED,
-			 column_file_descriptor,
-			 0
-			 );
+			 	  header_size,
+				  PROT_READ,
+				  MAP_SHARED,
+				  column_file_descriptor,
+				  0
+			 	 );
 
 	if (header == MAP_FAILED)
 	{
@@ -1427,40 +1472,14 @@ int obi_column_prepare_to_set_value(OBIDMS_column_p column, index_t line_nb)
 }
 
 
-int obi_column_prepare_to_get_value(OBIDMS_column_p column, index_t line_nb)
+int obi_column_prepare_to_get_value(OBIDMS_column_p column, index_t line_nb)	// TODO problem with some columns in a view being empty or shorter and triggering an error because they've been truncated when the view was closed. Fixed with obiview.c in update_lines() for now
 {
 	if ((line_nb+1) > ((column->header)->line_count))
 	{
 		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError trying to get a value that is beyond the current number of lines used in the column");
+		obidebug(1, "\nError trying to get a value that is beyond the current number of lines of the column");
 		return -1;
 	}
 	return 0;
 }
 
-
-char* obi_column_format_date(time_t date)	// TODO put in utils.c
-{
-	char*       formatted_time;
-	struct tm*  tmp;
-
-	formatted_time = (char*) malloc(FORMATTED_TIME_LENGTH*sizeof(char));
-
-	tmp = localtime(&date);
-	if (tmp == NULL)
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError formatting a date");
-		return NULL;
-	}
-
-	if (strftime(formatted_time, FORMATTED_TIME_LENGTH, "%c", tmp) == 0)
-	{
-		obi_set_errno(OBICOL_UNKNOWN_ERROR);
-		obidebug(1, "\nError formatting a date");
-		return NULL;
-	}
-
-	return formatted_time;
-}
-

src/obidmscolumn.h

@@ -34,14 +34,9 @@
                                 	   	 */
 #define MAXIMUM_LINE_COUNT (1000000000) /**< The maximum line count for the data of a column. //TODO
                                 	     */
-#define FORMATTED_TIME_LENGTH (1024)    /**< The length allocated for the character string containing a formatted date.
- 	 	 	 	 	 	 	 	 	     */
 #define COMMENTS_MAX_LENGTH (2048)      /**< The maximum length for comments.
  	 	 	 	 	 	 	 	 	     */
 
-typedef int32_t obiversion_t; 	/**< Used to store the column version number
-                                 */
-
 
 /**
  * @brief OBIDMS column header structure.
@@ -329,27 +324,34 @@ int obi_close_header(OBIDMS_column_header_p header);
 index_t obi_column_get_element_index_from_name(OBIDMS_column_p column, const char* element_name);
 
 
-// TODO doc
+/**
+ * @brief Prepares a column to set a value.
+ *
+ * @param column A pointer on an OBIDMS column.
+ * @param line_nb The number of the line at which the value will be set.
+ *
+ * @retval 0 if the operation was successfully completed.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 int obi_column_prepare_to_set_value(OBIDMS_column_p column, index_t line_nb);
 
 
-// TODO doc
+/**
+ * @brief Prepares a column to recover a value.
+ *
+ * @param column A pointer on an OBIDMS column.
+ * @param line_nb The number of the line at which the value will be recovered.
+ *
+ * @retval 0 if the operation was successfully completed.
+ * @retval -1 if an error occurred.
+ *
+ * @since April 2016
+ * @author Celine Mercier (celine.mercier@metabarcoding.org)
+ */
 int obi_column_prepare_to_get_value(OBIDMS_column_p column, index_t line_nb);
 
 
-/** TODO put in utils.c
- * @brief Formats a date in a way that is easy to read.
- *
- * @warning The pointer returned must be freed by the caller.
- *
- * @param date A date.
- *
- * @returns The date formatted in a way that is easy to read.
- *
- * @since October 2015
- * @author Celine Mercier (celine.mercier@metabarcoding.org)
- */
-char* obi_column_format_date(time_t date);
-
-
 #endif /* OBIDMSCOLUMN_H_ */

src/obidmscolumn_bool.c

@@ -15,11 +15,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************

src/obidmscolumn_char.c

@@ -15,11 +15,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************

src/obidmscolumn_float.c

@@ -15,11 +15,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************

src/obidmscolumn_idx.c

@@ -15,11 +15,6 @@
 
 #include "obidmscolumn.h"
 #include "obitypes.h"
-#include "obierrno.h"
-#include "obidebug.h"
-
-
-#define DEBUG_LEVEL 0	// TODO has to be defined somewhere else (cython compil flag?)
 
 
 /**********************************************************************